IEBP003: sample_ids

Every Sample() passes id=.

Category: best_practices · Applies to: eval, helper

What it does

Flags each Sample(...) call without an id= keyword.

Why is this bad?

Without a stable id a sample is identified by its position. Shuffling, --limit, reruns and dataset updates all change positions, so results can no longer be compared sample by sample.

Example

Sample(input=record["question"], target=record["answer"])

Use instead:

Sample(input=record["question"], target=record["answer"], id=record["id"])

Suppress on a line with # inspect-evals-lint: ignore[IEBP003] or ignore[sample_ids]; select or ignore it in configuration by either, or by the prefix IEBP.