IEBP003: sample_ids¶
Every Sample() passes id=, and every FieldSpec() names an id field.
Category: best_practices ยท Applies to: eval, helper
What it does¶
Flags each Sample(...) call without an id= keyword.
Also flags each FieldSpec(...) without id=, unless every loader call it is passed to as sample_fields sets auto_id=True. A fourth positional argument counts as id, and *args or **kwargs are taken to hold one. FieldSpec reads a field called id by default, but whether the records have one is not visible in the code, so the rule asks for the field to be named: id="id" when that is it. A function passed as sample_fields builds its own Sample() calls, which are checked as above.
The FieldSpec may reach its loader directly, through either branch of a conditional, or through a name. A name is resolved as Python scopes it: a function's own names, then enclosing functions', then the module's. sample_fields and auto_id may be passed by keyword, or positionally to Inspect's hf_dataset, csv_dataset, json_dataset and file_dataset and inspect_evals' load_csv_dataset and load_json_dataset. A loader's **kwargs is taken to set auto_id. A FieldSpec the rule can't trace to a loader needs its own id=. That covers one never passed to a loader, and one reached only through a container, an attribute such as self.fields, or another file.
auto_id numbers samples by their position in the unshuffled records. Inspect assigns those ids before shuffling, so they survive shuffles, but not filtering or dataset updates; a field from the records is more stable where one exists.
Why is this bad?¶
Without a stable id a sample is identified by its position. Shuffling, --limit, reruns and dataset updates all change positions, so results can no longer be compared sample by sample.
Example¶
Sample(input=record["question"], target=record["answer"])
hf_dataset("org/data", sample_fields=FieldSpec(input="question", target="answer"))
Use instead:
Sample(input=record["question"], target=record["answer"], id=record["id"])
hf_dataset("org/data", sample_fields=FieldSpec(input="question", target="answer", id="qid"))
See also¶
Suppress on a line with # inspect-evals-lint: ignore[IEBP003] -- <reason> or ignore[sample_ids] -- <reason>; select or ignore it in configuration by either, or by the prefix IEBP.