IEBP002: model_role_resolution

get_model(role=...) resolves deliberately: an explicit model, default= or required=True.

Category: best_practices · Applies to: eval, helper · Allowlist: [tool.inspect-evals-lint.allowlists.model_role_resolution]

What it does

Flags each get_model(role=...) call that passes no explicit model, no default= and no required=True. A literal default=None, model=None or required=False changes nothing at runtime and does not count. Each diagnostic is keyed by the role name (<dynamic> for a non-literal), which is what an allowlist entry names.

Why is this bad?

A role that is not bound at invocation falls back to the model under evaluation. A grader then grades the model's own output, and the scores still look plausible. Pinning a default or requiring the role makes the fallback a choice rather than an accident.

Example

grader = get_model(role="grader")

Use instead:

grader = get_model(role="grader", default="openai/gpt-4o")
# or
grader = get_model(role="grader", required=True)

Options

• allowlists.model_role_resolution: { package = ["role"] } entries reported as warnings while an existing surface is burned down.

Suppress on a line with # inspect-evals-lint: ignore[IEBP002] or ignore[model_role_resolution]; select or ignore it in configuration by either, or by the prefix IEBP.