IEBP013: grader_model_under_test¶

Choose a grader instead of defaulting to the model under evaluation.

Category: best_practices · Applies to: eval, helper

What it does¶

Reports an error when scorer code uses get_model() to select the model under evaluation and then calls .generate(...) on it.

The rule checks @scorer functions, their nested functions, and the package functions they call. This is the same scope as scorer_failure_scored (IEBP012).

It reports these forms when no role= or non-None default= is supplied:

Calls with no model argument or a literal None, such as get_model() or get_model(None).
Calls that select the evaluated model through state.model, such as get_model(state.model), get_model(str(state.model)), or get_model(state.model.name).
Calls that pass an optional function argument whose default is None, such as get_model(model) inside a function defined with model=None. This includes arguments used by a nested scoring function. If the caller leaves the argument unset, get_model(None) returns the evaluated model.

The rule only reports a call if its result is used for .generate(...). Reading get_model().name to choose a tokenizer, for example, is allowed. Functions named get_model that are defined by the package itself are also excluded.

Calls with role= are checked by model_role_resolution (IEBP002), which requires default= or required=True unless allowlisted.

Why is this bad?¶

Without a separate grader, the model under evaluation grades its own answers. This can inflate scores and make them unsuitable for comparison with results from an independent grader.

How to fix it¶

Use get_model(role="grader", default=...). Set the default to the grader used by the original benchmark's authors. A model name string is accepted; prefer a Model that also preserves the generation settings specified in their paper or code.

If the original benchmark grades without a model and your implementation adds one, choose a default grader and document that choice in the README. Changing the default can affect scores, so bump the task version to indicate that scores may no longer be comparable.

Alternatively, use get_model(role="grader", required=True) to require the user to configure a grader. If the role is missing, the error occurs when get_model runs. Inside score(), that is the first scored sample.

When using the evaluated model is intentional¶

If the rule reports an intentional use, suppress it with # inspect-evals-lint: ignore[grader_model_under_test] -- <reason> and explain the choice in the README. Examples include:

A scoring-only task uses --model to select the grader for another run's output.
A benchmark includes a self-judging defence.
A shared helper deliberately falls back to the evaluated model when neither a model argument nor a model role is supplied.

How model use is tracked¶

The rule recognises get_model().generate(...) and calls through an assigned variable, such as grader = get_model() followed by grader.generate(...). It also recognises fallback assignments such as grader = grader_model or get_model().

Within a function, tracking starts at the assignment and stops when the variable is assigned again. Assignments in other branches of the same if do not stop tracking.

Nested functions are checked if the outer function does not assign the variable again. A nested function is excluded if it defines its own variable with that name, either as a parameter or by assignment.

Known limits¶

The rule can miss self-grading in these cases:

The model is selected in a task body, a class's __init__, or a classmethod. The rule does not inspect those locations.
The model is passed to another function, for example model_graded_qa(model=get_model()).
The model or its generate method is assigned to another variable, for example grader = m or gen = m.generate. The rule also misses model names assigned separately, such as name = str(state.model) followed by get_model(name).
The result is stored using a walrus operator, an attribute such as self.grader, or tuple unpacking such as grader, n = get_model(), 3.
get_model is imported under another name.
A nested function uses a model variable that the outer function later assigns again.
A model_graded_* scorer uses its default grader role. Neither this rule nor IEBP002 checks that implicit choice.

The parameter check also has a limitation: any enclosing if or conditional expression that mentions the parameter prevents a report, regardless of which branch contains the call. The rule does not verify that the condition prevents None from reaching get_model.

Apart from the handling of if branches described above, tracking follows source order and does not determine which statements run.

Example¶

@scorer(metrics=[accuracy()])
def equivalence():
    async def score(state, target):
        result = await get_model().generate(prompt)
        ...

Use instead:

UPSTREAM_GRADER = "openai/gpt-4o-2024-08-06"  # the grader the benchmark's paper used

@scorer(metrics=[accuracy()])
def equivalence(model: str | Model | None = None):
    async def score(state, target):
        grader = get_model(model, role="grader", default=UPSTREAM_GRADER)
        result = await grader.generate(prompt)
        ...