IEBP001: get_model_location

get_model() is only called inside @solver or @scorer functions.

Category: best_practices · Applies to: eval, helper

What it does

Warns on each get_model() call that is not inside a function decorated with @solver or @scorer.

Why is this bad?

Resolving a concrete model at import time or inside @task fixes it before the caller can choose one. Resolving it inside the solver or scorer keeps the task declarative and lets --model-role and task parameters override it.

Example

GRADER = get_model("openai/gpt-4o")   # resolved at import

@scorer(metrics=[accuracy()])
def graded():
    async def score(state, target):
        return await GRADER.generate(...)

Use instead:

@scorer(metrics=[accuracy()])
def graded(model: str | Model | None = None):
    async def score(state, target):
        grader = get_model(model, role="grader", default="openai/gpt-4o")
        ...

Suppress on a line with # inspect-evals-lint: ignore[IEBP001] or ignore[get_model_location]; select or ignore it in configuration by either, or by the prefix IEBP.