IETS003: e2e_test

Some test runs eval() against mockllm/model.

Category: tests · Applies to: eval

What it does

Looks through the test directory for a file that calls eval() or eval_async() (or an alias imported from inspect_ai) and mentions mockllm/model.

Why is this bad?

An end-to-end run against the mock model catches wiring mistakes, a dataset that no longer loads, a solver that does not compose, without spending tokens.

Example

from inspect_ai import eval

def test_e2e():
    logs = eval(my_eval(), model="mockllm/model", limit=2)
    assert logs[0].status == "success"

Suppress on a line with # inspect-evals-lint: ignore[IETS003] or ignore[e2e_test]; select or ignore it in configuration by either, or by the prefix IETS.