IETS003: e2e_test

Some test runs eval() against a mockllm/ model.

Category: tests · Applies to: eval

What it does

Looks through the test directory for a file that calls eval() or eval_async() (or an alias imported from inspect_ai) and mentions a model under the mockllm/ provider. mockllm/model is the usual name, but any name after the prefix is the same mock, and a test that scripts custom_outputs per case commonly names each one (mockllm/epochs).

Why is this bad?

An end-to-end run against the mock model catches wiring mistakes, a dataset that no longer loads, a solver that does not compose, without spending tokens.

Example

from inspect_ai import eval

def test_e2e():
    logs = eval(my_eval(), model="mockllm/model", limit=2)
    assert logs[0].status == "success"

See also

• Reference: eval()

Suppress on a line with # inspect-evals-lint: ignore[IETS003] -- <reason> or ignore[e2e_test] -- <reason>; select or ignore it in configuration by either, or by the prefix IETS.