Rule index

Every rule has a code (IEFS, IECQ, IETS, IEBP prefixes for the four categories below) and a name; either is accepted by --select, --ignore and in suppression comments, and a prefix selects a whole category. A rule reports one diagnostic per site it finds something wrong at, each with a file and, where the finding is in a file's contents, a line and column. A rule with nothing to point at reports pass, or skip with the reason. Diagnostics are fail or warn; only fail makes the run exit non-zero. inspect-evals-lint --explain <code> prints the same text as a rule's page.

File structure

Code	Rule	Applies to	Summary
IEFS001	package_location	eval, helper	The package exists at // with an init.py
IEFS002	main_file	eval	.py or tasks.py exists and defines at least one @task function
IEFS003	init_exports	eval	init.py exports every @task function from the main file
IEFS004	registry	eval	The evaluation is registered so inspect eval can find its tasks
IEFS005	eval_yaml	eval	eval.yaml exists, is a mapping, and defines the required fields
IEFS006	readme	eval	README.md exists and has no TODO markers

Code quality

Code	Rule	Applies to	Summary
IECQ001	private_api_imports	eval, helper	No imports from private inspect_ai modules
IECQ002	score_constants	eval, helper	Score() values use the CORRECT/INCORRECT constants, not string literals
IECQ003	unscored_reason	eval, helper	Score.unscored() passes a reason= and the legacy unscored_reason metadata key is gone
IECQ004	external_dependencies	eval, helper	Third-party imports are declared in pyproject.toml

Tests

Code	Rule	Applies to	Summary
IETS001	tests_exist	eval	A test directory exists for the evaluation
IETS002	tests_init	eval, helper	The test directory and its sub-directories contain init.py
IETS003	e2e_test	eval	Some test runs eval() against mockllm/model
IETS004	record_to_sample_test	eval	record_to_sample is exercised by a test when the evaluation uses it
IETS005	custom_solver_tests	eval, helper	Every @solver function name appears somewhere in the tests
IETS006	custom_scorer_tests	eval, helper	Every @scorer function name appears somewhere in the tests
IETS007	custom_tool_tests	eval, helper	Every @tool function name appears somewhere in the tests

Best practices

Code	Rule	Applies to	Summary
IEBP001	get_model_location	eval, helper	get_model() is only called inside @solver or @scorer functions
IEBP002	model_role_resolution	eval, helper	get_model(role=...) resolves deliberately: an explicit model, default= or required=True
IEBP003	sample_ids	eval, helper	Every Sample() passes id=
IEBP004	task_overridable_defaults	eval, helper	@task parameters naming a solver, scorer, metric, grader or model have defaults
IEBP005	sandbox_image_pinning	eval, helper	Registry images in compose files use an immutable tag or digest
IEBP006	gpu_sandbox_check	eval	An evaluation requiring a GPU ships a maintenance sandbox check task
IEBP007	dockerfile_locking	eval	Dockerfile builds consume locked inputs: committed locks, digest-pinned images, fixed sources

Helper packages

Directories listed in helper-dirs (by default utils) hold code that evaluations import rather than an evaluation. They run the rules about behaviour and not the ones about an evaluation's structure and registration. Run: package_location, private_api_imports, score_constants, unscored_reason, external_dependencies, tests_init, custom_solver_tests, custom_scorer_tests, custom_tool_tests, get_model_location, model_role_resolution, sample_ids, task_overridable_defaults, sandbox_image_pinning. Not run: main_file, init_exports, registry, eval_yaml, readme, tests_exist, e2e_test, record_to_sample_test, gpu_sandbox_check, dockerfile_locking. Every rule declares its scopes in its @rule decorator, so a new rule decides up front whether shared code is in scope.

Categories

Each rule belongs to one of exactly four categories, the sections above. The JSON output and registry.CATEGORIES expose them, and downstream tooling (badges, the register lint service) is built around that fixed set. A new rule joins one of the four; adding a category would be a breaking change.

Suppression

• Line: # inspect-evals-lint: ignore[<rule>] on the offending line, or on any line of a multi-line statement; names, codes and code prefixes, comma-separated. ignore without a bracketed list is a configuration error. In a Dockerfile, on the line above the instruction.
• File: # inspect-evals-lint: ignore-file[<rule>] within the first ten lines.
• Paths: per-file-ignores = { "<glob>" = ["<rule>"] } in [tool.inspect-evals-lint].
• Never read: exclude = ["<glob>"] keeps files out of the AST-based rules entirely, for code shipped into a sandbox.

Suppressed findings still appear in reports, marked [suppressed], and count as passing. The former noautolint comments and .noautolint files are rejected with a message naming the replacement.