Rule index

Every rule has a code (IEFS, IECQ, IETS, IEBP, IESC prefixes for the categories below) and a name; either is accepted by --select, --ignore and in suppression comments, and a prefix selects a whole category. A rule reports one diagnostic per site it finds something wrong at, each with a file and, where the finding is in a file's contents, a line and column. A rule with nothing to point at reports pass, or skip with the reason. Diagnostics are fail or warn; only fail makes the run exit non-zero. inspect-evals-lint --explain <code> prints the same text as a rule's page.

File structure

Code	Rule	Applies to	Summary
IEFS001	package_location	eval, helper	The package exists at // with an init.py
IEFS002	main_file	eval	Some module defines a @task function, preferably .py or tasks.py
IEFS003	init_exports	eval	init.py exports every @task function in the package
IEFS004	registry	eval	The evaluation is registered so inspect eval can find its tasks
IEFS005	eval_yaml	eval	eval.yaml exists, is a mapping, and defines the required fields
IEFS006	readme	eval	README.md exists and has no TODO markers
IEFS007	readme_task_args	eval	Every -T argument in a README command names a parameter of the task it runs
IEFS008	readme_dependency_groups	eval	Every extra and dependency group a README command installs exists in the governing pyproject.toml

Code quality

Code	Rule	Applies to	Summary
IECQ001	private_api_imports	eval, helper	No imports from private inspect_ai modules
IECQ002	score_constants	eval, helper	Score() values use the CORRECT/INCORRECT constants, not string literals
IECQ003	unscored_reason	eval, helper	Score.unscored() passes a reason= and the legacy unscored_reason metadata key is gone
IECQ004	external_dependencies	eval, helper	Third-party imports are declared in pyproject.toml
IECQ005	suppression_syntax	eval, helper	Every suppression marker is one the linter reads, and says why
IECQ006	shared_code_extraction	eval, helper	Code is extracted from a completion with inspect_evals' extract_code_block, not a hand-rolled fence pattern
IECQ007	ruff_rules_enabled	eval, helper	The ruff configuration enables the curated ruff rules
IECQ008	fence_tolerant_json	eval, helper	A generated reply is passed through inspect_evals' strip_json_fence before strict JSON parsing

Tests

Code	Rule	Applies to	Summary
IETS001	tests_exist	eval	A test directory exists for the evaluation
IETS002	tests_init	eval, helper	The test directory and its sub-directories contain init.py
IETS003	e2e_test	eval	Some test runs eval() against a mockllm/ model
IETS004	record_to_sample_test	eval	record_to_sample is exercised by a test when the evaluation uses it
IETS005	custom_solver_tests	eval, helper	Every @solver or @agent function name appears somewhere in the tests
IETS006	custom_scorer_tests	eval, helper	Every @scorer function name appears somewhere in the tests
IETS007	custom_tool_tests	eval, helper	Every @tool function name appears somewhere in the tests
IETS008	custom_metric_tests	eval, helper	Every @metric function is tested through the real epoch reducer

Best practices

Code	Rule	Applies to	Summary
IEBP001	get_model_location	eval, helper	get_model() is only called inside @solver, @scorer or @agent functions
IEBP002	model_role_resolution	eval, helper	get_model(role=...) resolves deliberately: an explicit model, default= or required=True
IEBP003	sample_ids	eval, helper	Every Sample() passes id=, and every FieldSpec() names an id field
IEBP004	task_overridable_defaults	eval, helper	@task parameters naming a solver, scorer, metric, grader or model have defaults
IEBP005	sandbox_image_pinning	eval, helper	Registry images in compose files use an immutable tag or digest
IEBP006	gpu_sandbox_check	eval	An evaluation requiring a GPU ships a maintenance sandbox check task
IEBP007	dockerfile_locking	eval	Dockerfile builds consume locked inputs: committed locks, digest-pinned images, fixed sources
IEBP008	duplicate_filter_acknowledged	eval, helper	Every filter_duplicate_ids() call states how many duplicates it drops and links the upstream report
IEBP009	known_broken_reported	eval, helper	Every drop_known_broken() entry maps a sample id to the URL of its upstream report
IEBP010	shuffle_choices_seeded	eval, helper	shuffle_choices is a seed or off, never an unseeded True
IEBP011	shuffle_seeded	eval, helper	A dataset loader that shuffles samples is given a seed
IEBP012	scorer_failure_scored	eval, helper	A scorer does not turn a grader or sandbox failure into a score
IEBP013	grader_model_under_test	eval, helper	Choose a grader instead of defaulting to the model under evaluation
IEBP014	metric_epoch_safety	eval, helper	Custom metrics read scores as the epoch reducer leaves them and do not modify them
IEBP015	verdict_parse_anchored	eval, helper	A judge's verdict is read from an anchored label, not by substring or an unanchored one-token regex

Security

Code	Rule	Applies to	Summary
IESC001	sandbox_privileges	eval, helper	Compose services do not grant additional sandbox privileges or host access
IESC002	host_code_execution	eval, helper	Model output, tool arguments and sandbox content are not executed or deserialised on the host

Helper packages

Directories listed in helper-dirs (by default utils) hold code that evaluations import rather than an evaluation. They run the rules about behaviour and not the ones about an evaluation's structure and registration. Run: package_location, private_api_imports, score_constants, unscored_reason, external_dependencies, suppression_syntax, shared_code_extraction, ruff_rules_enabled, fence_tolerant_json, tests_init, custom_solver_tests, custom_scorer_tests, custom_tool_tests, custom_metric_tests, get_model_location, model_role_resolution, sample_ids, task_overridable_defaults, sandbox_image_pinning, duplicate_filter_acknowledged, known_broken_reported, shuffle_choices_seeded, shuffle_seeded, scorer_failure_scored, grader_model_under_test, metric_epoch_safety, verdict_parse_anchored, sandbox_privileges, host_code_execution. Not run: main_file, init_exports, registry, eval_yaml, readme, readme_task_args, readme_dependency_groups, tests_exist, e2e_test, record_to_sample_test, gpu_sandbox_check, dockerfile_locking. Every rule declares its scopes in its @rule decorator, so a new rule decides up front whether shared code is in scope.

Categories

Each rule belongs to one category. The JSON output and registry.CATEGORIES expose this set; changing it requires a JSON schema version update and coordinated consumer changes. Schema version 2 adds security. Existing rule codes and categories are unchanged.

• File structure covers package layout, registration, and metadata.
• Code quality covers Python API conventions and dependency declarations.
• Tests covers the presence of tests for evaluation components and execution.
• Best practices covers evaluation configuration, dataset handling, and reproducibility. Image pinning and dependency locking remain here because their primary check is whether inputs are fixed.
• Security covers isolation and access to the host, other containers, or execution capabilities. Runtime-user and build-isolation checks would also belong here as separate rules.

Suppression

• Line: # inspect-evals-lint: ignore[<rule>] -- <reason> on the offending line, or on any line of a multi-line statement; names, codes and code prefixes, comma-separated. In a Dockerfile, on the line above the instruction. In a Compose file, on the setting's line or the list item's line.
• File: # inspect-evals-lint: ignore-file[<rule>] -- <reason> within the first ten lines.
• Paths: per-file-ignores = { "<glob>" = ["<rule>"] } in [tool.inspect-evals-lint].
• Never read: exclude = ["<glob>"] keeps files out of the AST-based rules entirely, for code shipped into a sandbox.

Every comment gives a reason after --, saying why the finding is acceptable; suppression_syntax warns about a comment without one, and the suppression still applies. Suppressed findings still appear in reports, marked [suppressed], and count as passing. A marker the linter does not read (the former noautolint comments and .noautolint files, ignore without a rule list, a selector naming no rule) suppresses nothing and is reported by suppression_syntax with the replacement in its hint.