IEFS002: main_file

Some module defines a @task function, preferably .py or tasks.py.

Category: file_structure · Applies to: eval

What it does

Reads every Python file in the package for @task functions. Passes when <name>.py or tasks.py defines one, preferring whichever does so a stray empty <name>.py does not hide the tasks in tasks.py. Warns, once per module, when the tasks live only in other modules. Fails when no module defines a task, or when the conventional file exists but does not parse.

Why is this bad?

A package with no task is not an evaluation, whatever else it contains. Keeping the tasks in a predictably named module is a lesser matter, so it is a warning: it lets readers find them without opening every file, but a package that names its task module after the benchmark still runs.

Example

src/my_eval/my_eval.py     # defines @task my_eval()
src/my_eval/tasks.py       # accepted alternative
src/my_eval/v8.py          # defines the tasks: warned, not failed

See also

• Tasks: Task Basics
• Extensions: Components: Registration

Suppress on a line with # inspect-evals-lint: ignore[IEFS002] -- <reason> or ignore[main_file] -- <reason>; select or ignore it in configuration by either, or by the prefix IEFS.