IEFS007: readme_task_args¶
Every -T argument in a README command names a parameter of the task it runs.
Category: file_structure ยท Applies to: eval
What it does¶
Reads the inspect eval and inspect eval-set commands in the README's fenced code blocks and inline code, with or without a uv run prefix and across line continuations. Each task the command names is resolved against the package's @task functions: <namespace>/<task>, a bare <task>, or path/to/file.py@task. Every -T name=value must then name a parameter of every resolved task. Fails once per argument and task.
Tasks that resolve to nothing in the package, such as another package's tasks or a placeholder like my_eval_XX, are not checked. Neither is a task that takes **kwargs.
Why is this bad?¶
Inspect drops a -T argument the task does not take and only logs a warning. Readers copy README commands, so one that shows -T max_messages=75 for a task whose parameter is message_limit runs with the default. Nothing else checks hand-written examples: a renamed parameter leaves them behind, even in a repository whose README generator keeps a parameter list up to date.
Example¶
@task
def my_eval(message_limit: int = 50) -> Task: ...
inspect eval my_eval/my_eval -T max_messages=75
Use instead:
inspect eval my_eval/my_eval -T message_limit=75
See also¶
Suppress on a line with # inspect-evals-lint: ignore[IEFS007] -- <reason> or ignore[readme_task_args] -- <reason>; select or ignore it in configuration by either, or by the prefix IEFS.