IETS007: custom_tool_tests¶
Every @tool function name appears somewhere in the tests.
Category: tests ยท Applies to: eval, helper
What it does¶
Finds functions decorated with @tool in the package and checks each name appears in a test file. A tool registered under another name with @tool(name="submit") is also satisfied by a mention of that name, since that is what the tests and the transcript call it. For an evaluation the search covers tests/<name>/; for a helper package, the whole tests root, because shared components are usually tested next to the evaluation that motivated them. One diagnostic per untested function. This is a presence check, not a quality check.
Why is this bad?¶
A custom tool is the evaluation's own logic, the part no upstream test covers. One test that at least constructs it catches import errors and signature changes.
See also¶
Suppress on a line with # inspect-evals-lint: ignore[IETS007] -- <reason> or ignore[custom_tool_tests] -- <reason>; select or ignore it in configuration by either, or by the prefix IETS.