IEBP009: known_broken_reported

Every drop_known_broken() entry maps a sample id to the URL of its upstream report.

Category: best_practices · Applies to: eval, helper

What it does

Flags each drop_known_broken(...) call without a broken= keyword, and each entry of its broken dict whose value is a literal string with no http:// or https:// URL. The dict may be written inline or bound to a module-level name in the same file; a name the file does not define, a non-literal value and **kwargs are taken at face value. Entry diagnostics point at the entry's line so a suppression can sit beside it.

Why is this bad?

A hard-coded exclusion list is a workaround for a dataset defect. Without the report beside each id, nobody upstream knows about the defect, a reviewer cannot check the claim, and the entry outlives the fix. The URL is the evidence and the reminder.

Example

KNOWN_BROKEN = {"ruin_names_100": "options split on commas"}

Use instead:

KNOWN_BROKEN = {"ruin_names_100": "https://github.com/org/repo/issues/19"}

See also

• Datasets: Dataset Samples

Suppress on a line with # inspect-evals-lint: ignore[IEBP009] or ignore[known_broken_reported]; select or ignore it in configuration by either, or by the prefix IEBP.