Skip to content

IEFS005: eval_yaml

eval.yaml exists, is a mapping, and defines the required fields.

Category: file_structure ยท Applies to: eval

What it does

Parses eval.yaml in the package and reports one diagnostic per required field that is missing, or one for a file that is not valid YAML or not a mapping. With eval-yaml-required = false a missing file is a skip, for repositories whose metadata lives in the inspect_evals register; a present file is still validated.

Why is this bad?

eval.yaml is what listings, the register and the README generator read. A missing field there is a missing field everywhere downstream.

Example

title: GPQA
description: Graduate-level science questions.
group: Knowledge
contributors: [someone]
tasks:
  - name: gpqa_diamond

Options

  • eval-yaml-required
  • eval-yaml-required-fields

Suppress on a line with # inspect-evals-lint: ignore[IEFS005] or ignore[eval_yaml]; select or ignore it in configuration by either, or by the prefix IEFS.