Skip to content

IEBP015: verdict_parse_anchored

A judge's verdict is read from an anchored label, not by substring or an unanchored one-token regex.

Category: best_practices ยท Applies to: eval, helper

What it does

Flags two ways of reading a verdict that match it anywhere in the text, and warns on a third:

  1. A verdict word tested by containment: "yes" in response, "no" not in reply.lower(), or the word after a label, as in "answer: yes" in text. The words are a fixed vocabulary: yes, no, y, n, true, false, correct, incorrect, pass and fail, in any case. A test against a literal tuple, list, set or dictionary, or against the words or lines of a split string, is a membership check, not a search, and passes.
  2. re.search, re.findall, re.finditer or re.compile whose pattern is one of those words, an alternation of them such as (yes|no), or a one-character choice such as [01] or (A|B|C), with at most a label such as correct: before it, and no anchor. An anchor is ^, $, \A or \Z, which pins the match to a line or to the string. A word boundary (\b) or a lookaround is not one: it stops a match inside a word, but still finds the token anywhere in the reasoning. re.match and re.fullmatch read from the start of the string, which is the leading token a verdict-first prompt asks for, and are not flagged. A compiled pattern is reported once, where it is compiled, including in a class body.
  3. As a warning, a search over the whole output for a labelled number such as Ratio:\s*(\d+), with no anchor.

Text read from state.output, directly or through a name bound only to it, is the model under test's, whose format may be part of what the task measures, and is left alone in all three shapes. Only re called by that name is read; an alias or the regex module is not. Test files are not read.

Why is this bad?

A judge is asked to reason and then give a verdict, and its reasoning uses the verdict words as ordinary words. "yes" in reply matches "yesterday"; re.search(r"y", reply) matches the first "y" in any sentence; re.search(r"correct: (yes|no)", reply) matches the tail of incorrect: yes and inverts the verdict; re.search(r"[01]", reply) reads the "1" in "Sentence 1". Each one turns a right grade into a wrong one on ordinary judge prose, in either direction, without any error.

A word boundary does not repair this. re.findall(r"\b(yes|no)\b", reply) no longer matches "yesterday", but it still collects every "yes" and "no" the reasoning uses as ordinary words, and whichever one the code takes may not be the verdict. In a study of nine judge models from 2023 to 2026, each grading the same 435 answers, parsers that required the verdict on a labelled line disagreed between the first and the last match on at most 0.2% of replies. Parsers that searched for a bare verdict word anywhere disagreed on 17% to 33%, because the reasoning named both words.

The fix is to read the verdict from the label the prompt names, anchored to the start of a line (allowing emphasis around the label and a list marker before it), at the position the prompt asks for: the leading token for a verdict-first prompt, the last labelled line for a reason-then-verdict prompt. When nothing matches, that is a grader parse failure, not a verdict. A judge that returns several fields is better asked for a structured reply, through a tool call with a schema.

A deterministic checker, such as a test runner, can quote the model's own output back in an error message. A search for Ratio:\s*(\d+) over its whole output then reads a number the model printed. Read the checker's exit code first, take the score only from an accepted run, and anchor the label to the line the checker writes it on.

Where a parser reproduces the upstream benchmark's own reading on purpose, suppress the finding with a reason that says so.

Example

async def score(state: TaskState, target: Target) -> Score:
    reply = (await grader.generate(prompt)).completion
    match = re.search(r"correct: (yes|no)", reply)
    verdict = match.group(1) if match else "no"
    ...

Use instead:

VERDICT = re.compile(r"^[ \t]*(?:[-*][ \t]+)?\**correct\**:\s*\**(yes|no)\b", re.I | re.M)


async def score(state: TaskState, target: Target) -> Score:
    reply = (await grader.generate(prompt)).completion
    matches = VERDICT.findall(reply)
    if not matches:
        return Score.unscored(
            reason="grader_failed", metadata={"grader_failure_mode": "grader_parse_failure"}
        )
    verdict = matches[-1].lower()
    ...

See also

Suppress on a line with # inspect-evals-lint: ignore[IEBP015] -- <reason> or ignore[verdict_parse_anchored] -- <reason>; select or ignore it in configuration by either, or by the prefix IEBP.