IECQ008: fence_tolerant_json

A generated reply is passed through inspect_evals' strip_json_fence before strict JSON parsing.

Category: code_quality · Applies to: eval, helper

What it does

Flags json.loads, a pydantic model's model_validate_json or a TypeAdapter's validate_json called on the completion of a generate call made in the same function: either the attribute itself, or a name bound only to it. The generate result may be bound to a name or used inline. One diagnostic per call. A completion that reaches the parse through any other call, such as strip_json_fence or a project's own extraction, is left alone. So is the output of the model under test, read from state.output, whose format may be part of what the task measures. Test files are not read.

The rule applies only where the helper is available: in inspect_evals itself, where a helper package (helper-dirs) defines strip_json_fence, and in a repository whose pyproject.toml declares a dependency on inspect-evals. Elsewhere it is skipped.

Why is this bad?

A model asked for bare JSON, even through a response_schema, sometimes wraps it in a Markdown code fence. Some providers accept a response_schema without enforcing it. On OpenRouter, the Mistral and Fireworks hosts are listed as supporting structured outputs for GLM 5.3, yet in October 2026 they fenced 25.5% and 16.5% of its replies to a response_schema, while DeepInfra, BaseTen and Baidu fenced none. Unless the provider is pinned, OpenRouter picks the host per request, so the same model and config can return bare JSON on one call and fenced JSON on the next. A strict parser rejects the fenced reply, so a correct grade is thrown away and the sample is scored as a grader failure, or as whatever the fallback is. strip_json_fence(text) returns the JSON inside one fence, labelled json or unlabelled, that wraps the whole reply, and returns any other text unchanged, so a reply with prose around the JSON still fails as before.

Pin an OpenRouter grader to hosts that enforce the schema, through its provider model argument (for example {"only": ["deepinfra"]}). Strip the fence as well, since a user can swap in a grader the pin does not cover.

Stripping the fence can turn a failed parse into a grade, so scores can move, and a fix needs a comparability bump to the task version and a changelog entry. Suppress the finding, with a reason that names the rule, only where the upstream benchmark rejects fenced replies too.

Example

async def score(state: TaskState, target: Target) -> Score:
    result = await get_model(role="grader").generate(prompt, config=config)
    verdict = Verdict.model_validate_json(result.completion)
    ...

Use instead:

from inspect_evals.utils.code import strip_json_fence


async def score(state: TaskState, target: Target) -> Score:
    result = await get_model(role="grader").generate(prompt, config=config)
    verdict = Verdict.model_validate_json(strip_json_fence(result.completion))
    ...

Suppress on a line with # inspect-evals-lint: ignore[IECQ008] -- <reason> or ignore[fence_tolerant_json] -- <reason>; select or ignore it in configuration by either, or by the prefix IECQ.