Issue 00 · Autumn 2026日本語 · RSS
AgentKoji

Let LLMs Only Pick and Extract

LLM structured outputs fix the shape, not the truth. Don't let the model write prose: have it pick from options and extract quotes, then verify them by machine.

When adding an LLM to an app, people start with “write a summary” or “write a description.” Free text can’t be checked by machine, so plausible errors reach the screen. In our test environment we narrow the LLM’s job to two things, picking from fixed options and extracting from the source, and don’t let it write prose. Here is the reasoning and how we check results.

Prose can’t be verified

Free-text errors read fine. Picking and extracting can be checked: is the answer one of the options? Is the quote literally in the source? Where a screen needs prose, insert extracted phrases or chosen labels into human-written templates.

The official advice: quote first

Anthropic’s docs suggest extracting word-for-word quotes before working on long documents, having Claude find a supporting quote for each claim and retract claims without one, and explicitly allowing “I don’t know” (https://platform.claude.com/docs/en/test-and-evaluate/strengthen-guardrails/reduce-hallucinations, checked 2026-10-16). Check that the quotes really exist in the source by machine, not by asking the model.

Right type, wrong answer

Structured outputs use constrained decoding to guarantee schema-compliant responses: no JSON parse errors, guaranteed field types and required fields (https://platform.claude.com/docs/en/build-with-claude/structured-outputs, checked 2026-10-16). That guarantees shape only. An enum can block invalid labels, but a paraphrase is still a valid string.

We once gave a typed-judgment model a data-matching task. A large share of its answers were false positives, all with valid types; the evidence simply wasn’t in the input. Let a model judge only when the evidence is in its input; when string matching decides it, matching is stronger.

Checking extracts by machine

Locally, with a dummy source and outputs we pretended the LLM returned (no API calls):

OK 開封済みの商品は対象外です。 / -
NG 開封した商品は返品できません。 / not in source
NG 送料は お客様の負担となります。 / not in source
NG 返品は購入から14日以内に受け付けます。 / label not in options: 期限切れ

Line 2 is a paraphrase; line 3 has one stray space. Both are easy for humans to miss and fail a substring check reliably. We normalize with NFKC only. Put the check right before insertion; on failure, retry with a cap, else leave blank and log it. Track the failure rate: a sudden rise signals a model update or a change in source format.

Ask “none of these” separately

The docs recommend allowing “I don’t know,” yet in an earlier measurement, a “none of these” option mixed into the choices was never picked across 200 items (no API calls were made for this article). Ask first whether anything applies, then have the model choose only when it says yes.

Keep premises out of questions

Questions with built-in premises break on inputs where the premise fails: ask “why did this increase?” and you get reasons even when nothing increased. In that earlier measurement, repeated runs barely varied; if outputs seem to wobble, suspect the question or input first.

Decide with functions what functions can decide

Anything decided by string matching or numeric comparison belongs in plain functions with unit tests. Leave the LLM only what functions can’t express.

“Looks fine, but the contents are wrong” also happens in monitoring: Pitfalls of Always-On Bot Monitoring. If you want to run light pick-and-extract jobs on a local model, see Why Claude Code Is Slow on Local LLMs.

Summary