Implement validation, retry, and feedback loops for extraction quality

Enforcing output with a tool_use JSON schema (see 4.3) buys you exactly one guarantee: the response will be syntactically valid, schema-shaped JSON. Every required field is present, every enum value is legal, every type matches. What it does not buy you is correctness. A model can return perfectly valid JSON where the line items add up to $340 but total says $420, where a shipping address lands in the billing_address field, or where a year is transcribed as 2024 instead of 2025.

These are semantic validation errors, and they are invisible to the schema. The schema only knows shapes and types, not the business relationships between fields. So a production extraction system needs a second layer: application code that runs after the model returns and checks values against domain rules. Do the numbers reconcile? Is this field internally consistent? Does this reference resolve?

The split to memorize: tool use eliminates syntax errors; your validation layer catches semantic errors. Retrying to fix a JSON syntax error is almost never necessary once you use tool use. Retrying to fix a semantic error is exactly what the rest of this lesson is about.

When validation fails on a fixable problem, the remedy is a retry-with-error-feedback loop, not a blind re-run. A blind re-run (same prompt, same document) tends to reproduce the same mistake because nothing about the input changed. Instead, construct a follow-up request that gives the model everything it needs to self-correct: the original source document, the specific extraction it produced, and the specific validation errors stated concretely.

Concreteness matters. "That was wrong, try again" gives the model nothing to act on. "The line items sum to 340.00 but you reported total=420.00; recompute total from the line items or re-read the printed invoice total" points directly at the defect. You are effectively turning your validator's output into a targeted prompt.

messages = [
  {"role": "user", "content": original_document},
  {"role": "assistant", "content": failed_tool_use},
  {"role": "user", "content":
     "Validation failed: line_items sum to 340.00 but total=420.00. "
     "Re-extract so total equals the sum of line_items."}
]

This is a bounded loop. Validate, and if it still fails after a small number of attempts, stop and route to a human rather than looping forever.

This is the single most important judgment call in the task statement: retries only help when the information exists and the model got the transformation wrong. They cannot conjure information that is not in the source.

Retry will succeed for format mismatches and structural output errors: a mis-normalized date, a value placed in the wrong field, line items summed incorrectly, a currency symbol left in a numeric field. The correct answer is present in the document; the model just needs a nudge.

Retry will not succeed when the required field simply is not in the provided document, for example a po_number that only exists on a separate purchase order you never passed in. Re-prompting an absent fact pressures the model to fabricate a value to satisfy the request, which is worse than an honest null. The correct design here is to make the field optional and nullable (see 4.3), let the model return null, and route to a different data source or a human, not to keep retrying. Before you build a retry loop, ask: is this a transformation error or a missing-input error?

You can move validation into the extraction itself by designing the schema so the model surfaces the raw material for a check, rather than only its conclusion. The canonical pattern: alongside the value the document states, ask for the value you can independently compute.

For invoices, extract both stated_total (the number printed on the document) and calculated_total (the model's sum of the line items). Your validator then compares the two; a mismatch is an immediate, cheap signal of a transcription or arithmetic problem, on either the model's side or the document's. Similarly, add a conflict_detected boolean with a detail field so the model can flag when the source itself is internally inconsistent, for example two different totals printed in two places.

{
  "line_items": [ ],
  "calculated_total": 340.00,
  "stated_total": 420.00,
  "conflict_detected": true,
  "conflict_detail": "Line items sum to 340.00; printed total reads 420.00"
}

This converts a silent, hard-to-detect error into structured data you can branch on: reconcile automatically, retry, or escalate. It also distinguishes a model error from a document error, which a plain total field never could.

Validation and retry close the loop on a single extraction; a feedback loop closes the loop on the system over time. The key instrument is a detected_pattern field attached to each finding or extraction, recording which construct in the source triggered the output, for example the specific code pattern that made a reviewer flag a line, or the document structure a value was pulled from.

Why this matters: when a human later dismisses a finding as a false positive, detected_pattern lets you aggregate dismissals by pattern. If 90% of dismissed code-review findings share detected_pattern: "unused-import-in-test-file", you have pinpointed a category to suppress or re-prompt, instead of guessing. Without it, you only know that people dismissed findings, not what they were reacting to.

This is the extraction-quality analogue of the review feedback loops elsewhere in Domain 4: instrument the output with enough provenance that you can do systematic error analysis and improve the prompt, rather than tuning blind. The pattern generalizes, tag each output with the signal that produced it so dismissals become data.