Forcing Structured Output: JSON, Schemas, and Validation

The most common production bug in LLM applications is not a wrong answer. It is a right answer in the wrong shape. The model wrote a perfectly good summary and then wrapped it in three sentences of preamble, used smart quotes, and closed the JSON with a trailing comma. Everything downstream breaks, and the failure looks like a parser bug rather than a prompt problem. The fix is to stop asking nicely and start specifying a contract.

An output contract is four things

• A named shape: the exact keys, types, and nesting you will parse.
• A completeness rule: what to emit when a field has no value, and what is simply omitted.
• A constraint on the content: enumerations closed, strings bounded in length, numbers in range.
• A rejection rule: what the model must do when the input genuinely does not fit the schema.

Most teams write the first and stop. The third and fourth are where production breaks. A field with no value is the classic one: tell the model to emit null, and tell it never to invent a placeholder, because a plausible invented value is far more expensive to detect later than a visibly missing one.

A contract that survives contact with real input

{
  "task": "classify_support_ticket",
  "schema_version": 3,
  "result": {
    "category": {
      "type": "string",
      "enum": ["refund_request", "payment_failed", "plan_change", "other"],
      "description": "Exactly one. Use 'other' when none fit."
    },
    "confidence": {
      "type": "string",
      "enum": ["high", "medium", "low"]
    },
    "evidence": {
      "type": "string",
      "max_length": 200,
      "description": "The exact words in the input that drove the label. Copy them; do not paraphrase."
    },
    "needs_human": {
      "type": "boolean",
      "description": "true when confidence is 'low' or the user disputes a charge."
    }
  },
  "output_rules": [
    "Return JSON only. No prose, no code fences, no commentary.",
    "Every field is always present. Use null when a value is unknown.",
    "Never invent an enum value that is not listed."
  ]
}

Three details in that schema do the heavy lifting. Enums close the output space, which is the difference between a field you can trust and a free-text field you have to parse defensively. The max length on evidence is a guardrail that keeps one verbose response from eating a budget. And the evidence field forces the model to justify the label using text copied from the input, which is a cheap defense against confident fabrication.

Prefer constraints over requests

Where the platform offers schema-constrained or grammar-constrained decoding, turn it on. It is strictly better than asking for JSON in the instruction, because the model is prevented from producing anything outside the grammar at generation time instead of being asked politely not to. The instruction still matters for the field descriptions and the judgment calls, but the format itself stops being a compliance problem.

Validate, and write the repair loop

Even with constraints, treat every response as untrusted input. Parse, validate against the schema, and validate the semantics: a category of "refund_request" with empty evidence is schema-valid and useless. When validation fails, do not resample blindly. Send the original output, the specific validation error, and an instruction to fix only that, which turns a retry into a repair.

def extract_structured(model_call, schema, max_repairs=2):
    raw = model_call(prompt, schema=schema)
    for attempt in range(max_repairs + 1):
        try:
            parsed = json.loads(strip_fences(raw))
            schema.validate(parsed)
            assert_semantics(parsed)
            return parsed
        except (ValidationError, AssertionError) as err:
            if attempt == max_repairs:
                raise
            # Repair, don't resample: keep the model's work, fix the defect.
            raw = model_call(
                prompt,
                previous_output=raw,
                validation_error=str(err),
                instruction="Return corrected JSON only. Do not add commentary.",
                schema=schema,
            )
    raise RuntimeError("unreachable")

The difference between repair and resample is measurable. Resampling throws away a response that was probably 95 percent correct and pays full price to get a new one. Repair pays for a short corrective pass. It is also more debuggable, because the validation error becomes part of the record of what went wrong.

Versioning the schema is not optional

A schema is code, and it ages. A field gets renamed, a new enum value appears, a consumer starts depending on a field that was always documented as best-effort. The dangerous case is the silent one: the model starts emitting a value your new enum does not list, or stops populating a field that something downstream quietly treats as guaranteed. Version the schema in the identifier, keep old versions parseable, and never mutate a version in place.

• Bump the version whenever a field is removed, renamed, or its meaning narrows.
• Adding an optional field is usually backward compatible; removing or repurposing one is not.
• Keep readers for at least one version after a bump, and log which version each response used.
• Pin the version in the prompt text so the model cannot be asked to satisfy a schema it never saw.
• Re-run your frozen evaluation on every bump. A schema change is a prompt change and deserves the same gate.

Adding new enum values deserves special care. If you add a category to the allowed set, the model can start using it for cases it previously mapped elsewhere, which quietly moves volume between categories. That is a behavioral change even though the schema is still valid, and it will not show up in a validation check. Compare per-category volumes before and after, not just the pass rate.

Nested structures and arrays

Nesting is where hand-written output contracts get interesting. Deep structures invite the model to invent empty containers, and array lengths are rarely what you intended. Constrain both explicitly: give each array a purpose, say whether an empty array is acceptable, and state what an element must contain. If a field is a list of strings, say whether duplicates are allowed, because a model asked for "the key points" will happily return the same point three times.

{
  "result": {
    "claims": {
      "type": "array",
      "description": "Distinct claims, one per element. Never repeat a claim.",
      "minItems": 1,
      "maxItems": 5,
      "items": {
        "type": "object",
        "properties": {
          "text": { "type": "string", "maxLength": 160 },
          "source_id": {
            "type": "string",
            "description": "Must match an id present in the provided context."
          }
        },
        "required": ["text", "source_id"]
      }
    },
    "unresolved": {
      "type": "array",
      "description": "Questions the context does not answer. Use an empty array, never a placeholder string.",
      "maxItems": 3,
      "items": { "type": "string" }
    }
  }
}

Requiring a source_id that must exist in the provided context is the cheapest factuality control available in a structured output. It is not perfect, since the model can attach a valid id to an unsupported claim, but it makes casual fabrication fail far more often, and the failures are visible in review because the id either exists or does not.

Model-specific escape hatches, in one place

Every major provider now offers some form of structured output support, and the interfaces drift. The pattern is the same everywhere, so learn it once and then check the provider reference for the exact field names before you ship. What matters is that you use the native mode when it exists rather than hoping prose instructions hold.

Two things remain outside the reach of every mode above. Values can be syntactically and structurally valid and still be wrong, and a model can satisfy a schema by writing an empty object where it could not answer. A schema narrows what the model can say; it does not narrow what the model knows. Validation plus explicit "use an empty array, never a placeholder" instructions are still your responsibility.

A prompt that asks for JSON is a request. A schema, a validator, and a repair loop are a contract. Only one of them survives production traffic.

The shapes that break parsers

• Code fences. Models wrap JSON in markdown even when told not to, so strip fences before parsing or forbid them in the grammar.
• Trailing commas, single quotes, and unescaped newlines inside strings: all legal prose, all invalid JSON.
• Preamble and postamble, which survive most instructions and break strict parsers. Constrained decoding removes the class entirely.
• Multiple objects when you asked for one, often a model offering "here it is, and a simpler version." Reject anything that parses as more than one document.
• Unicode lookalikes: smart quotes and non-breaking spaces are the same glyph to a human and different bytes to a parser.