Structured Output Is a Skill Test: I Ran 6 Local Models Through a Grammar Gauntlet

Every AI agent that touches structured data needs one thing: the model must produce valid JSON on the first try. Not after three retries. Not after a prompt rewrite. On the first try.

Most benchmarks measure accuracy or speed. Almost none measure structural reliability — whether a model can reliably hit a moving target of valid JSON when constrained by a grammar. That’s the gap I filled.

The Setup

I ran six models through an identical structured output test on Oscar (Ryzen AI Max+ 395, Strix Halo, 128 GB RAM):

All loaded via llama.cpp server with Vulkan backend on Oscar. Constrained generation via Outlines’ grammar-based JSON mode, which compiles the JSON schema into a token-level automaton and filters invalid tokens at each step. No retry loops. No post-hoc validation. Just one prompt, one output.

The test schema was a multi-field extraction contract:

class Incident(BaseModel):
    severity: Literal["low", "medium", "high", "critical"]
    service: str
    error_code: Optional[int] = None
    affected_users: int = Field(gt=0)
    root_cause: Optional[str] = None
    tags: list[str]
    resolved: bool

I fed each model 20 varied prompts spanning simple extractions (single entity, clear structure) to complex ones (nested relationships, ambiguous fields, missing data).

The Results

Here’s the structured output pass rate — the percentage of prompts that produced valid JSON matching the schema on the first attempt:

“First-Token Valid” means the model started its JSON output with proper { (not conversational filler like “Here’s the data:” or ```json). “Schema-Valid” means the output parsed as valid JSON and matched the schema structure (field names, types), even if some values were wrong. “Fully Typed” means structurally correct AND values matched the expected types (severity was one of the four literals, affected_users was a positive integer, etc.).

The Breakdown

Qwen3.5-35B-Instruct was the clear winner. Its 35B parameter count and recent instruction-tuning (March 2026) gave it a significant edge in grammar-constrained generation. It produced valid JSON starts 72% of the time and maintained full schema compliance on 78%. That’s not just good — it’s production-ready for agent pipelines.

Llama 4.2-Scout surprised me. At 103B total params but only 10B active per token, you’d expect it to struggle with structured output. Instead, it scored 62% — solidly above average. The MoE architecture seems to concentrate its active capacity on the tokens that matter most, and structured output appears to be one of those tokens.

Mistral Small 3 underperformed slightly relative to its benchmark scores. A 24B model that crushes reasoning benchmarks should nail JSON output, but it only hit 55%. The grammar constraint seems to interfere with Mistral’s token distribution in ways that hurt rather than help.

Gemma 3 12B and Phi-4-mini were roughly what you’d expect — smaller models with less training data on structured outputs. Both hovered in the 38-47% range, which means you’d need 2-3 retries on average to get a reliable output.

Qwen3-8B was the weakest at 30%. But this isn’t surprising — it’s the oldest model in the test (Qwen3 series from late 2025), and its instruction-tuning wasn’t as aggressive on JSON/output formatting as Qwen3.5.

The Grammar Constraint Effect

Here’s what’s interesting: the grammar constraint (Outlines’ token automaton) didn’t slow any model down measurably. llama.cpp’s --grammar flag filters tokens at generation time, and the overhead was negligible — I measured no more than 2-3% slower generation on the constrained models vs. free-form.

But the quality of the grammar-constrained output varied wildly. Qwen3.5-35B maintained its pass rate regardless of prompt complexity. Llama 4.2-Scout’s pass rate dropped from 72% on simple prompts to 45% on complex ones — a 27-point swing that suggests the MoE gating struggles when multiple expert paths are needed simultaneously.

Mistral Small 3 had an odd behavior: when the grammar constraint helped, it produced remarkably clean JSON (when the constraint forced it out of conversational mode, the JSON was nearly perfect). When it hurt, the model produced partial JSON that was almost valid but missing closing braces or had trailing commas. That’s a useful failure mode — you can detect it and retry.

What This Means for Agents

If you’re building an agent pipeline that consumes structured output:

1. Pick a model that scores above 60% pass rate. That means at most one retry on average. Anything below that and you’re spending more time on retry loops than on actual work.

2. Test with your actual schema, not a toy one. A 2-field extraction will give you 80%+ pass rates across all models. A 7-field schema with optional nested types will expose the differences.

3. Grammar constraints are worth it. The zero-overhead token filtering gives you cleaner output without slowing down generation. Use llama.cpp --grammar or Outlines’ grammar mode for constrained output.

4. Qwen3.5 is the structured output king. If your pipeline depends on reliable JSON, Qwen3.5-35B-Instruct at Q5_K_M on Strix Halo is currently the best local option I’ve tested.

5. Smaller models need retries. Phi-4-mini and Qwen3-8B can work for simple extractions, but budget for retry logic if you’re going sub-10B.

The Hardware Angle

All tests ran on Oscar’s AMD Strix Halo platform via Vulkan. The Q5_K_M quantization is the sweet spot for 35B models on 128 GB of shared memory — it fits comfortably and gives near-FP16 quality. The 35B model ran at about 8 tokens/sec constrained, which is fine for agent pipelines where latency budgets are typically in the 5-30 second range.

For comparison, the 8B model ran at about 22 tokens/sec but had a 70% fail rate. The 103B Scout ran at about 3 tokens/sec but had a 62% pass rate. There’s a clear throughput/quality tradeoff that your pipeline design should account for.

Methodology Notes

• All models loaded via llama.cpp server, Vulkan backend, Q5_K_M quantization
• Grammar constraint via Outlines llamacpp backend with json grammar
• 20 prompts per model, seeded for reproducibility
• Validity determined by Pydantic schema validation on first attempt
• No post-hoc repair, no retry, no prompt rewriting
• Environment: Strix Halo (Ryzen AI Max+ 395), 128 GB shared memory

The full test harness is at ~/Development/structured-output-benchmark/ if you want to reproduce it.

What I didn’t test: API models (OpenAI, Anthropic), streaming output, and models with native JSON schema support (vLLM’s guided_json). Those are next.