Why Your LLM Returns Broken JSON in Production — And How to Fix It

Without schema enforcement, LLMs return broken JSON 8–15% of the time in production. I know because I shipped two AI SaaS products that call LLMs hundreds of times per job — one in Next.js with Gemini, one in Python with GPT-4o. I used three different levels of LLM structured output across both projects. Here's what each level actually costs in retries, latency, and code complexity — and what I'd replace tomorrow with native structured output.

The Three Levels of LLM Structured Output — and What Each One Actually Guarantees

Prompt engineering, JSON mode, and native structured output are three different reliability tiers — each guarantees a different property, and only the third eliminates parse failures at the token generation level.

The 8–15% production parse failure rate for prompt-only JSON comes from a TokenMix.ai analysis of 2 million API calls without schema enforcement. That's not a theoretical number — it's what happens at scale when the model is free to emit markdown fences, trailing commas, renamed fields, and truncated objects. At Level 1 you're hoping. At Level 2, you're catching syntax. At Level 3, the model physically cannot produce tokens that violate your schema.

• Level 1 — Prompt-only: "Return ONLY valid JSON." Failure rate: 8–15%. What breaks: markdown fences, trailing commas, extra explanation text, renamed fields, truncation mid-JSON.
• Level 2 — JSON Mode: Syntactically valid JSON guaranteed. Schema NOT guaranteed. OpenAI's response_format: {type: 'json_object'}. Schema mismatch rate: 2–5%.
• Level 3 — Native Structured Output: Constrained decoding. The model cannot produce tokens that violate your schema. Failure rate: <0.1%. OpenAI Structured Outputs (strict), Gemini responseSchema, Anthropic tool_use.

What "constrained decoding" actually means

When constrained decoding is active, the model's token vocabulary is masked at each generation step. Only tokens that keep the output valid JSON — and schema-conformant — are available at each position. The model cannot produce a trailing comma because that token is masked at that position. It cannot rename a field because only the declared field names are valid tokens at that point in the output. This is not prompting. It is enforcement at the generation level — happening inside the model's sampling loop, before any token ever reaches your application.

This is why Level 3 drops failures below 0.1%. It's not better prompting. It's a fundamentally different mechanism.

The failure mode that none of the three levels prevents

Schema enforcement guarantees structure. It does not guarantee semantic correctness. A Pydantic model with min_length=20 on an answers list guarantees you get 20 answers. It does not guarantee those answers are accurate, on-topic, or the right length in words. The model can return schema-valid JSON with the wrong values — a field that should be "urgent" returns "normal" because the model collapsed to a safer token. This is the semantic failure mode that none of the three levels solves.

The Next.js Implementation: Belt-and-Suspenders Structured Output With Gemini

The Next.js affiliate SaaS I built uses four defence layers — prompt instruction, Gemini's responseSchema, regex fence extraction, and Zod validation — and never hit a production parse failure, but two of those four layers are now unnecessary overhead.

The project has 10 AI tools, all routing through a single generateTextWithSchema<T>() function in src/lib/ai/gemini.ts. It was built before Gemini's responseSchema parameter was stable, so the architecture reflects a belt-and-suspenders philosophy: every possible failure mode has a fallback. That made sense at the time. It's now technical debt.

How generateTextWithSchema<T>() works

The function runs four layers in sequence. Layer 1 is the system prompt — "Return ONLY valid JSON" with no explanation, no markdown, no fences. Layer 2 is Gemini's native responseSchema parameter — constrained decoding at the model level. Layer 3 is application-side regex extraction that strips markdown fences and extracts the first {...} or [...] match. Layer 4 is downstream Zod validation in the Server Action that called the function.

The retry logic adds exponential backoff: up to 3 retries, 1,000ms initial delay, multiplier of 2, max 10,000ms, 25% jitter. JSON parse failures are caught and re-thrown as retryable errors — meaning a fence-wrapped response that survives Layer 2 but fails Layer 3 triggers a retry at the full cost of another API call.

// ── CURRENT APPROACH: 4-layer belt-and-suspenders ─────────────────────────────

// Layer 1: System prompt instructs JSON-only response
// Layer 2: responseSchema provides constrained decoding at model level
// Layer 3: Regex strips markdown fences (unnecessary with constrained decoding)
// Layer 4: Caller validates with Zod (still needed for business rules)

async function generateTextWithSchema<T>(
  prompt: string,
  systemPrompt: string, // contains "Return ONLY valid JSON. No explanation. No markdown."
  responseSchema?: object
): Promise<string> {
  const result = await generateText({
    model: gateway(TEXT_MODEL),
    system: systemPrompt,   // Layer 1: prompt engineering
    prompt,
    // responseSchema passed via provider options — Layer 2: constrained decoding
  });

  // Layer 3: Regex fence extraction — unnecessary when constrained decoding is active
  // If responseSchema is honoured, the model cannot emit fences.
  // This layer is a safety net for the case where responseSchema is ignored.
  const cleaned = result.text
    .replace(/```json\n?/g, '')
    .replace(/```\n?/g, '')
    .trim();
  const match = cleaned.match(/[\[{][\s\S]*[\]}]/);
  return match ? match[0] : cleaned;
  // Caller: JSON.parse() + Zod.safeParse() — Layer 4: business-rule validation
}

// ── RECOMMENDED REFACTOR: generateObject() — 2 layers removed ─────────────────

// Layer 1 (prompt JSON instruction):  REMOVED — redundant with responseSchema
// Layer 3 (regex fence extraction):   REMOVED — SDK returns typed object directly
// Layers 2 + 4 remain:
//   responseSchema via SDK (constrained decoding) + Zod in caller (business rules)

import { generateObject } from 'ai';
import { google } from '@ai-sdk/google';
import { z } from 'zod';

async function generateStructuredOutput<T>(
  prompt: string,
  schema: z.ZodType<T>,
  systemPrompt?: string
): Promise<T> {
  const { object } = await generateObject({
    model: google('gemini-2.5-flash'),
    schema,           // Zod schema → SDK translates to responseSchema automatically
    system: systemPrompt,
    prompt,
  });
  return object;
  // object is fully typed T — no JSON.parse, no regex, no fence stripping
  // Business-logic validation (word counts, required patterns) still runs in caller
}

Why 0 production failures doesn't mean the approach is right

The belt-and-suspenders approach worked — 0 real production parse failures across all 10 Gemini-powered tools. But "it works" and "it's the right architecture" are different claims. If Gemini's responseSchema parameter is ignored on a model update or edge case, the regex extraction falls back to prompt-only — and that's Level 1 territory with 8–15% failure rates. The safety net is the prompt layer. A safety net is not a guarantee.

The regex extraction pipeline is also unnecessary overhead when constrained decoding is active. Every response passes through regex matching, string replacement, and pattern extraction for zero benefit. The technical debt is real: four layers of defence where two are now redundant.

The Python Implementation: JSON Mode + Application Validation Across 161 API Calls

The FastAPI AI report SaaS I built uses OpenAI JSON mode combined with application-side schema validation — catching the syntax failures JSON mode prevents, while still retrying on the schema mismatches it doesn't prevent.

This is the project that generates 1,725-page AI reports. Each report triggers a Celery task that makes 161 sequential GPT-4o API calls — one per batch of content. The structured output strategy is JSON mode, and the cost of getting it wrong is measurable: at a 2% schema mismatch rate across 161 calls, approximately 3 retries per job are expected. At $15–25 per report, those retries are a real line item, not an edge case.

How the Horoscope runner handles 161 structured calls

The runner uses a combination of approaches. JSON mode (response_format={'type': 'json_object'}) guarantees syntactically valid JSON. Batch-specific prompt functions — build_daily_batch_prompt(), build_weekly_batch_prompt(), etc. — instruct the model to return specific JSON shapes. _validate_batch_response() enforces the expected structure on every returned batch and raises a ValueError on schema mismatch, triggering a tenacity retry. MAX_RETRIES = 3, exponential backoff multiplier=2, min=4s, max=60s.

The trailing-comma cleanup is the most honest signal in the codebase. JSON mode guarantees syntactically valid JSON — but it apparently doesn't always succeed, because the code contains a cleanup pass that strips trailing commas before retrying a failed json.loads(). That cleanup code should not exist in a system using constrained decoding. It exists because JSON mode doesn't constrain generation at the token level.

# ── CURRENT APPROACH: JSON mode + application-side schema validation ──────────

# Guarantees:     syntactically valid JSON (JSON mode)
# Does NOT guarantee: expected schema shape — hence _validate_batch_response
# Trailing-comma cleanup exists because JSON mode doesn't prevent trailing commas

response = await client.chat.completions.create(
    model="gpt-4o",
    response_format={"type": "json_object"},   # Level 2: syntax only
    messages=[
        {"role": "system", "content": build_system_prompt()},
        {"role": "user",   "content": build_daily_batch_prompt(batch_spec)},
    ],
    max_tokens=TOKEN_BUDGET_DAILY,
    temperature=TEMPERATURE,
    timeout=180.0,
)

raw_text = response.choices[0].message.content

# Trailing-comma cleanup — this code should not exist in a production system
# using constrained decoding. It exists because JSON mode doesn't prevent them.
try:
    parsed = json.loads(raw_text)
except json.JSONDecodeError:
    cleaned = re.sub(r',\s*([}\]])', r'\1', raw_text)   # strip trailing commas
    parsed = json.loads(cleaned)

# Application-side schema check — raises ValueError — tenacity retry
# With 161 calls and a 2% mismatch rate, this fires ~3 times per job
_validate_batch_response(parsed, batch_spec)


# ── RECOMMENDED REFACTOR: Pydantic model — constrained decoding (Level 3) ─────

# Trailing-comma cleanup:               REMOVED — constrained decoding prevents them
# _validate_batch_response (structure): REMOVED — schema enforced at token level
# Business-logic validation (quality):  STILL REQUIRED — schema can't check these

from pydantic import BaseModel, Field

class DailyBatchAnswer(BaseModel):
    date: str                                    # "YYYY-MM-DD"
    answers: list[str] = Field(min_length=20, max_length=20)

class DailyBatchResponse(BaseModel):
    batches: list[DailyBatchAnswer]

response = await client.beta.chat.completions.parse(
    model="gpt-4o",
    response_format=DailyBatchResponse,          # Level 3: constrained decoding
    messages=[
        {"role": "system", "content": build_system_prompt()},
        {"role": "user",   "content": build_daily_batch_prompt(batch_spec)},
    ],
    max_tokens=TOKEN_BUDGET_DAILY,
    temperature=TEMPERATURE,
    timeout=180.0,
)

# Refusal check — constrained decoding adds this failure mode; JSON mode does not
if response.choices[0].message.refusal:
    raise ValueError(
        f"Model refused structured output: {response.choices[0].message.refusal}"
    )

batch = response.choices[0].message.parsed      # Already a DailyBatchResponse
# Business-logic validation still required — schema doesn't check word counts
_validate_answer_quality(batch)

The per-batch checkpoint that makes retries safe

Each of the 161 API calls is followed by a PersonalHoroscopeRepository.upsert_batch() write to MongoDB. The completed batch IDs are persisted, and on task restart, already-completed batches are skipped. This checkpoint pattern means a retry — whether from a JSON failure, a schema mismatch, or a rate limit — costs one batch re-run, not the entire job. At $15–25 per report, the checkpoint limits retry cost to a fraction of the total. Without it, a failure at batch 140 would waste everything that came before it.

Level 1 vs Level 2 vs Level 3: What Each Approach Actually Guarantees

Each level eliminates a different class of failure — and none of them eliminates the need for business-logic validation in your application code.

"The trailing-comma cleanup function is the most honest signal in a codebase. It means JSON mode is doing syntax work that constrained decoding would have prevented — and every line of that cleanup code is a line that pays for the choice not to use native structured output. I wrote that cleanup code. I know exactly what it means."

What I'd Replace Tomorrow: Two Specific Migrations

Both projects have clear migration paths to Level 3 structured output that eliminate unnecessary code, reduce retry costs, and make the JSON contract explicit in types rather than in prompt text.

Next.js — replace generateTextWithSchema with generateObject

The migration removes two of the four defence layers. The system prompt's JSON instruction goes away — it's redundant with responseSchema and adds prompt tokens on every call. The regex fence extraction goes away — constrained decoding means fences cannot be generated. What remains is generateObject() from the Vercel AI SDK, the Zod schema passed directly, and the same downstream Zod validation in the Server Action for business rules. The Zod schema becomes the single source of truth — not duplicated between a prompt instruction and an application-side validator.

The migration across all 10 tools is a mechanical search-and-replace: swap generateTextWithSchema calls for generateObject calls, remove the JSON instruction from each system prompt, delete the regex extraction function entirely. The Zod schemas already exist downstream — they move up to become the schema argument.

Python — replace JSON mode with OpenAI Structured Outputs (Pydantic)

The migration is a 3-line change per API call plus the Pydantic model definitions. Replace response_format={'type': 'json_object'} with response_format=DailyBatchResponse. Replace .create() with .parse(). Add the refusal check. Remove the trailing-comma cleanup (its reason for existence disappears). Remove _validate_batch_response() structure checks — schema conformance is enforced at the token level. Retain all word-count and content-quality validation — those are business rules the schema cannot express.

At 161 calls per job and a 2% mismatch rate, migrating to Level 3 eliminates approximately 3 retries per report. At $15–25 per report, that's not just cleaner code — it's measurably lower cost per job.

// ── Production retry wrapper — works at both Level 2 and Level 3 ──────────────
//
// Level 2 errors to retry:
//   JSON.parse failures, schema validation errors (2–5% schema mismatch case),
//   429, 500, 503, network errors
//
// Level 3 errors to retry:
//   Refusals (check .refusal), 429, 500, 503, network errors
//   Schema errors: GONE — constrained decoding handles them
//   Parse errors:  GONE — SDK returns typed object directly
//
// Non-retryable at any level:
//   Authentication failures (401) — retrying wastes money, fix the key
//   Content policy violations — retrying the same prompt will fail again

type RetryableErrorType =
  | 'rate_limit'
  | 'server_error'
  | 'network'
  | 'schema_mismatch'
  | 'refusal'
  | 'auth'
  | 'unknown';

function classifyError(err: unknown): { retryable: boolean; type: RetryableErrorType } {
  const status = (err as any)?.status;
  if (status === 429) return { retryable: true,  type: 'rate_limit'    };
  if (status === 401) return { retryable: false, type: 'auth'          };
  if (status >= 500)  return { retryable: true,  type: 'server_error'  };
  if (err instanceof SyntaxError)
                      return { retryable: true,  type: 'schema_mismatch' };
  if ((err as any)?.message?.includes('refusal'))
                      return { retryable: false, type: 'refusal'        };
  if ((err as any)?.code === 'ECONNRESET')
                      return { retryable: true,  type: 'network'        };
  return { retryable: false, type: 'unknown' };
}

async function withRetry<T>(
  fn: () => Promise<T>,
  options: { maxRetries?: number; initialDelayMs?: number; label?: string } = {}
): Promise<T> {
  const { maxRetries = 3, initialDelayMs = 1000, label = 'llm_call' } = options;

  for (let attempt = 1; attempt <= maxRetries; attempt++) {
    try {
      return await fn();
    } catch (err) {
      const { retryable, type } = classifyError(err);
      const isLast = attempt === maxRetries;

      // Structured retry log — searchable in production log aggregators
      console.warn(JSON.stringify({
        event: 'llm_retry', label, attempt, maxRetries,
        errorType: type, retryable, willRetry: retryable && !isLast,
      }));

      if (!retryable || isLast) throw err;

      // Exponential backoff with 25% jitter, capped at 10s
      const base   = initialDelayMs * Math.pow(2, attempt - 1);
      const jitter = base * 0.25 * Math.random();
      await new Promise(r => setTimeout(r, Math.min(base + jitter, 10_000)));
    }
  }
  throw new Error(`${label}: exhausted ${maxRetries} retries`);
}

When Level 1 and Level 2 Are Still the Right Choice

Prompt-only and JSON mode are still valid for prototyping, for schemas too complex for constrained decoding, and for providers that don't yet support native structured output — but they require explicit application-side validation as a compensating control.

Cases where Level 3 is overkill or unavailable

• Schema too large or deeply nested: Constrained decoding overhead increases with schema complexity. Very large schemas with unions and deep nesting can make constrained decoding slower and more likely to trigger refusals. JSON mode plus strict application validation may be faster.
• Provider doesn't support native structured output: Some smaller providers and self-hosted models via Ollama support JSON mode but not full schema enforcement. Level 2 is the ceiling — compensate with strict application-side validation and aggressive retry.
• Exploratory prototyping: If you genuinely don't know the output shape yet, prompt-only is acceptable for rapid iteration. The rule: before it ships to users, it moves to Level 3.
• Reasoning models (o1, o3): As of mid-2026, some reasoning models have constraints around structured output. Check provider documentation before assuming Level 3 is available for every model you use.

The compensating control pattern for Level 1/2

When you can't use Level 3, the minimum viable production pattern is: Level 2 (JSON mode) + strict Pydantic/Zod schema validation + tenacity retry with exponential backoff + structured logging of parse failures. Never ship Level 1 (prompt-only) to production without at minimum a JSON mode upgrade. The 8–15% failure rate from the TokenMix.ai 2-million-call analysis is not acceptable at any meaningful scale. At 161 calls per job, an 8% failure rate means 12–13 failures per job. That's not a reliability problem — it's a design problem.

You can read more about production LLM reliability patterns on hassanr.com — including the batch processing and async scaling post that covers the job architecture behind these 161 API calls, and the crash-safe long-running AI jobs post on the checkpoint pattern in detail.

Frequently Asked Questions