Context Engineering for AI Agents in Production: The Four Decisions That Matter

I was doing context engineering in both my production AI SaaS products before I had a name for it. In one project — an AI report generation SaaS I built — I pre-compute 365 days of domain-specific data before making a single OpenAI call, because without that grounded context, the model hallucinate facts it has no reliable training data for. In the other — an affiliate marketing SaaS I built — I split every AI tool call into a constant system layer and a variable user layer, because mixing them degrades format compliance as inputs grow. Neither of these was a prompt trick. Both were information architecture decisions. That's context engineering. Here's the framework.

Why Prompt Engineering Breaks Down When Your AI Pipeline Gets Complex

Prompt engineering optimises how you phrase instructions to a model. Context engineering optimises what information the model sees — and once you're building multi-step AI pipelines, it's the latter that determines whether the system works in production.

The shift from chatbots to agents is what breaks prompt engineering as the primary skill. A chatbot has one turn, one prompt, one response. An agent has 161 turns — or 10 parallel tool calls — each with a different context state. At that scale, prompt wording matters far less than information architecture: what grounded data is in the window, what is excluded, how the context is structured, and how much budget each call gets. According to the LangChain State of Agent Engineering report, 57% of organisations now have AI agents in production — with most production failures traced to poor context management, not model capability. And 65% of developers report that AI misses relevant context during refactoring, testing, or code review (Qodo, 2025, n=609).

The "lost in the middle" phenomenon compounds this: language models process content at the beginning and end of their context window more reliably than content buried in the centre. As pipelines grow, context management failures look like model failures — but they're architecture failures, and no amount of rephrasing the prompt fixes them.

The context window is RAM, not storage

The foundational mental model: the context window is volatile working memory. It is flushed on every call. It has a size limit. Content at the boundaries of the window gets more attention than content in the centre. Treating it like a database — dump everything in, retrieve later — is the root cause of most production agent failures. Treating it like RAM — carefully manage what's loaded, evict what isn't needed, budget the space — is context engineering.

Context Engineering: The Four Decisions That Determine What Your Model Sees

Context engineering is the deliberate design of everything a language model sees on every inference call — and in production, it comes down to four decisions: what to inject, what to exclude, how to structure it, and how much to budget.

The four decisions

These four decisions are made in every AI pipeline. The question is whether they're made deliberately — context engineering — or accidentally, and then debugged after users notice failures.

• Decision 1 — INJECT: What grounded, computed, or retrieved facts does the model need to answer correctly without hallucinating? Anything the model cannot reliably know from training data — recent events, computed facts, user-specific data, domain calculations — must be injected.
• Decision 2 — EXCLUDE: What information, while available, would add noise, dilute attention, or trigger "lost in the middle" degradation? Every token in the context window has a literal API cost plus attention cost. Include only what the model needs for this specific call.
• Decision 3 — STRUCTURE: What goes in system prompt vs user prompt, in what order, with what formatting to maximise attention on critical instructions? Models process the beginning and end of context most reliably — put critical instructions at the top.
• Decision 4 — BUDGET: What is the right token allocation for this call, and what happens to output quality when the budget is exceeded? Larger is not better. Latency, cost, and quality all degrade with context bloat.

What context engineering is NOT

Context engineering is not RAG — RAG is one tool within context engineering. It is not memory management alone — that's one decision within the discipline. It is not structured output — that's about what comes out, not what goes in. Context engineering is the overarching discipline that includes all of these as sub-decisions. The term was coined by Andrej Karpathy in mid-2025 and has since become the framing used by Anthropic, Sourcegraph, and LangChain for what was previously described as "prompt design" or "RAG engineering." A 2026 DataHub report found 95% of data teams plan to invest specifically in context engineering training this year.

Context Engineering in the Python AI Pipeline: Grounding 161 API Calls With Pre-Computed Data

The most important context engineering decision in a long-running AI pipeline is deciding what grounded, computed facts to inject before any AI call runs — because the model cannot reliably know domain-specific, date-specific, or computed facts from training data alone.

The pre-computation stage as a context engineering decision

In an AI report generation SaaS I built, every job runs a deliberate two-stage pipeline. Stage 1 is pure pre-computation — no AI calls. All domain-specific calculations run first, producing structured data for every time period the AI will cover. This includes date-specific domain data for every day in the report's scope. Stage 2 is AI generation: each of the 161 sequential batch calls receives a precisely scoped slice of the Stage 1 output injected into the user prompt.

The context engineering insight: without the pre-computed injection, the model would generate content based on its training data approximations — producing outputs that sound plausible but contain factually wrong domain-specific details for specific dates. The model isn't failing at language generation; it's failing because it was never given the grounded facts it needed. The pre-computation stage exists specifically to give the model verified, computable context it cannot reliably derive from training alone. This is a context decision, not a data engineering decision.

The token budget as a quality control mechanism

# Context engineering: pre-computed data injected per batch, not all at once
# Stage 1 runs before any OpenAI call — produces structured context for Stage 2

# System prompt — CONSTANT across all 161 calls
# Defines role, output format, quality constraints
# Always at the top of context — maximum attention weight
SYSTEM_PROMPT = """You are a personalised report writer for domain-specific readings.
Generate responses that are 80-150 words each.
Do NOT use generic advice. Every answer must reference the specific domain
context provided in the user message.
Return ONLY a valid JSON object matching this structure: {
  "responses": [{"question_id": str, "answer": str}]
}"""

# User prompt — VARIABLE per batch (batch 1 of 161 to batch 161 of 161)
# Contains the pre-computed domain context for THIS batch's date range only
# Does NOT contain all 365 days — that would be context bloat
def build_daily_batch_prompt(batch_spec: DailyBatchSpec, precomputed_data: dict) -> str:
    # Inject ONLY the 3-day slice of pre-computed data for this batch
    # Decision 2 (EXCLUDE): full 365-day dataset is never injected per call
    # Injecting all 365 days: context bloat + "lost in the middle" + 30x cost
    batch_context = precomputed_data["daily"][batch_spec.start_index:batch_spec.end_index]
    return f"""
Domain Context for this batch (pre-computed, factually verified):
{format_batch_context(batch_context)}

Generate personalised responses for all 20 questions below.
Each response must reference the specific context above.
{format_questions(batch_spec.questions)}"""

# Token budget decisions — context engineering, not arbitrary limits
# Budget = smallest value that produces full-quality output for this batch type
TOKEN_BUDGET_DAILY  = 12_000   # 3 days x 20Q x ~150 words x 1.3 buffer
TOKEN_BUDGET_WEEKLY =  8_000   # 2 weeks — smaller scope per question
TOKEN_BUDGET_MONTHLY = 8_500   # 1 month — reduced from 2 months (see below)
TOKEN_BUDGET_YEARLY  = 4_000   # 1 year overview — narrative, not Q&A format

# WHY MONTHLY WAS REDUCED FROM 2 MONTHS TO 1 MONTH:
# Original: 2-month batches (6 API calls for 12 months)
# Problem:  Months 11-12 produced 20-30 word answers instead of 80-150 words
# Root cause: "lost in the middle" — questions in the latter half of a long
#             context window received degraded attention weight from the model
# Fix:      1 month per batch (12 API calls for 12 months)
#           Added 6 API calls and ~3 minutes per job
#           Month 1 questions are now at the START of a fresh context window
#           Result: full-quality 80-150 word answers across all positions
# This is a context budget decision, not a prompt decision.
# No amount of prompt rewording fixes the "lost in the middle" problem.

Context Engineering in the Next.js AI Toolkit: System Prompt vs User Prompt as Architecture

The most important context engineering decision in a multi-tool AI SaaS is enforcing a strict boundary between the constant system layer — role, format, constraints — and the variable user layer — specific inputs — because mixing them causes format instructions to get less attention as user data grows.

The system/user separation as an architecture decision

In an affiliate marketing SaaS I built, every one of the 10 AI tools uses generateTextWithSchema<T>() with a deliberate system/user split. System prompt is the constant layer: what the model is in this call. User prompt is the variable layer: what the model is working with in this call. The common mistake is including format instructions in the user prompt alongside the user's input data. As user data grows longer, format instructions get pushed toward the middle of the combined context — exactly where attention degrades. Keeping format instructions in system (always at top, always maximum attention weight) and user data in user (variable, can grow) prevents this degradation.

// Context engineering: deliberate separation of constant vs variable context
// System prompt = CONSTANT — what the model IS in this tool call
// User prompt   = VARIABLE — what the model is WORKING WITH in this call
// Never mix them: mixing causes format degradation as user input grows

const SYSTEM_PROMPT = `You are an expert direct-response copywriter
specialising in affiliate marketing on social platforms.

Your output must follow this exact JSON structure:
${JSON.stringify(outputSchema, null, 2)}

Quality constraints:
- Never make specific income claims
- Never use countdown timers or false scarcity
- Write in second person, benefit-focused, conversational tone
- Each headline: 6-10 words maximum`

// ALL format constraints live in the system prompt (constant, top of context)
// None appear in the user prompt — they would be diluted by user data growth

async function generateAdCopy(input: AdCopyInput): Promise<AdCopyOutput> {
  // Gate 3 of 4: Zod validation BEFORE context construction
  // Malformed, oversized, or missing-field input never reaches the context window
  // This is a context quality gate, not just a type safety check
  const validated = AdCopyInputSchema.parse(input)

  // User prompt = VARIABLE — only the specific, validated user data for this call
  // No format instructions here — they live in the system prompt at full attention
  const userPrompt = `
Product: ${validated.productName}
Commission: ${validated.commissionRate}%
Target audience: ${validated.targetAudience}
Platform: ${validated.platform}
Ad placement: ${validated.placement}
Key benefit to emphasise: ${validated.keyBenefit}
${validated.additionalContext ? `Additional context: ${validated.additionalContext}` : ''}`

  // Context window at execution:
  // [SYSTEM: role + format + constraints — maximum attention, constant position]
  // [USER: clean validated input — variable, can grow without displacing system]
  // Format instructions are never buried by growing user data

  const result = await generateTextWithSchema<AdCopyOutput>(
    userPrompt,
    SYSTEM_PROMPT
  )
  return result
}

Why Zod validation is a context engineering gate

Gate 3 in the Server Action pipeline — Zod validation — runs before the context is constructed. This means only clean, normalised, schema-conformant data enters the context window. Raw user input with typos, missing fields, inconsistent formatting, or unexpected values would produce inconsistent model outputs because context quality directly determines output quality. Validating first ensures the variable layer of context is always well-formed on every call. The rate limiting gate (Gate 2, 30 req/min per user) also serves a context purpose: it prevents context flooding from rapid repeat calls that would waste API tokens on near-duplicate requests.

The "Lost in the Middle" Problem: Why More Context Often Means Worse Output

Language models reliably process content at the beginning and end of their context window, but attention degrades for content buried in the centre — and this effect gets worse as context grows, causing production quality failures that look like model problems but are actually context architecture failures.

What "lost in the middle" means in practice

In the AI report generation SaaS, the monthly batch originally ran 2 months per call — 6 API calls for a 12-month report. The questions for month 2 were positioned in the middle-to-end of a long context window where attention weight drops. The result: months 11–12 consistently produced 20–30 word answers instead of the required 80–150 words. The model wasn't failing — it was attending less to questions at positions where context attention degrades. The fix was splitting to 1 month per batch: 12 calls instead of 6, adding 6 API calls and approximately 3 minutes per job. Month 1 questions are now at the start of a fresh context window — full attention, full quality answers.

The benchmark evidence confirms this direction. Mem0's 2026 LoCoMo benchmark found that a two-layer memory architecture using 6,956 tokens achieved 91.6% accuracy at a p95 latency of 1.44 seconds. A full-context baseline using 26,000 tokens achieved only 72.9% accuracy at p95 17.12 seconds. The smaller, curated context was 18.7 percentage points more accurate with 4 times fewer tokens and 91% lower latency. Bigger context is not better context.

The context budget heuristic that prevents this

// Context budget utility — prevents "lost in the middle" degradation
// Rule: keep total input context under 60% of model's max to maintain
// attention quality at both ends of the window
// (The 60% rule is a practical production heuristic — adjust per model)

interface ContextBudgetResult {
  safe: boolean
  totalTokens: number
  usagePercent: number
  recommendation?: string
}

function estimateTokens(text: string): number {
  // Rough approximation: 1 token ≈ 4 characters for English text
  // Use tiktoken or your provider's tokenizer for production accuracy
  return Math.ceil(text.length / 4)
}

function checkContextBudget(
  systemPrompt: string,
  userPrompt: string,
  maxContextTokens = 8192
): ContextBudgetResult {
  const systemTokens = estimateTokens(systemPrompt)
  const userTokens   = estimateTokens(userPrompt)
  const totalTokens  = systemTokens + userTokens
  const usagePercent = (totalTokens / maxContextTokens) * 100

  // Above 80%: "lost in the middle" degradation is very likely
  // Action: reduce batch size, split the call, or compress context
  if (usagePercent > 80) {
    return {
      safe: false, totalTokens, usagePercent,
      recommendation: `Context at ${Math.round(usagePercent)}% — split into smaller calls`
    }
  }

  // 60-80%: quality degradation risk for content at end of long outputs
  // Action: monitor output quality at end of responses, consider splitting
  if (usagePercent > 60) {
    return {
      safe: false, totalTokens, usagePercent,
      recommendation: `Context at ${Math.round(usagePercent)}% — monitor output quality at end of responses`
    }
  }

  return { safe: true, totalTokens, usagePercent }
}

// Real usage: the monthly batch fix in the AI report SaaS was essentially this check.
// 2-month batches: ~14,000 tokens against an 8,500 token budget → forced
//   compression of months 11-12 → 20-30 word answers instead of 80-150 words.
// 1-month batches: ~7,000 tokens → 82% of budget → within safe zone,
//   full quality at all positions in the response.
// The fix added 6 API calls and ~3 minutes per job. Worth every second.

The Context Engineering Checklist for Every Production AI Call

Before any AI call goes to production, four questions determine whether the context is engineered correctly — and most production AI failures can be traced to a "no" on at least one of them.

Question 1: Is every fact the model needs but cannot reliably know from training data explicitly injected into the context?
Yes: Pre-computed domain data injected per batch in the AI report pipeline — specific, verified, date-accurate.
No: Relying on the model's parametric knowledge for date-specific, domain-specific, or user-specific facts → hallucination that sounds plausible but is factually wrong.

Question 2: Is every piece of information in the context window there because this specific call needs it — not because it was convenient to include?
Yes: Only the 3-day slice of pre-computed data per daily batch, not all 365 days.
No: Injecting the full dataset "to be safe" → context bloat, cost inflation, attention dilution, and "lost in the middle" degradation for items further in the window.

Question 3: Are your format instructions and role definition in the system prompt rather than mixed with variable user data?
Yes: System/user separation enforced in all 10 tools in the affiliate SaaS — format instructions never appear in user prompt.
No: Format instructions in user prompt alongside growing user data → format compliance degrades as input length increases because instructions get buried in the middle.

Question 4: Is your total context budget below 60% of the model's context limit, with your most critical content at the start of the system prompt?
Yes: Monthly batch reduced to 1 month per batch after 2-month batches produced degraded quality for months 11–12.
No: Context budget exceeded → "lost in the middle" quality degradation for content positioned later in the window.

"Prompt engineering asks how to phrase the question. Context engineering asks what the model needs to know to answer it correctly. In production AI systems with 10 tools or 161 sequential calls, the phrasing matters far less than the information architecture. I learned this by watching a model produce 20-word answers for month 12 of a 2-month batch — and realising the model didn't need better instructions. It needed a smaller context window."

More production AI architecture patterns — including the batch processing system behind these 161 API calls and the structured output strategy that handles schema enforcement across all of them — are documented in detail at hassanr.com.

Frequently Asked Questions