Prompt Chaining vs. Single Mega-Prompts: When Each Wins

Five prompt steps, each 95% reliable, chained end to end: 0.95^5 = 0.774. You just built a pipeline that fails roughly one run in four, out of components that each look excellent in isolation. Nobody debugging that pipeline will look at any single step and see a problem, because no single step has one.

That multiplication is the most important number in agent design and it is the one most readily ignored, because per-step accuracy is what you can measure easily and end-to-end accuracy is what your users experience. Ten steps at 95% is 0.599. To hit 95% end to end across ten steps, each step needs 99.5%, which for anything involving natural language judgment is not a number you get by writing a better prompt.

The obvious conclusion is wrong

The naive read of that math is: use fewer steps, put everything in one prompt, done. And sometimes that is right. But a mega-prompt is not a 100%-reliable single step, it is a single step whose reliability degrades with its own length in specific, documented ways.

Instruction dilution. Attention is a finite budget distributed across the context. Give a model forty instructions and the twenty-ninth gets a smaller share of it than it would have gotten as one of five. This shows up as constraints that are followed sometimes, which is worse to debug than constraints that are never followed, because you cannot reproduce it.

Lost in the middle. Liu et al. measured retrieval accuracy as a function of where the relevant information sat in the context and found a pronounced U-shape: models perform best when the needed content is at the beginning or the end, and measurably worse when it is in the middle, with the degradation large enough that some models with long contexts did worse than the same models given only the top few retrieved documents. The paper is Lost in the Middle: How Language Models Use Long Contexts. The practical consequence for prompt design is unglamorous but concrete: the instructions buried at line 300 of your 600-line system prompt are the ones being dropped, and moving them to the end is a free intervention you should try before restructuring anything.

Error entanglement. In a chain, a bad step-3 output is visible as a bad step-3 output. In a mega-prompt, a bad intermediate inference is invisible; it silently contaminates everything downstream inside one forward pass, and all you see is a wrong final answer with no trace of where it went wrong.

A decision rule you can apply without arguing

Split into a chain when any of these is true:

• The steps need different tools or different models. A step that calls a search API and a step that writes prose are different operations with different failure modes. Forcing them into one call means the whole call retries when only the search failed. It also means you pay reasoning-model prices for the formatting step.
• An intermediate artifact needs to be inspected, validated, or cached. If you want to schema-check a plan before executing it, the plan must exist as a discrete output. This is the strongest argument for chaining, because a validated boundary converts a silent wrong answer into a caught error.
• One step is expensive and its input is stable. A document summarization step whose input changes daily and whose output is consumed by fifty downstream queries should be its own step, so you compute it once. Merging it into the mega-prompt means recomputing it fifty times.
• Steps have different fan-out. One extraction pass over 200 chunks feeding one synthesis call is inherently two steps. There is no single prompt shape for that.

Keep it as one prompt when:

• The steps share the same context and each one needs the full picture. Splitting forces you to re-serialize that context into every step, which costs tokens and loses nuance at each handoff.
• No intermediate output is independently valuable. If nothing inspects it, caches it, or branches on it, the boundary buys you nothing and costs you a round trip.
• The transformation is genuinely holistic. "Rewrite this for a technical audience" does not decompose into tone, vocabulary, and structure passes without the passes fighting each other.

Cost and latency, counted honestly

Chains are serial. Each step is a full round trip: network, queue, prefill, decode. On a typical hosted API, time to first token is 300 to 800 ms before any generation happens. Five steps means you are paying that fixed overhead five times, so a chain that generates the same total output as a mega-prompt is meaningfully slower on wall clock even with identical token counts.

Token accounting cuts the other way, and both effects are real:

Mega-prompt:   1 x (4000 in + 900 out)          = 4000 in,  900 out
5-step chain:  5 x (1200 shared context + ...)  = 6000 in, 1100 out
               (shared preamble re-sent per step)

Chains re-send context. That is the tax, and it is why naive decomposition can cost 50% more input tokens for the same work. Two things reduce it: prompt caching, where a stable prefix shared across steps is billed at a large discount on cache hits, and prompt-specific context pruning, where step 4 receives only step 3's output instead of the accumulated transcript. The second is the more disciplined fix and it is where most chains are sloppy, because appending everything is easier than deciding what each step actually needs.

What makes chains actually viable

Go back to 0.95^5 = 77%. Chains multiply failure only if a failed step kills the run. The entire practical argument for chaining is that step boundaries are places you can catch and recover.

for step in pipeline:
    for attempt in range(3):
        out = call(step, ctx)
        if step.validate(out):        # schema check, assertion, cheap judge
            break
        ctx.add_correction(step.last_error)
    else:
        raise StepFailed(step.name)   # loud, localized, resumable
    checkpoint(step.name, out)        # persist so a retry resumes here

With a validator that catches 80% of step failures and up to three attempts, a 95% step behaves like roughly 99.6% for pipeline purposes, and 0.996^5 is about 98%. The chain now beats the mega-prompt on reliability rather than losing to it. The multiplication did not change; the per-step number did, because you added a check.

Checkpointing is the other half and it is operational, not clever. Persist each step's output keyed by input hash. When step 4 of 5 fails at 3 a.m., the retry resumes at step 4 instead of re-running and re-paying for the first three. Without this, a chain is a distributed transaction with no journal, and every failure costs full price.

A chain with validated, checkpointed boundaries is strictly better than a mega-prompt for anything multi-tool. A chain without them is strictly worse, and most chains in production do not have them. That is the actual dividing line, not the number of steps.

Unimatrix leans on this by making the shared context a retrieval call instead of a re-pasted block, so each step in a chain pulls only what it needs from memory rather than carrying the whole transcript. The prompt patterns we use for that are collected in prompts.