Cutting Prompt Token Cost Without Losing Output Quality

Prompt cost is a design constraint, not a bill that arrives later. Two things drive it: how many tokens you send, and what you pay per token. Most teams optimize neither, then wonder why the bill moved. The useful move is to understand which part of your prompt is actually costing money, because in most applications the answer is not the instruction. It is the context you re-send on every call.

Cache the stable prefix

Most prompt caching works on a prefix basis: the leading portion of the prompt is cached and only the trailing portion is re-read. This inverts the usual instinct to put the variable part first, because the variable part now sits at the end by design. System instructions, rubrics, schemas, and few-shot examples are stable across calls and belong at the front, where they get cached. The user input, the retrieved documents, and the current date go at the end.

# Cached: everything here is identical across calls, and is read once
# rather than re-billed on every request.
SYSTEM INSTRUCTIONS
OUTPUT SCHEMA (JSON, v3)
EVALUATION RUBRIC
EXAMPLE 1 ... EXAMPLE 5

# Uncached: this part changes per request.
CURRENT DATE: 2026-09-29
USER INPUT: {{user_input}}

# Wrong order, and expensive: the variable part breaks the cache immediately.
USER INPUT: {{user_input}}
CURRENT DATE: 2026-09-29
SYSTEM INSTRUCTIONS
...

Compact the variable part

The part that is not cached is the part worth squeezing. Retrieved documents should be chunked to the smallest span that still answers the question, deduplicated, and ordered so the most relevant content is nearest the instruction the model is meant to follow. History should be truncated to what the task needs, and clearly marked as truncated so the model does not treat an abrupt cut as a complete record.

Route by difficulty, not by hope

Using a small model for everything wastes money on hard cases, and using a large model for everything wastes more. Routing pays when you have a signal. Cheap and surprisingly effective signals exist: input length, whether retrieval returned confident matches, how many steps the task needs, and a short classifier pass that labels the request as simple, moderate, or hard. Route on the signal, and measure the escape rate, because a router that sends hard work to a small model pays twice: once for the bad answer, once for the retry.

def route(request, signal):
    if signal.input_tokens > 8_000 or signal.steps_needed > 5:
        return "large"          # long context or multi-step: do not economise here
    if signal.retrieval_confidence < 0.4:
        return "large"          # weak retrieval usually means a genuinely hard case
    if signal.label == "simple":
        return "small"
    return "medium"

# Cost per successful task, not cost per call. A cheap call that needs
# three retries is the expensive call.
def cost_per_success(spend, successes):
    return spend / successes if successes else float("inf")

That last function is the metric that actually matters. Cost per call is a vanity number; cost per successful task is the one that reflects whether your efficiency work helped, and it makes retries and escapes impossible to hide.

Cheap is not the same as efficient. Measure what you spend per task that worked.

Where the tokens actually go

Before optimizing anything, measure the split. Most unexpected bills come from a handful of causes, and they look different from each other in the usage numbers, so one measurement tells you which fix applies. Divide usage into the stable system prompt, the variable input, the conversation history, retrieved documents, tool definitions, and the output. Anything that grows with conversation length deserves attention first.

Two numbers are worth watching on every dashboard: input tokens per request and output tokens per request. A rising input with flat output usually means a leaking conversation or an over-broad retrieval query. A rising output with flat input usually means the prompt stopped constraining length, and it is the cheapest of these to fix because the fix is one instruction.

A weekly review catches what daily numbers hide. Compare the same metric against the same weekday a week earlier rather than against yesterday, because usage is usually tied to traffic rather than to your prompts, and a Monday comparison will convince you that your fix failed.

Cache, shorten, or route: one decision per component

Each part of a prompt has exactly one correct cost strategy, and picking the wrong one is why optimization efforts stall. Caching pays off for a prefix that is byte-identical across requests. Shortening pays off for a prefix that changes every time. Routing pays off when the variable part is what actually determines difficulty. Applying all three to the same segment means paying the maintenance cost of each while getting the benefit of only one.

Where this backfires

• Compressing instructions below the point where behavior changes. Shortening a rubric is easy; shortening it past its clarity is a regression.
• Caching a prefix that changes for one user. One dynamic character in the stable block disables caching for every call.
• Routing on a signal you have not validated. An unmeasured difficulty classifier is a guess with extra infrastructure.
• Optimizing tokens while ignoring retries, because a rejected structured response is billed three times.