Cutting LLM Costs

A feature that delights users in a demo can quietly bankrupt the unit economics when a million people use it. LLM calls are metered by the token - every word in and every word out has a price - and at scale those fractions of a cent compound fast. Cost control is not a nice-to-have you bolt on later; it is often what decides whether an AI feature can ship at all.

Understand the Bill First

You pay for input tokens (your prompt, including any retrieved context and examples) and output tokens (what the model generates), usually at different rates, with output more expensive. Two levers follow immediately: send fewer tokens in, and generate fewer tokens out. Most cost waste hides in bloated prompts - giant system messages, redundant examples, whole documents stuffed in when a paragraph would do.

1. Caching: Do Not Pay Twice

Exact-match caching. If the same input has been answered before, return the stored answer for free. Trivial to implement, and for anything with repeated queries (FAQs, common lookups) it can eliminate a large share of calls.

key = hash(prompt)
if key in cache:
    return cache[key]          # $0, instant
answer = call_model(prompt)
cache[key] = answer
return answer

Semantic caching. Real questions are rarely identical but often equivalent. Embed the incoming query, and if a past query is close enough in meaning, reuse its answer. Powerful, but set the similarity threshold carefully - too loose and you serve the right answer to the wrong question.

Prompt caching. Many providers now let you cache a large static prefix (a long system prompt, a fixed knowledge block) so you are not billed full price to re-process it on every call. If you send the same 3,000-token instructions every time, this alone can cut input cost sharply.

2. Routing: Match the Model to the Task

Not every request needs your most capable, most expensive model. A classification or a simple rewrite can go to a small cheap model; only the genuinely hard reasoning needs the flagship. A router - even a simple one - decides per request.

def route(task):
    if task.type in ("classify", "extract", "short_rewrite"):
        return SMALL_MODEL      # cents on the dollar
    if task.needs_deep_reasoning:
        return FLAGSHIP_MODEL
    return MID_MODEL

A common pattern is the cascade: try the cheap model first, and only escalate to the expensive one when the cheap answer fails a confidence check or a validation. Most traffic is handled cheaply; the flagship is reserved for the tail that needs it.

3. Smaller Models, Sharpened

Small models are dramatically cheaper and faster, and for a narrow, well-defined task a small model - optionally fine-tuned on that one task - can match a large general model at a fraction of the cost. The move: prototype with the big model to prove the task is solvable, then distill down to the smallest model that still passes your eval set. Do not run a flagship in production for a job a small model does fine.

4. Trim the Tokens

Concrete reductions that add up: prune system prompts to what is load-bearing; drop redundant few-shot examples once the model is reliable without them; in RAG, retrieve fewer, better chunks rather than many mediocre ones and re-rank so you can send top 3 instead of top 10; cap max output length so the model cannot ramble; and ask for terse formats (JSON, bullet points) rather than verbose prose when a program will consume the result.

5. Batch and Stream

For non-urgent workloads, many providers offer batch pricing at a steep discount in exchange for slower turnaround - ideal for offline processing, nightly summarization, or bulk classification. For user-facing latency, streaming does not cut cost but improves perceived speed, which can let you use a slightly cheaper model without users noticing.

Measure Cost Per Outcome, Not Per Call

The metric that matters is not cost per API call but cost per successful outcome - per resolved ticket, per correct extraction, per satisfied user. A cheaper model that fails half the time and triggers retries or human escalation is not cheaper. Instrument cost alongside quality, watch them together, and optimize the ratio.

The Order of Operations

In practice: cache aggressively (the cheapest token is the one you never send), route so most traffic hits small models, trim prompts and outputs, distill to the smallest model that passes evals, and batch what can wait. Do these and the same feature that looked unaffordable at scale becomes not just viable but profitable - which is the difference between an AI experiment and an AI product.