When teams budget an AI feature, they price the model call and stop. That number is a fraction of the real cost, and optimizing it in isolation leads to worse decisions, not better ones. The cost of an AI feature is a total-cost-of-ownership question, and the metric that actually matters is not what you think.
Inference Is the Visible Tip
The per-call price of the model is the one cost everyone sees, so it gets all the attention. It is real - and at scale it compounds fast, which is why caching, routing to smaller models, and prompt trimming matter. But it is the visible tip of a much larger iceberg, and a feature can have cheap inference and ruinous total cost.
The Costs Under the Waterline
Evaluation. Building and maintaining the eval set, running graders, and reviewing results is ongoing work, not a one-time setup. It is the cost of knowing your feature still works, and it never goes to zero. Human review. For anything consequential, a person checks or approves the output. That human time is frequently the single largest line item, dwarfing inference, and it scales with volume. Monitoring. Quality drifts as models update and inputs shift; catching that requires instrumentation and someone watching it. Maintenance. Prompts rot, models get deprecated, retrieval sources change, and each requires engineering time to keep the feature working. Retries and failures. Every wrong answer that triggers a retry, an escalation, or a support ticket has a cost that never appears in the inference bill but is entirely caused by the feature.
What teams budget: inference cost per call
What it actually is: inference
+ eval set upkeep + grading
+ human review time (often the biggest)
+ monitoring & drift detection
+ prompt / model maintenance
+ cost of wrong answers (retries, escalations, support)The Metric That Matters: Cost Per Successful Outcome
Here is the key reframe. The right unit is not cost per API call but cost per successful outcome - per resolved ticket, per correct extraction, per satisfied user. This single change flips many decisions. A cheaper model that is wrong more often is not cheaper: it triggers more retries, more human corrections, and more escalations, so its cost per successful outcome can be higher than the "expensive" model that gets it right the first time. You cannot see this if you only track cost per call; you can only see it if you track cost per outcome, with quality and cost measured together.
Why This Changes Decisions
Once you measure cost per outcome, the counterintuitive moves become obvious. Spending more per call on a better model can lower total cost by removing downstream human correction. Investing in evals and prompt quality - pure cost on paper - pays for itself by raising the success rate and shrinking the expensive failure tail. Adding a cheap validation step that catches bad outputs before they reach a human saves far more than it costs. Every one of these looks wasteful through the cost-per-call lens and clearly correct through the cost-per-outcome lens.
The Takeaway
Budget the whole iceberg, not the tip. An AI feature's true cost includes evaluation, human review, monitoring, maintenance, and the price of being wrong - and human review often exceeds inference by a wide margin. Measure cost per successful outcome, not per call, and instrument cost and quality together so you can optimize the ratio rather than blindly minimizing the one number that is easy to see. The teams that do this ship AI that is genuinely economical; the teams that optimize inference in isolation ship features that look cheap in the model bill and quietly cost a fortune everywhere else.