When Model Costs Break Your Budget, the Real Problem Is Your Decision Table

Signals to watch

• cost overrun without clear cause
• fragmented cost discussions
• no single owner for inference spend
• scaling without attribution

Composite industry case. This page describes a reusable operating problem and decision method. It does not represent a named customer, real conversation, contract, revenue result or testimonial.

The invisible second budget

You approved inference spend based on one thing: per-token price. Then usage grew, features multiplied, and the cost line started moving in ways the pricing page could not explain. The model provider did not change its rates. Something else did.

The problem is not that inference costs more than expected. The problem is that five variables — feature value, request volume, cache hit rate, routing strategy and quality risk — are discussed in five different Slack channels, none of which talk to each other. Every team sees a piece of the number. Nobody sees the whole table.

Why teams misread it

When cost overruns appear, the natural reflex is to blame the model. It is the line item with the vendor name, the one line everyone agrees to point at. But the model’s per-token price is often the least elastic variable in the system. The real levers are elsewhere.

A team adding a new AI-powered feature rarely asks: what does one session cost across the full pipeline? The feature lead owns the experience. The engineer owns the integration. The operations person sees the bill at month end. Each person optimises for their own metric, and the hole in the middle keeps growing.

Another pattern: requests that should not reach the model do reach the model. No cache layer, no routing pre-check, no fallback for low-value queries. Every call is treated as equal, so every call costs the same — and the expensive ones are invisible until the invoice arrives.

The evidence review framework

A decision table resolves this. It is a single document — a shared document — that names every variable that affects inference cost and assigns three things to each one: an owner, the evidence required to review it, and a decision window.

Start with five rows.

Feature attribution. What product feature generated the request? If your system does not tag inference calls by feature, estimate it from the context you have — session entry point, prompt pattern, response format. Assign an owner who can trace the real count. Set a two-week window.

Volume by tier. Separate high-value sessions from low-value ones. Not every interaction needs a full model pass. Define two or three tiers and estimate what share of current traffic falls into each. Assign evidence: a sample of 200 recent requests, manually labelled. One week.

Cache coverage. What percentage of identical or near-identical prompts repeat across sessions? Estimate from logs or from what you know about user behaviour. The owner is the person who can implement a cache. Evidence: a 48-hour traffic sample. Decision window: three days.

Routing logic. Are all requests sent to the same model? A fallback chain — cheap model first, escalate on confidence — is a routing decision, not a technical one. Evidence: a breakdown of current model assignments. Owner: the product lead. Window: one week.

Quality risk tolerance. Every routing and caching decision trades cost for quality. You need a documented boundary — what degradation is acceptable, and for which features. This is the hardest row because it requires judgement. Evidence: a comparison of five sample outputs at two different cost levels. Owner: product lead. Window: ongoing, reviewed biweekly.

The team next step

Build the table in a shared document today. Do not wait for tooling. Do not wait for perfect data. Use estimates, label them as estimates, and set the next review date for each row.

The goal is not precision in week one. The goal is a single place where the conversation about inference cost lives, with named owners who have a date on their calendar. Once the table exists, the invisible second budget becomes visible. You stop asking “why is the bill high” and start asking “which row do we adjust this week.”

What automation cannot replace

Automation can tag requests, surface patterns, organise evidence and flag when a variable drifts outside its expected range. It can show you that feature B now generates three times the traffic of feature A, or that cache coverage dropped overnight after a deployment. That signal discovery and evidence organisation is exactly where a tool belongs.

But the decision table itself — the choice of which variable matters, the assignment of who owns it, the judgment of what quality risk is acceptable — those are human decisions. They require product context, user understanding and a willingness to say no to a feature that costs more than it delivers.

The tool’s job is to make the table easy to keep current. The product lead’s job is to make the table honest. Neither one works without the other.

Frequently asked questions

Why do teams blame the model provider when inference cost overruns happen?

Because volume, feature attribution, caching coverage, routing strategy and quality risk live in separate conversations. Without a unified decision table, the model provider's pricing page becomes the easiest scapegoat.

Can a spreadsheet replace a structured decision table?

A spreadsheet tracks numbers. A decision table assigns owners, evidence and a next-review date to each variable. The difference is the difference between a log and a workflow.

Is inference cost management something automation can fully handle?

No. Automation surfaces signals and organizes evidence, but the judgment call — which variable to adjust and at what quality trade-off — requires a human who understands the product context.