Why AI Inference Costs Are Rewriting Cloud Economics

For two years the industry talked about training as though it were the only line item that mattered. That framing is now obsolete. The dominant, recurring cost of running an AI product is inference — the compute burned every single time a user asks a model to do something — and it is quietly reshaping how cloud providers price capacity, how startups raise money, and which products survive contact with a real user base.

The structural difference between training and inference

Training is a capital event. It is expensive, it is bounded, and when it finishes you own an artifact. A company can raise a round, spend it on a training run, and point to a model as the asset that resulted. Accountants understand that shape.

Inference is the opposite. It is an operating expense that scales linearly, and sometimes super-linearly, with success. Every new user adds cost. Every longer conversation adds cost. Every reasoning step the model takes before answering adds cost. A product that goes viral does not amortise its compute across a larger base the way a traditional SaaS product does — it simply pays more.

That inversion matters because the entire software industry's financial intuition was built on near-zero marginal cost. Serving one more user of a database-backed web application costs fractions of a cent. Serving one more user of a frontier model can cost meaningful money, and unlike bandwidth, that price has not collapsed on a predictable curve.

Why the unit economics resist easy fixes

There are three compounding pressures:

• Memory bandwidth, not raw FLOPs, is the bottleneck. Generating tokens one at a time means repeatedly streaming model weights out of high-bandwidth memory. Adding more raw arithmetic throughput does not help if the accelerator is waiting on memory.
• Latency requirements block the obvious optimisation. The standard way to improve accelerator utilisation is batching — grouping many requests together. But interactive products need a first token in a few hundred milliseconds, which caps how long you can wait to assemble a batch.
• Reasoning models multiply token counts. Techniques that improve answer quality by having a model think through intermediate steps can increase the tokens generated per user question by an order of magnitude, even though the user never sees most of them.

What this does to cloud pricing

Hyperscalers have spent a decade selling general-purpose compute with commodity pricing dynamics. Accelerated compute breaks that model. Supply is constrained by an advanced packaging and high-bandwidth memory pipeline that cannot be expanded on a quarterly cadence, so providers ration access rather than compete purely on price.

The visible result is a stratified market. Large committed customers sign multi-year capacity reservations at negotiated rates. Everyone else pays on-demand pricing that is materially higher and, more importantly, offers no guarantee that capacity exists when needed. That is a meaningful change from the elastic-cloud promise that defined the previous decade.

A second-order effect is the rise of specialised inference providers. Companies whose entire technical proposition is serving open-weight models efficiently — through aggressive quantisation, custom kernels, speculative decoding and purpose-built silicon — can undercut general-purpose clouds because they optimise for exactly one workload shape. Their existence puts a genuine ceiling on what hyperscalers can charge for comparable serving.

The strategic responses worth watching

Model routing over model maximalism

The most consequential engineering pattern of the past year is routing: sending easy requests to small, cheap models and escalating only genuinely hard requests to frontier models. Teams that implement routing well frequently report large cost reductions with little measurable quality loss, because the majority of production traffic is not difficult.

Caching as an architectural primitive

Prompt caching, where the provider retains the processed representation of a repeated prefix, turns long system prompts and document context from a per-request cost into a near-amortised one. For retrieval-heavy applications that repeatedly send the same corpus, this is one of the few genuinely large discounts available without quality trade-offs.

Open weights as a pricing floor

Capable open-weight models matter less because any single company will self-host them and more because they establish a credible alternative. A buyer who can plausibly threaten to move to self-hosted infrastructure negotiates differently from one who cannot. That leverage exists even if the threat is never executed.

Where this leaves smaller builders

The uncomfortable conclusion is that AI products face a discipline that the last software generation largely escaped. Gross margin has to be engineered deliberately, not assumed. Practically, that means instrumenting cost per request from the first prototype, treating token budgets as a product constraint rather than an infrastructure detail, and designing features so that the expensive path is the exception rather than the default.

It also means being honest about which products genuinely need frontier capability. A large share of shipped AI features are classification, extraction and summarisation tasks that a small fine-tuned model handles at a fraction of the cost. Reaching for the largest available model is often a default rather than a decision.

What to watch over the next year

Three signals will tell you which way this goes. First, whether high-bandwidth memory supply loosens enough to soften accelerator pricing — that is a supply-chain question with a long lead time, not a software one. Second, whether inference-specific silicon captures a meaningful share of serving workloads, which would introduce real price competition into a market that currently has little. Third, whether efficiency techniques keep delivering compounding gains, because the last two years of quantisation and decoding research have repeatedly beaten expectations.

If all three land favourably, inference becomes a manageable cost line and the current anxiety looks like a transitional phase. If none do, expect a visible consolidation, where the AI products that survive are the ones whose value per query comfortably exceeds the cost of producing it.