The AI Cost Crisis That Nobody Planned For
When enterprises approved AI pilot budgets, nobody modelled what production scale would cost. A model that costs $200 per month in development costs $80,000 per month in production at actual transaction volume. That math was not in the business case. Finance is now asking hard questions and AI teams are scrambling to answer them.
The problem is structural. AI workloads have cost profiles unlike any other enterprise technology. Inference costs scale with usage in ways that are hard to predict. Training costs spike irregularly. GPU availability creates pressure to over-provision rather than right-size. The combination produces bills that surprise even experienced technology leaders.
The good news is that most enterprise AI infrastructure spend is not optimal spend. In our work assessing production AI programs, we routinely find 30 to 50% cost reduction available through optimisations that have minimal impact on performance. This article covers the highest-impact categories.
Across the enterprise AI programs we have assessed, average infrastructure waste runs at 38% of total spend. The largest single category is idle GPU compute (over-provisioned for peak demand, running at 15% utilisation off-peak). Second is inefficient model serving (synchronous serving where batch would suffice). Both are fixable without changing a single model.
The Cost Lever Taxonomy
AI infrastructure cost has four primary drivers, each with different optimisation levers and different effort requirements. Understanding which driver dominates your bill determines where to focus first.
High-Impact Optimisations: The Ranked List
This table ranks the optimisations we most frequently implement in enterprise AI cost reduction engagements. Savings estimates are based on actual implementations and will vary by workload profile.
| Optimisation | Typical Savings | Implementation Effort | Works Best When |
|---|---|---|---|
| Right-size to smaller model | 40 to 70% | Medium | Task is well-defined, quality can be measured, fine-tuning is feasible |
| Semantic caching for LLM calls | 20 to 50% | Low | Query distributions are clustered; many near-identical requests |
| Spot instances for training | 60 to 80% | Low | Training jobs are checkpointed and can tolerate interruption |
| int8 or fp16 quantisation | 50 to 75% compute | Low | Recall degradation under 2% is acceptable for the use case |
| Async batch serving | 30 to 60% | Medium | Use case tolerates 1 to 60 second response time (analytics, scoring, generation) |
| Autoscale to zero for dev | 70 to 90% off-hours | Low | Dev and staging environments are not used 24/7 |
| Speculative decoding for LLMs | 2 to 3x throughput | Medium | Output lengths are predictable and a smaller draft model is available |
| Model distillation | 60 to 85% | High | High-volume task with stable distribution and budget for training |
| Checkpoint pruning and lifecycle | 15 to 30% storage | Low | Storage costs are growing; no checkpoint retention policy exists |
| Reserved capacity for production | 30 to 40% compute | Low | Production workload is predictable; 1 or 3-year commitment acceptable |
Semantic Caching: The Fastest Win
Semantic caching is the practice of storing the outputs of LLM calls and returning cached results for queries that are semantically similar to a cached query, even if they are not word-for-word identical. For enterprise applications where users ask similar questions (support chatbots, internal knowledge bases, product search), cache hit rates of 20 to 40% are typical. Cache hit rates above 50% are common in use cases with narrow query distributions.
The implementation requires a vector store for the query cache (a small, fast one like Redis with vector support or a dedicated cache tier), an embedding model to vectorise incoming queries, and a similarity threshold above which a cached result is returned rather than triggering a new LLM call.
The threshold setting is the critical decision. A threshold that is too high returns irrelevant cached results and damages user experience. A threshold that is too low captures too few cache hits to justify the infrastructure cost. For most enterprise use cases, a cosine similarity threshold of 0.92 to 0.95 balances quality and hit rate. Always A/B test the threshold with a sample of real queries before deploying to production.
Semantic caching integrates naturally with the inference architecture as a layer upstream of the model serving infrastructure, with no changes to the model itself.
Model Right-Sizing: The Highest Leverage Optimisation
The single highest leverage cost optimisation in most enterprise AI programs is using a smaller model for the task. This sounds obvious. In practice it requires courage to implement, because "bigger is safer" is the default assumption and smaller models require validation work to confirm they meet quality requirements.
The validation framework is straightforward. Define a quality metric for the task (accuracy for classification, ROUGE for summarisation, human preference rate for generation). Run the current model and candidate smaller models against a representative test set. If the smaller model meets the quality threshold, it is the right model regardless of its parameter count.
The parameter counts that matter: for structured extraction tasks (named entity recognition, classification, field extraction), models in the 7B to 13B range, fine-tuned on domain data, typically match GPT-4 class models. For reasoning-heavy tasks (complex analysis, multi-step problem solving), larger models still have an edge. For simple generation tasks (summaries, reformatting, basic Q&A), 3B to 7B models are often adequate.
A Top 20 bank replaced a GPT-4 class model with a fine-tuned 8B parameter model for loan document extraction. After three weeks of fine-tuning and validation on 50,000 historical documents, the smaller model achieved 97.3% field extraction accuracy versus 97.8% for the larger model, at 82% lower inference cost per document. The quality delta was acceptable; the cost saving was not optional.
The Six Waste Categories to Audit First
Building an AI FinOps Practice
Cost optimisation as a one-time exercise recovers some waste. Cost optimisation as an ongoing practice prevents the waste from returning. Mature enterprise AI programs have a FinOps function specifically for AI infrastructure, mirroring the cloud FinOps practices that became standard for general cloud spend.
The organisational design of AI FinOps matters. Cost visibility without accountability produces dashboards nobody acts on. The most effective model is cost attribution to the product or business team consuming the AI service, with the platform team accountable for infrastructure efficiency. Product teams control usage volume. Platform teams control cost per unit. Both metrics need owners.
LLM-Specific Cost Optimisation
Large language model inference has a different cost profile from traditional ML. The cost is primarily token-based (input tokens plus output tokens), not compute-time-based. This changes where the optimisation levers are.
Input context management is the first lever. Prompt templates that include unnecessary context, long system prompts that get repeated on every call, and retrieval systems that return 20 documents when 3 would suffice all inflate input token counts. A retrieval system that returns 5,000 tokens of context per query costs 5x more per query than one that returns 1,000 tokens with equal or better relevance. The data architecture upstream of the LLM determines retrieval quality and token efficiency together.
Output length control is the second lever. Many enterprise LLM use cases do not need verbose responses. Structured output formats (JSON, specific schemas) are typically shorter than natural language explanations of the same information. For programmatic use cases, requesting structured output reduces both output tokens and the parsing complexity downstream.
Model routing is the third lever. Not all queries in a given application have equal complexity. A routing layer that classifies query complexity and routes simple queries to a smaller, cheaper model while sending complex queries to the larger model can reduce costs by 40 to 60% in applications with mixed query distributions. This requires a fast, cheap classification step but the economics are typically favourable above a few thousand daily queries.
Connecting Cost Optimisation to AI Strategy
Cost optimisation is not a containment exercise. It is a sustainability exercise. AI programs that run over budget get cancelled or scaled back. Programs that demonstrate cost efficiency earn continued investment. The enterprise AI programs that survive and grow are the ones that can show a clear cost per business outcome, not just a total infrastructure bill.
The framing that works with finance is cost per prediction, cost per document processed, or cost per transaction scored, benchmarked against the business value of the decision being informed. When cost per prediction drops from $0.40 to $0.08 through optimisation, the ROI case for scaling from 100,000 predictions per month to 1,000,000 becomes straightforward to make.
For teams building the financial case for continued AI investment, the AI strategy advisory includes financial modelling that connects infrastructure cost to business outcome metrics, creating the language that finance and the board respond to.
For teams implementing the technical optimisations, the AI implementation advisory includes an infrastructure cost review as a standard workstream, with prioritised recommendations based on your specific workload profile.