Real-time AI inference is harder than it looks in demos. A model that scores 94% accuracy in offline evaluation may exhibit 400ms latency spikes in production, fail under traffic bursts, or degrade silently when input distribution shifts. The architecture decisions made at inference serving time determine whether your AI investment delivers its promised business value or creates a reliability liability.
This guide covers the architecture patterns, hardware decisions, and optimisation techniques for enterprise real-time inference. It is based on production deployments across fraud detection, clinical AI, recommendation systems, and conversational AI applications where latency and reliability directly affect business outcomes.
42ms
p99 latency target for real-time fraud scoring in the financial services case study referenced throughout this guide. The model architecture, hardware, and serving stack were all redesigned once to meet this threshold after the initial deployment produced 380ms p99.
Latency Requirements by Use Case
Before designing inference infrastructure, establish the actual latency requirement. Most teams over-engineer for latency they do not need, or under-engineer for latency they do. The table below reflects production benchmarks from 200+ enterprise deployments.
| Use Case Category | p50 Target | p99 Target | Failure Mode if Exceeded |
|---|
| Real-time fraud scoring (payment) | 8ms | 42ms | Payment blocked; customer friction |
| Recommendation (e-commerce click) | 25ms | 80ms | Fallback to non-AI recommendations |
| Clinical decision support (alert) | 200ms | 800ms | Clinician workflow disruption |
| Customer service routing (IVR) | 150ms | 500ms | Perceived lag in conversation |
| Document classification (intake) | 500ms | 2000ms | Queue back-up; throughput degradation |
| GenAI response (chat) | 1000ms | 4000ms | User abandonment above 5s first token |
| Predictive maintenance (sensor) | 5s | 30s | Alert lag; missed early warnings |
| Batch scoring (overnight) | — | Processing window | Late reports; SLA breach |
Inference Infrastructure: Four-Layer Architecture
Production inference infrastructure has four functional layers that must be designed together. Optimising one layer while neglecting another consistently produces latency spikes that are difficult to diagnose.
Layer 01
Load Balancing and API Gateway
Request routing, authentication, rate limiting, and observability. Handles traffic bursts, distributes load across inference replicas, enforces circuit breakers for degraded model instances. API contract versioning ensures backward compatibility during model updates.
NGINXEnvoyKongAWS API GatewayAzure APIM
Layer 02
Feature Serving
Online feature store retrieval at sub-10ms latency. Pre-computed entity features retrieved by primary key without on-the-fly computation. Freshness SLAs define how recently features must have been computed. Cache warming prevents cold-start latency spikes during deployments.
RedisDynamoDBFeastTectonBigtableCassandra
Layer 03
Model Serving
Model inference execution on CPU, GPU, or specialised accelerators. Batching of concurrent requests to improve hardware utilisation without sacrificing latency SLAs. Model versioning with rolling deployments and shadow mode for production validation before traffic cutover.
TorchServeTensorRTTriton Inference ServerONNX RuntimevLLMRay Serve
Layer 04
Observability and Feedback
Latency histograms at p50, p90, p95, p99, p999. Prediction distribution monitoring for drift detection. Error rate and timeout tracking. Ground truth capture for continuous evaluation. Alerting on latency threshold breaches and error rate spikes.
PrometheusGrafanaDatadogArize AIWhyLabsEvidently
Hardware Selection: CPU vs GPU vs Specialised Accelerators
Hardware selection is the single highest-impact cost and latency decision in inference infrastructure. The right choice depends on model architecture, throughput requirements, and latency targets. The wrong choice results in either excessive cost or unacceptable latency.
CPU Inference
Traditional ML Models
Best for: Gradient boosting, linear models, shallow trees
Gradient boosting models (XGBoost, LightGBM) serving under 50,000 requests per second typically run efficiently on CPU. Lower cost per inference than GPU. Simpler infrastructure. Optimal for tabular data models where model size is small. Latency typically 5 to 30ms for these architectures.
GPU Inference
Deep Learning Models
Best for: Neural networks, transformers, embeddings
Neural networks with more than 10M parameters require GPU for sub-100ms inference at production throughput. Batch inference dramatically improves utilisation. T4 for cost-efficient serving; A10G or A100 for latency-critical transformer inference. Key decision: Tensor parallelism for LLMs above 7B parameters.
Specialised Accelerators
LLMs and High-Volume Inference
Best for: LLMs, real-time vision, edge AI
AWS Inferentia 2 and Google TPU v5e deliver 2 to 4x better price-performance versus GPU for stable production LLM workloads. Requires specific framework support and model compilation. Edge inference (IoT, mobile) uses purpose-built silicon: NVIDIA Jetson, Apple Neural Engine, Qualcomm AI 100.
Need help designing your inference architecture?
Our AI implementation advisors have designed production inference systems from sub-50ms fraud detection to LLM serving at 40,000 concurrent users. Independent, hardware-vendor-neutral guidance.
Talk to an Implementation Advisor →
Six Latency Traps That Catch Most Enterprises
The same latency problems appear repeatedly across enterprise AI deployments. Each is preventable with the right design decisions made early.
Cold Start Latency from Container Spinning
Auto-scaling adds new inference containers on traffic spikes. Cold starts on GPU instances take 60 to 180 seconds for model loading. The first requests routed to a new instance experience catastrophic latency. This problem is invisible in load testing if tests do not include scale-up scenarios.
Fix: Maintain minimum warm replica count above zero. Use model caching in container image. Implement scale-to-zero only for non-SLA-critical workloads. Pre-warm cache on container start before accepting traffic.
Feature Store Cache Miss at Peak Traffic
Online feature store serves pre-computed features from in-memory cache. Cache hit rate of 95% is common. But during traffic bursts, the 5% cache misses all require on-demand feature computation simultaneously, saturating the feature pipeline and causing latency spikes that appear unrelated to the model.
Fix: Pre-warm feature cache for high-traffic entities before peak periods. Monitor cache hit rate and alert at 90% threshold. Separate read replicas for inference vs. training workloads. Implement circuit breaker returning default features when feature pipeline is degraded.
Memory Bandwidth Saturation on GPU
GPU inference for large models is often memory-bandwidth-bound rather than compute-bound. Adding more GPUs does not improve latency when a single GPU is already bandwidth-saturated. Batch size tuning becomes critical and counter-intuitive: increasing batch size can increase throughput without meaningfully increasing latency.
Fix: Profile with NVIDIA Nsight before hardware scaling decisions. Quantise model weights (INT8 or FP16) to reduce memory bandwidth requirements by 2x. Test batch sizes from 1 to 64 on your specific model to find the optimal throughput-latency trade-off.
Serialisation Overhead in the Request Path
JSON serialisation and deserialisation for feature payloads can contribute 15 to 40ms to inference latency at high throughput. This overhead is invisible in low-traffic testing but becomes dominant in production when the serving layer handles thousands of requests per second.
Fix: Use Protocol Buffers or MessagePack instead of JSON for high-throughput inference APIs. Profile end-to-end request path, not just model inference time, to identify serialisation contributions. gRPC instead of REST for internal service communication eliminates most serialisation overhead.
Database Lookups in the Prediction Path
A feature that requires a synchronous database query in the prediction path adds 20 to 100ms of latency that cannot be compressed. This pattern appears when feature engineering logic is moved into the serving layer without being pre-computed. It is impossible to meet sub-50ms SLAs with synchronous database calls in the hot path.
Fix: Every feature used by real-time models must be pre-computed and stored in the online feature store. No synchronous database calls in the prediction path. If freshness requirements prevent pre-computation, the latency target may be incompatible with that feature set.
Tail Latency Spikes from Garbage Collection
Python-based serving stacks (TensorFlow Serving, PyTorch) experience periodic garbage collection pauses that create p99 and p999 latency spikes even when p50 latency is stable. These spikes are often misdiagnosed as model or hardware issues and are invisible in standard monitoring that reports average latency only.
Fix: Monitor p99 and p999 latency, not just p50. Configure JVM or Python GC parameters for low-pause collection. Consider compiled inference runtimes (TensorRT, ONNX Runtime) for latency-critical workloads. Profile GC pause frequency and duration under production load.
Six Optimisation Techniques Worth Applying
Once the architecture is in place and infrastructure is correctly sized, six optimisation techniques reliably improve inference latency without requiring model retraining.
Optimisation 01
Model Quantisation
Typical gain: 2x latency improvement, 4x memory reduction
Convert model weights from FP32 to INT8 or FP16. Marginal accuracy loss (typically less than 0.5% on well-tested models) in exchange for significant latency and memory improvements. TensorRT and ONNX Runtime both support automatic post-training quantisation.
Optimisation 02
Dynamic Batching
Typical gain: 3 to 5x throughput, minimal latency impact
Batch concurrent requests together for a single forward pass. Triton Inference Server and TorchServe support dynamic batching with configurable latency tolerance. Effective when request arrival rate exceeds single-request inference time. Does not help at low concurrency.
Optimisation 03
KV Cache Optimisation (LLMs)
Typical gain: 40 to 60% throughput improvement for LLMs
Key-value cache stores intermediate attention computations for previously processed tokens. PagedAttention (used by vLLM) manages GPU memory for KV cache efficiently, enabling higher concurrency. Critical for LLM serving at production throughput.
Optimisation 04
Semantic Caching
Typical gain: 40 to 70% latency reduction for repeated queries
Cache model outputs keyed by input embedding similarity, not exact match. Near-duplicate queries return cached results without inference. Effective for RAG pipelines and knowledge base chatbots where question patterns repeat. Redis with vector similarity search enables this at low overhead.
Optimisation 05
Speculative Decoding (LLMs)
Typical gain: 2 to 3x throughput for autoregressive generation
A small draft model generates multiple token candidates that a large verification model validates in parallel. Accepts correct tokens, rejects incorrect ones. Effective when the draft model acceptance rate exceeds 70%. Requires careful tuning for production deployment.
Optimisation 06
Cascaded Inference
Typical gain: 60 to 80% cost reduction with maintained accuracy
Route easy requests to a fast, cheap small model and hard requests to a larger model. A router model or confidence threshold determines routing. 80% of requests are typically easy enough for the small model. Dramatic cost reduction without meaningful accuracy degradation for well-designed routing logic.
White Paper
AI Implementation Checklist
200-point production readiness checklist covering inference infrastructure, load testing requirements, monitoring setup, and rollback procedures. Standard at 22 Fortune 500 enterprises.
Download Free →
Monitoring Real-Time Inference in Production
An inference system that is not monitored comprehensively will degrade silently. The metrics that matter go beyond what most infrastructure teams initially instrument.
Latency distribution, not averages. Instrument p50, p90, p95, p99, and p999 latency. p99 latency is the experience of 1 in 100 users. p999 is the experience of 1 in 1,000, and for high-volume systems this represents thousands of requests per day. Alert on p99 breaches, not average latency. Average latency can be stable while p99 latency is 10x worse.
Prediction distribution monitoring. The distribution of model output scores should remain stable over time. Score distribution shift is an early signal of input distribution drift or a data pipeline problem. A fraud model that was scoring 8% of transactions as high risk that now scores 3% has changed in a way that warrants investigation, even if both values might be defensible individually.
Feature freshness and availability. Monitor the age of features being served. A feature with a 15-minute freshness SLA that is being served with 2-hour-old values will degrade model performance in ways that appear to be model quality problems but are actually data pipeline failures. Instrument freshness as a first-class metric.
Business outcome correlation. Where possible, link inference output to business outcomes. Fraud model scores to actual fraud cases. Recommendation scores to click rates. This linkage is what enables continuous model evaluation and catches silent degradation before it becomes a business problem.
94%
of production AI incidents are detectable through proper monitoring before they escalate to business impact. The investment in observability infrastructure consistently returns more value than the same investment in model complexity.
Deployment Patterns for Zero-Downtime Updates
Model updates in production require careful management. A naive deployment that replaces the current model with a new one can cause latency spikes, accuracy degradation, or serving errors that affect users.
Blue-Green deployment. Maintain two identical serving environments. Deploy the new model to the inactive environment, run validation tests, then shift traffic. Rollback is instant by redirecting traffic back. Requires double infrastructure cost during the transition window. Appropriate for major model updates.
Canary deployment. Gradually increase traffic to the new model while monitoring latency and output distribution. Start at 1%, then 5%, 10%, 25%, 50%, 100%. Automated canary analysis compares metrics between old and new model. Rollback triggered automatically if metrics degrade. Appropriate for iterative model improvements.
Shadow mode validation. Run the new model on live traffic without affecting the production response. Compare outputs to the current production model. Validate that distribution and latency match expectations before routing any user-affecting traffic. Essential for high-risk model changes in regulated environments.
Need Production AI Infrastructure Expertise?
Senior implementation advisors help you design inference infrastructure that meets your latency SLAs, scales reliably, and avoids the deployment traps that cost enterprises months of debugging.
Talk to an Implementation Advisor →
The AI Advisory Insider
Weekly practitioner intelligence on AI infrastructure, MLOps, and production deployment. No vendor promotion.
