{"id":1546,"date":"2026-06-03T08:10:40","date_gmt":"2026-06-03T15:10:40","guid":{"rendered":"https:\/\/www.kenwalger.com\/blog\/?p=1546"},"modified":"2026-06-03T08:57:53","modified_gmt":"2026-06-03T15:57:53","slug":"operating-real-time-ai-slas-observability-and-knowing-when-its-broken","status":"publish","type":"post","link":"https:\/\/www.kenwalger.com\/blog\/ai\/operating-real-time-ai-slas-observability-and-knowing-when-its-broken\/","title":{"rendered":"Operating Real-Time AI: SLAs, Observability, and Knowing When It’s Broken"},"content":{"rendered":"

The previous four posts in this series covered the three architectural pillars of real-time AI at scale: feature pipelines, feature stores, and vector search. Each post addressed the design decisions and failure modes specific to one layer of the stack.<\/p>\n

This final post is about the layer that sits above all of them: operations.<\/p>\n

You can design a technically sound pipeline, a well-structured feature store, and a carefully maintained vector index \u2014 and still have a system that’s difficult to run in production, slow to recover from failures, and chronically unclear about whether it’s actually working. The difference between a system that’s architecturally sound and one that’s operationally mature is the difference between a system that was designed and one that was operated<\/em>.<\/p>\n

This post is about what operational maturity looks like for real-time AI systems: how to define what “working” means, how to know when it isn’t, and how to recover when things go wrong.<\/p>\n

\n

Start With the SLA: What Are You Actually Promising?<\/h2>\n

Every discussion of operations should begin with the service level agreement \u2014 not as a compliance document, but as a forcing function for clarity.<\/p>\n

An SLA for a real-time AI system needs to answer four questions:<\/p>\n

1. What is the latency target?<\/strong>
\nNot just average latency \u2014 P99. The 99th percentile is where user-visible degradation lives. “Average latency is 50ms” is compatible with “1% of requests take 2 seconds,” which is likely unacceptable for a real-time user-facing system. Define your latency target at P99, and optionally P999 for systems where tail latency matters especially.<\/p>\n

2. What is the availability target?<\/strong>
\nWhat fraction of requests must succeed, over what time window? 99.9% availability means roughly 8.7 hours of allowable downtime per year. 99.99% means 52 minutes. The difference in operational complexity between those two targets is significant \u2014 know which one you’re designing for.<\/p>\n

3. What is the freshness target?<\/strong>
\nFor real-time AI specifically, this is a dimension that generic SLA frameworks often omit. How stale can features be before the system is considered degraded? How old can vector index updates be before search quality is affected? Freshness is a correctness dimension, not just a performance dimension.<\/p>\n

4. What is the recall target?<\/strong>
\nFor systems that use vector search, recall is part of the quality contract. A system returning search results with 60% recall is functionally broken for many use cases, even if it’s technically available and within latency targets. Define a minimum acceptable recall threshold and treat violations as SLA breaches.<\/p>\n

These four dimensions \u2014 latency, availability, freshness, recall \u2014 form the complete SLA surface for a real-time AI system. Most teams define the first two and ignore the last two. The last two are where silent degradation hides.<\/p>\n

\n

The Latency Budget: Where Time Actually Goes<\/h2>\n

Once you have a P99 latency target, the next step is a latency budget \u2014 an explicit allocation of that target across each component in the serving path.<\/p>\n

A typical real-time inference serving path looks something like this:<\/p>\n

Request received\n    \u2502\n    \u251c\u2500\u2500 Feature retrieval (online store lookup)\n    \u2502\n    \u251c\u2500\u2500 Vector search (ANN index query)\n    \u2502\n    \u251c\u2500\u2500 Feature assembly (merge, null handling, type coercion)\n    \u2502\n    \u251c\u2500\u2500 Model inference (forward pass)\n    \u2502\n    \u251c\u2500\u2500 Post-processing (result formatting, business logic)\n    \u2502\n    \u2514\u2500\u2500 Response returned\n<\/code><\/pre>\n
Without a latency budget, each component is implicitly allocated “whatever it takes.” With a budget, each component has an explicit ceiling, and crossing that ceiling is an actionable signal rather than background noise.<\/p>\n
A worked example for a 100ms P99 target:<\/p>\n\n\n\n\n\n\n\n\n\n\n\n
Component<\/th>\nBudget<\/th>\nNotes<\/th>\n<\/tr>\n<\/thead>\n
Network (ingress + egress)<\/td>\n10ms<\/td>\nLargely fixed; optimize for geographic proximity<\/td>\n<\/tr>\n
Feature retrieval<\/td>\n15ms<\/td>\nBatch point lookup; single round-trip<\/td>\n<\/tr>\n
Vector search<\/td>\n25ms<\/td>\nANN query; tunable via ef<\/code> parameter<\/td>\n<\/tr>\n
Feature assembly<\/td>\n5ms<\/td>\nIn-process; should be negligible<\/td>\n<\/tr>\n
Model inference<\/td>\n35ms<\/td>\nDepends on model size and hardware<\/td>\n<\/tr>\n
Post-processing<\/td>\n5ms<\/td>\nBusiness logic; should be bounded<\/td>\n<\/tr>\n
Total<\/strong><\/td>\n95ms<\/strong><\/td>\n5ms headroom at P99<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n
The budget makes tradeoffs visible. If the model inference step takes 60ms instead of 35ms, you know immediately which other components need to compress to compensate \u2014 or that the overall target needs to be renegotiated. Without the budget, a 60ms model inference step is just “the model is slow,” with no clear next action.<\/p>\n
Latency budgets should be enforced in monitoring. If feature retrieval regularly exceeds its allocation, that’s an alert, not just a data point.<\/p>\n
\n
Observability: The Full Signal Stack<\/h2>\n
Observability for real-time AI systems requires monitoring signals at every layer of the stack. Most infrastructure monitoring covers the compute and network layers well. The AI-specific layers \u2014 feature freshness, value distributions, recall \u2014 are almost always underinstrumented.<\/p>\n
The complete signal stack looks like this:<\/p>\n
<\/p>\n
A few of these signals deserve particular attention because they’re routinely absent from production monitoring even in mature engineering organizations.<\/p>\n
Feature null rate at inference time.<\/strong> When a feature value is missing \u2014 because an entity is new, because a pipeline failed, because a schema changed \u2014 most feature stores serve a default value silently. The null rate tells you how often this is happening. A sudden spike in null rate is a leading indicator of pipeline failure, schema drift, or cold start volume changes. Without tracking it, you’re flying blind on a significant dimension of input quality.<\/p>\n
Prediction distribution drift.<\/strong> If the statistical distribution of your model’s outputs shifts \u2014 more extreme scores, a different mean, a collapsed variance \u2014 something upstream has changed. It might be a feature pipeline issue, a data quality problem, or genuine change in the underlying population. Monitoring output distribution doesn’t tell you which, but it tells you something changed, which is the signal that starts the investigation.<\/p>\n
Training-serving skew over time.<\/strong> We covered training-serving skew as an architectural problem in Posts 2 and 3. Here it’s an operational metric. Periodically sampling serving-time feature values and comparing their distribution to training-time values catches skew that accumulates gradually \u2014 not from a single bad deployment, but from slow drift in source data, transformation logic, or serving behavior.<\/p>\n
\n
Failure Modes and Recovery Patterns<\/h2>\n
Pipeline Failures<\/h3>\n
Batch pipeline failures are the most straightforward: a job fails, the scheduler reports it, and the on-call engineer can rerun it. The question is whether the feature store degrades gracefully in the interim.<\/p>\n
Design for stale-but-available.<\/strong> A feature store that returns stale values when the pipeline is delayed is better than one that returns errors. Stale values keep the model running, possibly with reduced quality. Errors stop the model from running entirely. Build explicit staleness thresholds: values older than N minutes trigger alerts; values older than M minutes trigger fallback behavior.<\/p>\n
Streaming pipeline failures are more complex. A streaming job that falls behind on processing \u2014 accumulating lag in the event queue \u2014 may not fail outright. It may continue processing, but with increasing delay, silently delivering features that are progressively more stale. Stream lag monitoring<\/strong> is the signal: track the gap between when events are produced and when they’re processed, and alert when it crosses a threshold.<\/p>\n
# Stream lag alert \u2014 conceptual\ndef check_stream_lag(consumer_group, max_lag_seconds):\n    lag = kafka_consumer.get_lag(consumer_group)\n    processing_rate = kafka_consumer.get_processing_rate(consumer_group)\n\n    estimated_catchup_seconds = lag \/ processing_rate if processing_rate > 0 else float('inf')\n\n    if estimated_catchup_seconds > max_lag_seconds:\n        alert(\n            f\"Stream lag critical: {lag} messages behind, \"\n            f\"estimated {estimated_catchup_seconds:.0f}s to catch up\"\n        )\n<\/code><\/pre>\n
Feature Store Failures<\/h3>\n
The online store is on the critical path for every inference request. Its failure mode is a total serving outage unless the system is designed with a fallback.<\/p>\n
Fallback strategies in priority order:<\/strong><\/p>\n
    \n
  1. Serve from cache.<\/strong> If the serving layer caches recent feature retrievals, a brief online store outage can be absorbed without user impact for entities whose features were recently accessed.<\/p>\n<\/li>\n
  2. Serve defaults.<\/strong> Pre-computed default feature vectors \u2014 global averages, segment priors, or zero vectors \u2014 can keep the model running at reduced quality during an outage.<\/p>\n<\/li>\n
  3. Degrade gracefully.<\/strong> For some use cases, serving a simpler non-ML fallback (most popular items, rule-based decisions) is preferable to serving degraded ML predictions.<\/p>\n<\/li>\n
  4. Fail fast.<\/strong> For use cases where prediction quality is critical and degraded predictions are worse than no predictions, explicit failure with a clear error is the right answer.<\/p>\n<\/li>\n<\/ol>\n
    The right strategy depends on your use case. What’s universally wrong is having no strategy \u2014 discovering during an incident that the serving layer has no fallback path and needs to be designed under pressure.<\/p>\n
    Vector Index Failures<\/h3>\n
    Vector index failures are typically not binary. The index doesn’t go down \u2014 it degrades. Recall drops. Latency increases. Results become less relevant.<\/p>\n
    The operational response to index degradation depends on how it’s detected:<\/p>\n
    If recall drops below threshold:<\/strong> Trigger an index rebuild or compaction. In a segment-based architecture, compacting the most degraded segments may be sufficient. In a monolithic index, a full rebuild is required \u2014 which means managing traffic during the rebuild window.<\/p>\n
    If latency increases without load increase:<\/strong> Check tombstone accumulation. An index with a high fraction of deleted vectors will show latency increases before recall visibly degrades. Triggering a cleanup or rebuild early \u2014 before recall becomes a problem \u2014 is cheaper than reacting after the fact.<\/p>\n
    During an embedding model migration:<\/strong> The dual-index serving strategy is the safest path. Route queries to both the old and new index, returning results from the new index where available and falling back to the old index for records not yet recomputed. Monitor the migration percentage and recall on both indices throughout.<\/p>\n
    \n
    Capacity Planning: Designing Ahead of the Problem<\/h2>\n
    Real-time AI systems fail at scale in predictable ways. Capacity planning is the practice of anticipating those failures before they occur.<\/p>\n
    Feature store capacity<\/strong> is driven by three variables: the number of entities, the number of features per entity, and the update rate. As any of these grow, both storage cost and write throughput requirements increase. The online store is typically the binding constraint \u2014 it’s expensive, and adding capacity requires planning time.<\/p>\n
    Model the growth of each variable separately. A user feature store that grows linearly with your user base is predictable. One that grows with user activity \u2014 where active users generate many feature updates per day \u2014 can grow superlinearly. Know which one you have.<\/p>\n
    Vector index capacity<\/strong> is driven by vector count, vector dimensionality, and query rate. Memory requirements for HNSW indices are roughly:<\/p>\n
    Memory (bytes) \u2248 num_vectors \u00d7 (dimension \u00d7 4 bytes + M \u00d7 8 bytes)\n\nWhere M is the HNSW connectivity parameter (typically 16-64)\n\nExample: 10M vectors, 1536 dimensions, M=32\n\u2248 10M \u00d7 (1536 \u00d7 4 + 32 \u00d7 8)\n\u2248 10M \u00d7 (6144 + 256)\n\u2248 10M \u00d7 6400\n\u2248 64 GB\n<\/code><\/pre>\n
    At 10 million vectors of typical embedding dimensionality, you’re looking at 50-100GB of memory just for the index \u2014 before accounting for the base vectors themselves. Planning for this before you hit the wall is significantly cheaper than scaling under pressure.<\/p>\n
    Inference compute capacity<\/strong> is the most familiar capacity planning domain, but AI workloads have spikier profiles than many web workloads. Model inference is CPU or GPU-bound, not I\/O-bound, which means autoscaling has a longer warmup tail. Design for headroom that can absorb spikes without triggering cold start of new inference instances under load.<\/p>\n
    \n
    Incident Response: What to Do When It Breaks<\/h2>\n
    When a real-time AI system degrades in production, the diagnosis path should be structured \u2014 not because engineers aren’t capable of reasoning under pressure, but because structured diagnosis is faster and less error-prone than ad hoc investigation.<\/p>\n
    A simple decision tree for real-time AI incidents:<\/p>\n
    Is end-to-end latency elevated?\n\u251c\u2500\u2500 YES \u2192 Check component latency breakdown\n\u2502         \u251c\u2500\u2500 Feature retrieval elevated? \u2192 Online store health\n\u2502         \u251c\u2500\u2500 Vector search elevated? \u2192 Index health (recall, tombstones)\n\u2502         \u2514\u2500\u2500 Model inference elevated? \u2192 Compute resource saturation\n\u2502\n\u2514\u2500\u2500 NO \u2192 Is prediction quality degraded?\n         \u251c\u2500\u2500 Is feature freshness stale? \u2192 Pipeline health (lag, job failures)\n         \u251c\u2500\u2500 Is null rate elevated? \u2192 Schema change or cold start spike\n         \u251c\u2500\u2500 Is output distribution shifted? \u2192 Feature distribution drift\n         \u2514\u2500\u2500 Is recall below threshold? \u2192 Index degradation\n<\/code><\/pre>\n
    The key discipline is following the tree rather than jumping to conclusions. In complex systems, the symptom that’s most visible is often not the one that’s most actionable. A latency spike might be caused by vector search, or by feature retrieval, or by upstream traffic patterns that are saturating the online store. The monitoring signals tell you which \u2014 if they’re in place.<\/p>\n
    Runbooks<\/strong> \u2014 documented step-by-step procedures for common failure scenarios \u2014 dramatically reduce mean time to recovery. A runbook for “online store latency spike” that lists the specific metrics to check, the commands to run, and the escalation path removes the cognitive load of structuring the investigation under pressure. Writing runbooks before incidents is one of the highest-leverage operational investments a team can make.<\/p>\n
    \n
    The Operational Maturity Progression<\/h2>\n
    Operational maturity for real-time AI systems isn’t a binary state. It develops in layers, and most teams are somewhere in the middle. A useful progression:<\/p>\n
    Level 0 \u2014 Reactive<\/strong>: The team discovers problems when users report them. No AI-specific monitoring. Recovery is ad hoc.<\/p>\n
    Level 1 \u2014 Instrumented<\/strong>: Basic metrics are in place for latency and availability. AI-specific signals (freshness, recall, distribution drift) are absent or manual.<\/p>\n
    Level 2 \u2014 Alerted<\/strong>: Alerts exist for the key AI-specific signals. On-call engineers are notified of degradation before users report it. Recovery is faster but still manual.<\/p>\n
    Level 3 \u2014 Documented<\/strong>: Runbooks exist for common failure scenarios. Incident response is structured and consistent. Post-mortems are conducted and drive improvements.<\/p>\n
    Level 4 \u2014 Automated<\/strong>: Common remediation actions are automated. Stream lag triggers automatic consumer group scaling. Index tombstone thresholds trigger automatic compaction. Freshness violations trigger automatic pipeline retries.<\/p>\n
    Most teams building real-time AI systems for the first time are at Level 0 or 1. Getting to Level 2 \u2014 instrumented and alerted on the AI-specific signals \u2014 is the single highest-leverage operational investment available. Levels 3 and 4 follow from the foundation that Level 2 provides.<\/p>\n
    \n
    Closing the Series<\/h2>\n
    This series started with a simple observation: real-time AI systems that hum in development routinely hit problems in production, and those problems aren’t model problems \u2014 they’re infrastructure and operations problems.<\/p>\n
    The five posts have traced the full operational arc:<\/p>\n