{"id":1532,"date":"2026-05-13T07:53:12","date_gmt":"2026-05-13T14:53:12","guid":{"rendered":"https:\/\/www.kenwalger.com\/blog\/?p=1532"},"modified":"2026-04-28T08:26:09","modified_gmt":"2026-04-28T15:26:09","slug":"feature-freshness-designing-pipelines-that-keep-up-with-the-world","status":"publish","type":"post","link":"https:\/\/www.kenwalger.com\/blog\/ai\/feature-freshness-designing-pipelines-that-keep-up-with-the-world\/","title":{"rendered":"Feature Freshness: Designing Pipelines That Keep Up With the World"},"content":{"rendered":"

In the previous post<\/a>, we identified three categories of pressure that expose architectural weaknesses when AI pipelines scale: load variability, data velocity, and index drift. This post is about data velocity \u2014 specifically, the feature freshness problem.<\/p>\n

The core question is deceptively simple: how old is the data your model is reasoning about when it makes a prediction?<\/strong><\/p>\n

For some workloads, a few hours of staleness is harmless. For others, a few minutes can meaningfully degrade prediction quality. And for a growing class of real-time applications \u2014 fraud detection, dynamic pricing, live personalization \u2014 the answer has to be measured in seconds.<\/p>\n

Getting feature freshness right is primarily an architectural problem, not a modeling problem. The model doesn’t control how fresh its inputs are. The pipeline does.<\/p>\n

\n

Why Features Go Stale (And Why It Matters)<\/h2>\n

A feature is a representation of something that happened in the world: a user clicked something, a transaction was attempted, an inventory level changed. That event occurred at a specific moment in time. The feature value derived from it has a half-life \u2014 a window during which it accurately represents reality.<\/p>\n

When the pipeline can’t deliver features fast enough, the model receives a picture of the world that’s already out of date. For stationary signals \u2014 a user’s age, a product’s category \u2014 staleness is irrelevant. But for behavioral signals \u2014 recent purchase history, session activity, account velocity \u2014 staleness is a direct hit to prediction quality.<\/p>\n

Consider fraud detection. A model trained to catch account takeover attempts needs to know what the account has done in the last few minutes, not the last few hours. A batch pipeline refreshing features every two hours is structurally incapable of catching a credential-stuffing attack that executes in 20 minutes. The model isn’t wrong. The data is wrong.<\/p>\n

The same dynamic plays out across recommendation systems (a user’s interest signal from three hours ago is not the same as their interest signal right now), dynamic pricing (demand changes faster than hourly batch cycles can track), and content moderation (viral spread happens in minutes).<\/p>\n

Freshness is a system property, not a model property.<\/strong> Which means the solution lives in the pipeline.<\/p>\n

\n

The Two Pipeline Architectures<\/h2>\n

Batch Pipelines: Simple, Reliable, and Structurally Limited<\/h3>\n

A batch pipeline computes features on a schedule. A job runs every hour (or every day, or on-demand), reads from a source of truth, computes aggregations and transformations, and writes the results to a feature store for the model to consume at inference time.<\/p>\n

Batch pipelines are operationally mature. The tooling is well-understood \u2014 Spark, dbt, Airflow \u2014 and the failure modes are predictable. When a batch job fails, you know about it immediately and you can rerun it. They’re also cost-efficient: compute runs when you schedule it, not continuously.<\/p>\n

Their limitation is structural. The minimum freshness a batch pipeline can deliver is bounded by the job interval. An hourly job delivers features that are, at best, a few minutes old and, at worst, nearly an hour old. For workloads that need sub-minute freshness, no amount of operational optimization changes this fundamental constraint.<\/p>\n

Batch pipelines are the right answer when your features don’t change faster than your batch interval, or when the cost of staleness is low. They’re the wrong answer when your model depends on recent behavioral signals.<\/p>\n

Streaming Pipelines: Fresh, Continuous, and More Complex to Operate<\/h3>\n

A streaming pipeline processes events as they arrive. Rather than computing features on a schedule, it reacts to each event in the source stream \u2014 a user action, a transaction, a sensor reading \u2014 and updates the relevant feature values immediately.<\/p>\n

The result is features that are seconds old rather than minutes or hours old. For workloads where that difference matters, streaming is the only viable architecture.<\/p>\n

The tradeoff is operational complexity. Streaming systems \u2014 typically built on Kafka for transport and Flink or Spark Structured Streaming for processing \u2014 have more moving parts than batch pipelines. Failures are harder to reason about: what happens to in-flight events when a processing node goes down? How do you handle out-of-order events? How do you test a streaming job end-to-end without a production-like event stream?<\/p>\n

These aren’t reasons to avoid streaming. They’re reasons to be intentional about when you adopt it, and to invest properly in the operational infrastructure when you do.<\/p>\n

\n

The Practical Answer: Lambda Architecture<\/h2>\n

Most production systems that need real-time ML don’t need all<\/em> of their features to be fresh in real time. They need some<\/em> features \u2014 typically behavioral signals \u2014 to be fresh, while relying on batch computation for historical aggregates and slowly-changing dimensions.<\/p>\n

This is the insight behind the Lambda architecture pattern, which has become the most widely deployed approach for production ML feature pipelines.<\/p>\n

The architecture has two parallel processing paths:<\/p>\n