Design
Turn model intent into explicit feature definitions.
ML DATA ENGINEERING · FEATURE PLATFORM
A useful feature is not merely a formula in a notebook. It is a time-aware, tested and reproducible data product that must behave consistently during training and prediction. A feature store provides the system around that promise.
Turn model intent into explicit feature definitions.
Reconstruct historical values without future leakage.
Deliver the same meaning for batch training and online inference.
Track owners, lineage, freshness, versions and quality.
01 · THE ECONOMICS
A ratio can take one line of SQL. Making it correct for five years of training data, available within milliseconds, monitored every day and understandable to another team is a different engineering problem.
Domain interviews, profiling, experiments and many features that never reach production.
Large joins, rolling windows, backfills and repeated training datasets consume compute and engineering time.
Point-in-time joins must use only information available when each prediction would have occurred.
Freshness, failed pipelines, schema changes, skew, drift, latency, incident response and ownership remain after launch.
Different teams rebuild “customer activity” with different windows, filters and missing-value policies.
Leakage or training-serving skew may improve an offline score while weakening real predictions.
02 · FROM COLUMN TO DATA PRODUCT
“orders_30d” is incomplete unless we know the entity, event time, inclusion rules, null policy, freshness, owner and version.
The interval ends at t but excludes t. This prevents an event occurring at prediction time from entering a historical training row before it would have been known.
03 · WHAT A FEATURE STORE DOES
A feature store is not simply a database with a fashionable name. It is an ML data system that connects a registry, historical retrieval and—in low-latency use cases—online serving.
04 · OFFLINE AND ONLINE ARE DIFFERENT JOBS
Training wants large, correct historical datasets. Real-time inference may want the latest value in milliseconds. The store coordinates meaning; it does not erase the physical trade-off.
Optimized for scans, history, point-in-time retrieval, backfills and training-set generation. Often built on a warehouse or lakehouse.
Optimized for keyed lookup and predictable low latency. Usually holds the latest feature values, not all history.
Describes feature identity and ownership. It helps discovery and reuse but cannot guarantee that a poor definition becomes meaningful.
05 · LAB: DEFINE HISTORICAL FEATURES
This simplified PostgreSQL-compatible example teaches the principle. Production systems need tuned storage, incremental computation and tested orchestration.
CREATE TABLE ml.training_events (
customer_id INTEGER NOT NULL,
prediction_time TIMESTAMP NOT NULL,
label INTEGER
);
-- One row represents one customer at one prediction time.
CREATE VIEW ml.v_customer_features_training AS
SELECT
t.customer_id,
t.prediction_time,
t.label,
COUNT(o.order_id) FILTER (
WHERE o.status = 'completed'
) AS completed_orders_30d,
COALESCE(SUM(o.amount) FILTER (
WHERE o.status = 'completed'
), 0) AS completed_value_30d,
MAX(o.ordered_at) AS latest_known_order_time
FROM ml.training_events AS t
LEFT JOIN raw.orders AS o
ON o.customer_id = t.customer_id
AND o.ordered_at >= t.prediction_time - INTERVAL '30 days'
AND o.ordered_at < t.prediction_time
GROUP BY t.customer_id, t.prediction_time, t.label;06 · LAB: MATERIALIZE THE LATEST VALUES
The primary key models one latest feature vector per entity. A production online store also needs atomic writes, freshness guarantees, availability, latency SLOs and recovery.
CREATE TABLE ml.customer_features_online (
customer_id INTEGER PRIMARY KEY,
completed_orders_30d INTEGER NOT NULL,
completed_value_30d DECIMAL(14,2) NOT NULL,
feature_timestamp TIMESTAMP NOT NULL,
feature_version VARCHAR(20) NOT NULL
);
INSERT INTO ml.customer_features_online
SELECT
c.customer_id,
COUNT(o.order_id) FILTER (WHERE o.status='completed'),
COALESCE(SUM(o.amount) FILTER (WHERE o.status='completed'), 0),
CURRENT_TIMESTAMP,
'customer_activity_v1'
FROM raw.customers c
LEFT JOIN raw.orders o
ON o.customer_id=c.customer_id
AND o.ordered_at >= CURRENT_TIMESTAMP - INTERVAL '30 days'
GROUP BY c.customer_id
ON CONFLICT (customer_id) DO UPDATE SET
completed_orders_30d=EXCLUDED.completed_orders_30d,
completed_value_30d=EXCLUDED.completed_value_30d,
feature_timestamp=EXCLUDED.feature_timestamp,
feature_version=EXCLUDED.feature_version;07 · WHEN A FEATURE STORE EARNS ITS COST
A feature store has its own platform cost. It becomes worthwhile when reuse, time correctness, serving consistency and governance save more than the system costs to operate.
08 · OPERATING CHECKLIST
A feature platform should reduce hidden work, not hide the work so completely that nobody can challenge its assumptions.
Test that historical rows never read future events.
Compare offline and online values for sampled entities.
Measure feature age and retrieval latency against explicit SLOs.
Watch null rate, ranges, distribution shifts and entity coverage.
Trace code, source, window, owner and models using each version.
Measure compute and storage, then retire unused features safely.
CHALLENGES
Find the leakage bug in a historical join that uses CURRENT_TIMESTAMP instead of prediction_time.
Define “customer spend in 7 days” including entity, window boundaries, event time, default and owner.
Propose an offline-to-online validation query and an acceptable mismatch threshold.
Design a backfill after the cancelled-order rule changes. Which models and datasets are affected?
Estimate monthly cost for computing 200 features for 20 million entities. State every assumption.
Argue against adopting a feature store for a small batch project, then state the condition that would change your decision.
THE CENTRAL IDEA