Model
Organize facts, dimensions, entities and time.
OLAP → FEATURE ENGINEERING → FEATURE STORAGE
OLAP gives feature engineering the history, scale and analytical operators it needs. Feature storage turns selected results into durable data products: reproducible for training, efficient for batch scoring and—when required—fast enough for online prediction.
Organize facts, dimensions, entities and time.
Build windows and point-in-time features at scale.
Materialize history and latest values intentionally.
Match storage to batch or online latency.
01 · REFERENCE ARCHITECTURE
The data plane moves and computes values. The control plane defines what those values mean, who owns them, how fresh they should be and which models depend on them.
02 · WHY OLAP IS A STRONG FEATURE SOURCE
OLAP is not automatically correct for ML, but its physical and logical design is closer to feature computation than an operational database is.
Facts, snapshots and slowly changing dimensions can reconstruct past states rather than overwrite them.
Orders, line items, sessions and customer-day tables state what one row represents.
Read selected columns, prune partitions and compress repeated values during large scans.
Distributed engines can aggregate many entities and time windows concurrently.
Conformed dimensions and reviewed metrics reduce repeated source interpretation.
Feature totals and coverage can be checked against governed facts before publication.
03 · HOW OLAP SEES DATA
OLAP organizes data for comparison across time, product, customer, geography, channel and scenario. Modern warehouses and lakehouses implement cube-like analysis with columnar tables and distributed engines.
time = 2026-08market = SETinstrument = GOLDcell → SUM(volume), AVG(spread)Gold activity in August across all markets.
Gold and silver, Asian venues, Q1–Q2.
Year → quarter → day → trade.
Security → sector → market.
Market by month instead of month by market.
Rolling demand, growth and volatility.
04 · FACT, DIMENSION AND GRAIN
Before aggregation, state what one row means. Daily, event-level and snapshot grains answer different questions and have very different costs.
day · week · month · year
symbol · asset_class · sector
instrument × market × observation_time
price · volume · bid · ask · open_interest
exchange · country · session
actual · forecast · base · stress
Compact for portfolio research; no intraday order-book reconstruction.
Supports microstructure; costs much more storage, ordering and compute.
Useful for exposure trends; changes between snapshots are invisible.
05 · WHERE OLAP IS USED
The same analytical model can support dashboards, investigation, planning, feature engineering, controls and audit.
Liquidity, exposure, credit, fraud, treasury and scenarios.
Sales mix, baskets, promotions, inventory and cohorts.
Yield, downtime, quality, energy and maintenance.
Traffic, congestion, quality, churn and incidents.
Capacity, waiting, pathways and cost under privacy controls.
Demand curves, generation, outage, weather and price.
Budget, delivery, geography, projects and outcomes.
Funnel, retention, experiments and unit economics.
OLAP becomes useful only after the decision is named. “Analyse customers” is too broad; “which customers are likely to stop purchasing within 30 days, evaluated every Monday?” defines an entity, observation time, horizon and action. The same discipline applies across industries.
Possible grains: account-day, instrument-minute, customer-week and basket-line. Keep monetary unit, currency, venue and event time explicit.
Model machine-cycle, production-batch, asset-hour and meter-interval grains separately. Mixing them creates false averages and hides downtime.
Network-event, subscriber-hour, session and experiment-assignment grains answer different questions. Preserve assignment time to avoid contaminating experiment features.
Encounter, patient-day, facility-week, service-request and project-month grains require privacy boundaries and slowly changing organizational dimensions.
At [observation time], for each [entity], compute [measure/window] from events available by that time, grouped by [dimensions], to support [decision] within [horizon].06 · GOLD DEMAND AND SUPPLY
Physical flows, ETF holdings, futures positioning, inventory and liquidity are related but not interchangeable. OLAP preserves their units, frequency, source and publication time before feature engineering combines them.
Categorized physical supply minus demand, retaining publication lag and revisions.
ETF holding change relative to its own history—not total physical demand.
Deferred versus nearby contracts with roll convention controlled.
Reported inventory change with availability timestamp.
A higher price does not directly prove physical demand exceeded physical supply. Expectations, currency, interest rates, liquidity and positioning also affect price formation.
Gold data arrives at incompatible grains and publication schedules: trades can be sub-second, ETF holdings daily, reported inventories daily or weekly, and mine production quarterly with revisions. Preserve each native table first, then create point-in-time joins using when a value became available—not only the period it describes.
At quarterly grain, classify mine production and recycling as supply; jewellery, technology, bar/coin investment and official-sector net purchases as demand. Preserve tonnes and revision vintage.
Convert fund holdings to a consistent metal unit, difference by report date, aggregate after source alignment, then standardize against trailing history.
Choose nearby and deferred contracts under a documented roll calendar. Never stitch contract codes without a roll rule.
Use location, inventory category, unit, report date and available_at. Store corrections under a new data vintage so training remains reproducible.
mine = 900 t, recycling = 300 t
jewellery = 520 t, technology = 80 t
investment = 410 t, official = 140 t
physical_balance = (900 + 300) - (520 + 80 + 410 + 140) = +50 t
Meaning: +50 t under this classification and vintage. Verify that the report was published before the observation time.07 · STOCK DEMAND, SUPPLY AND LIQUIDITY
Structural supply relates to shares and free float. Executable demand and supply appear as bids and asks. Trades show matched activity, not every intention.
Relative displayed bid and ask size, aligned by venue and time.
Quoted spread divided by mid-price; one liquidity descriptor.
Volume relative to corporate-action-adjusted free float.
Past return dispersion; not guaranteed future risk.
A stock has several “supply” concepts. Shares outstanding describe issued ownership units; free float estimates shares more available to trade; the ask book shows current displayed sell orders; and executed volume records only matched trades. They belong to different fact tables and time grains.
Store effective_from/effective_to for shares outstanding and float. Adjust historical denominators for splits, reverse splits, rights, buybacks and new issuance.
Order-book features require venue, sequence number and event time. Orders can be cancelled or hidden; displayed depth is an observable intention, not guaranteed demand.
A trade indicates that a buyer and seller matched. It does not reveal every unexecuted intention. Aggregate by instrument–venue–interval before comparing markets.
Use corporate-action-adjusted prices and a declared return interval. Missing intervals, market closures and thin trading change the meaning of volatility.
best_bid = 101.50, bid_size = 4,300
best_ask = 102.00, ask_size = 3,100
mid = (101.50 + 102.00) / 2 = 101.75
relative_spread = 0.50 / 101.75 = 0.4914%
book_imbalance_l1 = (4,300 - 3,100) / (4,300 + 3,100) = 0.1622
Meaning: displayed L1 size tilts toward bids at this instant. This does not prove price must rise; orders may cancel, hidden liquidity may exist, and deeper levels may reverse the picture.08 · OTHER FINANCIAL PATTERNS
These are data-product and feature examples, not profit forecasts or investment recommendations.
net_exposure_7d · basis_change · cash_gap
utilization_3m · payment_ratio · missed_count_12m
sector_weight · duration · concentration · drawdown
velocity_1h · new_device_7d · circular_flow_score
claim_frequency · loss_ratio · reporting_delay
surprise_vs_consensus · revision_size · trend_3m
09 · SQL EXAMPLE: DAILY MARKET FEATURES
Real data differs by venue, license, session, corporate-action convention and timestamp precision.
WITH daily AS (
SELECT instrument_key, trading_date,
MAX(adjusted_close) AS close,
SUM(volume) AS volume,
AVG((ask-bid)/NULLIF((ask+bid)/2,0)) AS relative_spread
FROM analytics.fact_market_observation
WHERE available_at <= trading_date + INTERVAL '1 day'
GROUP BY instrument_key, trading_date
), returns AS (
SELECT *, LN(close/NULLIF(LAG(close) OVER
(PARTITION BY instrument_key ORDER BY trading_date),0)) AS return_1d
FROM daily
)
SELECT instrument_key, trading_date AS feature_date,
volume, relative_spread, return_1d,
AVG(volume) OVER (PARTITION BY instrument_key ORDER BY trading_date
ROWS BETWEEN 19 PRECEDING AND CURRENT ROW) AS avg_volume_20obs,
STDDEV_SAMP(return_1d) OVER (PARTITION BY instrument_key ORDER BY trading_date
ROWS BETWEEN 19 PRECEDING AND CURRENT ROW) AS volatility_20obs
FROM returns;
Define adjustment, holidays, missing prices, listings, delistings, sessions, revisions and licensing. Valid SQL can still produce an invalid data product.
03 · BUILD A FEATURE SPINE
The entity spine anchors one entity at one prediction time. Every feature must join to that row without using information arriving later.
1012026-08-01 09:002026-08-081012026-08-15 09:002026-08-22[orders_30d,
spend_90d,
days_since_last_order,
segment_v3]04 · FEATURE ENGINEERING LAYERS
Not every transformation belongs in one giant feature query. Layers make ownership, testing, reuse and cost visible.
Immutable or replayable events, source timestamps and ingestion metadata.
Types, deduplication, keys, late-event policy and quality flags.
Facts, dimensions, entity-day aggregates and reviewed business rules.
Point-in-time windows, encodings, defaults and feature versions.
Historical offline rows, batch snapshots and latest online values.
05 · SQL LAB: MATERIALIZE FEATURES
A table built incrementally is one practical offline feature store pattern. The exact syntax depends on the warehouse or lakehouse platform.
CREATE TABLE IF NOT EXISTS features.customer_activity_daily (
customer_id BIGINT NOT NULL,
feature_date DATE NOT NULL,
orders_30d INTEGER NOT NULL,
spend_30d DECIMAL(14,2) NOT NULL,
days_since_order INTEGER,
feature_version VARCHAR(30) NOT NULL,
computed_at TIMESTAMP NOT NULL,
PRIMARY KEY (customer_id, feature_date, feature_version)
);
-- Build only the affected feature_date partition.
INSERT INTO features.customer_activity_daily
SELECT
c.customer_id,
:feature_date,
COUNT(o.order_id),
COALESCE(SUM(o.net_amount), 0),
DATE_PART('day', :feature_date - MAX(o.order_date)),
'customer_activity_v1',
CURRENT_TIMESTAMP
FROM analytics.dim_customer c
LEFT JOIN analytics.fact_order o
ON o.customer_id = c.customer_id
AND o.order_date >= :feature_date - INTERVAL '30 days'
AND o.order_date < :feature_date
AND o.is_completed = TRUE
GROUP BY c.customer_id
ON CONFLICT (customer_id, feature_date, feature_version)
DO UPDATE SET
orders_30d=EXCLUDED.orders_30d,
spend_30d=EXCLUDED.spend_30d,
days_since_order=EXCLUDED.days_since_order,
computed_at=EXCLUDED.computed_at;Without observation time, a value such as spend_30d cannot be reconstructed or compared. Storing only the latest value may support online prediction, but it cannot reproduce historical training sets.
06 · WHY MATERIALIZED FEATURES CAN OUTRUN A VIEW
A normal logical VIEW generally stores SQL, not results. Each query may repeat joins, filters, windows and aggregation. A materialized feature table pays that work during the build and makes later reads narrower and more predictable.
Materialized features can be dramatically faster than a complex logical view when they avoid repeated scans and joins, are partitioned for the access pattern, and remain small enough to read efficiently. They are not universally faster: a database may rewrite or cache a simple view well, the materialized table may be stale or poorly clustered, and building unused features wastes more than it saves. Benchmark the actual query, data volume, concurrency and freshness target.
07 · PERFORMANCE LEVERS
Precomputation alone is not an architecture. The stored result must align with the dominant access path and operational constraints.
Partition historical features by feature_date or another dominant time boundary so training reads skip unrelated history.
Co-locate rows by entity and time to reduce scanned blocks during point-in-time retrieval.
Read only selected feature columns; avoid very wide tables when models use disjoint feature groups.
Recompute affected dates and entities rather than full history; define how late events reopen old partitions.
Maintain customer-day or entity-hour intermediates so many rolling windows reuse smaller inputs.
For online inference, materialize only the latest approved vector keyed by entity, with atomic update and TTL where appropriate.
Cache helps hot repeated reads but needs invalidation, capacity and correctness rules; it is not a substitute for historical storage.
Avoid many tiny files in a lakehouse; compaction improves metadata and scan efficiency.
08 · STORE EACH FEATURE ON PURPOSE
The definition should remain one governed concept even when history, batch snapshots and online values use different storage engines.
Full point-in-time history for training, evaluation, audit and backfill.
customer_id, feature_date, value, versionFrozen input for one campaign or scheduled scoring run, improving reproducibility.
run_id, customer_id, feature_vectorCompact, low-latency lookup for a prediction request; history remains offline.
customer_id → {features, timestamp, version}09 · QUALITY, FRESHNESS AND COST
Materialization creates a new stateful data product. It must be monitored as carefully as the source and the model.
10 · DECISION GUIDE
Choose the simplest representation that satisfies correctness, reuse, freshness, latency and recovery.
Use when freshness matters, source plans are efficient and repeated compute is inexpensive.
Use for repeatable training, heavy windows, many models or stable batch scoring.
Use when a request cannot wait for warehouse computation and latest values can be synchronized reliably.
Build reusable entity-time summaries before creating hundreds of near-duplicate features.
THE CENTRAL IDEA