← warin.me · Data Science and Engineeringภาษาไทย

OLAP → FEATURE ENGINEERING → FEATURE STORAGE

Compute with history. Store with intent. Serve without repeating the expensive work.

OLAP gives feature engineering the history, scale and analytical operators it needs. Feature storage turns selected results into durable data products: reproducible for training, efficient for batch scoring and—when required—fast enough for online prediction.

01

Model

Organize facts, dimensions, entities and time.

02

Compute

Build windows and point-in-time features at scale.

03

Store

Materialize history and latest values intentionally.

04

Serve

Match storage to batch or online latency.

01 · REFERENCE ARCHITECTURE

The feature path has a data plane and a control plane.

The data plane moves and computes values. The control plane defines what those values mean, who owns them, how fresh they should be and which models depend on them.

OLAP feature engineering and storage reference architectureOperational and event data enter OLAP storage, are modeled into facts and dimensions, transformed into point-in-time features, materialized in offline feature tables and optionally synchronized to an online store. A registry, quality, lineage and orchestration control the whole path. DATA PLANE · VALUES FLOW LEFT TO RIGHT SourcesOLTP · events · APIsfiles · streamsCDC / ELT OLAP Storagewarehouse · lakehousecolumnar files · partitionshistorical snapshotshistory retained Analytical Modelfacts · dimensionsentity spine · event timecleaned business grainreusable layer Feature Computewindows · aggregatesPIT joins · encodingbatch / streamtested definitions Offlinefeature tablehistorytraining Training / Batchdataset · model · scoring Online Storelatest by keyOnline InferenceAPI · low-latency lookup CONTROL PLANE · MEANING, QUALITY AND OPERATIONSRegistry & ownershipSchema & versionsLineage & catalogQuality & driftOrchestrationCost & SLO
Figure 1. OLAP supplies historical analytical structure. Feature computation converts that structure into model inputs; materialized stores match those inputs to training and serving workloads.

02 · WHY OLAP IS A STRONG FEATURE SOURCE

Features need history, broad scans and stable analytical grain.

OLAP is not automatically correct for ML, but its physical and logical design is closer to feature computation than an operational database is.

01

Retained history

Facts, snapshots and slowly changing dimensions can reconstruct past states rather than overwrite them.

02

Analytical grain

Orders, line items, sessions and customer-day tables state what one row represents.

03

Columnar execution

Read selected columns, prune partitions and compress repeated values during large scans.

04

Parallel compute

Distributed engines can aggregate many entities and time windows concurrently.

05

Shared business layer

Conformed dimensions and reviewed metrics reduce repeated source interpretation.

06

Reconciliation

Feature totals and coverage can be checked against governed facts before publication.

03 · HOW OLAP SEES DATA

A transaction becomes a measure observed through several dimensions.

OLAP organizes data for comparison across time, product, customer, geography, channel and scenario. Modern warehouses and lakehouses implement cube-like analysis with columnar tables and distributed engines.

OLAP cube with Time, Market and Instrument dimensions An isometric analytical cube. A highlighted cell represents one aggregate at the intersection of one time period, one market and one instrument. TIMEMARKETINSTRUMENT ONE CELLone coordinate → one aggregate
MEASURES volume · value · count · return · spread One cell is an aggregation at one coordinate.
time = 2026-08market = SETinstrument = GOLDcell → SUM(volume), AVG(spread)
SLICE

Fix one dimension

Gold activity in August across all markets.

DICE

Select a sub-cube

Gold and silver, Asian venues, Q1–Q2.

DRILL-DOWN

Move into detail

Year → quarter → day → trade.

ROLL-UP

Summarize upward

Security → sector → market.

PIVOT

Change viewpoint

Market by month instead of month by market.

WINDOW

Compare through time

Rolling demand, growth and volatility.

04 · FACT, DIMENSION AND GRAIN

The fact says what happened; dimensions explain the context.

Before aggregation, state what one row means. Daily, event-level and snapshot grains answer different questions and have very different costs.

DIM_DATE day · week · month · year
DIM_INSTRUMENT symbol · asset_class · sector
FACT_MARKET_OBSERVATION instrument × market × observation_time
price · volume · bid · ask · open_interest
DIM_MARKET exchange · country · session
DIM_SCENARIO actual · forecast · base · stress
instrument × day

Compact for portfolio research; no intraday order-book reconstruction.

instrument × venue × event_time

Supports microstructure; costs much more storage, ordering and compute.

account × month_end

Useful for exposure trends; changes between snapshots are invisible.

05 · WHERE OLAP IS USED

OLAP supports any work that compares several axes before a decision.

The same analytical model can support dashboards, investigation, planning, feature engineering, controls and audit.

Finance & markets

Liquidity, exposure, credit, fraud, treasury and scenarios.

Retail

Sales mix, baskets, promotions, inventory and cohorts.

Manufacturing

Yield, downtime, quality, energy and maintenance.

Telecom

Traffic, congestion, quality, churn and incidents.

Health operations

Capacity, waiting, pathways and cost under privacy controls.

ϟ

Energy

Demand curves, generation, outage, weather and price.

Government

Budget, delivery, geography, projects and outcomes.

Digital products

Funnel, retention, experiments and unit economics.

05 expanded · Translate a business question into grain, measures and reusable features.

OLAP becomes useful only after the decision is named. “Analyse customers” is too broad; “which customers are likely to stop purchasing within 30 days, evaluated every Monday?” defines an entity, observation time, horizon and action. The same discipline applies across industries.

FINANCE × RETAIL

Exposure, liquidity and customer behaviour

Possible grains: account-day, instrument-minute, customer-week and basket-line. Keep monetary unit, currency, venue and event time explicit.

  • Measures: balance, notional, P&L, volume, net sales, margin and inventory.
  • Features: utilization_30d, liquidity_gap, basket_diversity_90d, promo_dependency.
  • Decisions: credit review, treasury allocation, replenishment and targeted retention.
MANUFACTURING × ENERGY

Yield, reliability and resource intensity

Model machine-cycle, production-batch, asset-hour and meter-interval grains separately. Mixing them creates false averages and hides downtime.

  • Measures: good units, scrap, cycle time, kWh, temperature and outage minutes.
  • Features: rolling_yield_20_batches, vibration_rms_1h, energy_per_good_unit, outage_rate_30d.
  • Decisions: maintenance window, quality hold, load scheduling and anomaly investigation.
TELECOM × DIGITAL PRODUCT

Service quality, journeys and experiments

Network-event, subscriber-hour, session and experiment-assignment grains answer different questions. Preserve assignment time to avoid contaminating experiment features.

  • Measures: throughput, latency, dropped sessions, active users, conversion and revenue.
  • Features: congestion_minutes_24h, failed_session_ratio, funnel_velocity, retention_28d.
  • Decisions: capacity action, incident triage, churn intervention and product rollout.
HEALTH × GOVERNMENT

Capacity, pathways and accountable outcomes

Encounter, patient-day, facility-week, service-request and project-month grains require privacy boundaries and slowly changing organizational dimensions.

  • Measures: waiting time, occupancy, completed service, budget used and outcome count.
  • Features: wait_p90_7d, pathway_delay, budget_burn_rate, geographic_coverage_gap.
  • Decisions: staffing, referral, service prioritization and programme review—not automated clinical judgement.

A reusable design sentence

At [observation time], for each [entity], compute [measure/window] from events available by that time, grouped by [dimensions], to support [decision] within [horizon].

06 · GOLD DEMAND AND SUPPLY

“Gold demand” is not one column; it is a family of flows across different markets.

Physical flows, ETF holdings, futures positioning, inventory and liquidity are related but not interchangeable. OLAP preserves their units, frequency, source and publication time before feature engineering combines them.

PHYSICAL SUPPLY Mine production country × quarter × tonnes Recycling region × quarter × tonnes
MARKET & INVENTORY Spot / futures currency × venue × timestamp ETF / inventory fund/location × report_date
PHYSICAL DEMAND Jewellery & technology region/sector × quarter × tonnes Investment & official category × period
FEATURE 01

physical_balance_4q

Categorized physical supply minus demand, retaining publication lag and revisions.

FEATURE 02

etf_flow_zscore_20d

ETF holding change relative to its own history—not total physical demand.

FEATURE 03

futures_curve_slope

Deferred versus nearby contracts with roll convention controlled.

FEATURE 04

inventory_change_5d

Reported inventory change with availability timestamp.

Do not collapse the mechanisms

A higher price does not directly prove physical demand exceeded physical supply. Expectations, currency, interest rates, liquidity and positioning also affect price formation.

06 expanded · Build gold features without pretending every flow is the same market.

Gold data arrives at incompatible grains and publication schedules: trades can be sub-second, ETF holdings daily, reported inventories daily or weekly, and mine production quarterly with revisions. Preserve each native table first, then create point-in-time joins using when a value became available—not only the period it describes.

01 · PHYSICAL BALANCE

physical_balance_4q

At quarterly grain, classify mine production and recycling as supply; jewellery, technology, bar/coin investment and official-sector net purchases as demand. Preserve tonnes and revision vintage.

  • balance = mine + recycling − jewellery − technology − investment − official
  • Positive means a categorized surplus under this definition, not an automatic price forecast.
02 · ETF FLOW

etf_flow_zscore_20d

Convert fund holdings to a consistent metal unit, difference by report date, aggregate after source alignment, then standardize against trailing history.

  • flow_t = holding_t − holding_t−1
  • z = (flow_t − mean(flow_20d)) / std(flow_20d)
  • A high z-score means unusual ETF flow, not total gold demand.
03 · FUTURES CURVE

futures_curve_slope

Choose nearby and deferred contracts under a documented roll calendar. Never stitch contract codes without a roll rule.

  • slope = (P_deferred / P_nearby − 1) / year_fraction
  • The curve can reflect funding, storage, convenience yield, positioning and liquidity.
04 · REPORTED INVENTORY

inventory_change_5d

Use location, inventory category, unit, report date and available_at. Store corrections under a new data vintage so training remains reproducible.

  • change_5d = inventory_t − inventory_t−5
  • Reported inventory is one observable stock, not all metal available globally.

Worked quarterly example

mine = 900 t, recycling = 300 t jewellery = 520 t, technology = 80 t investment = 410 t, official = 140 t physical_balance = (900 + 300) - (520 + 80 + 410 + 140) = +50 t Meaning: +50 t under this classification and vintage. Verify that the report was published before the observation time.
DO COMPAREFlows with matching unit, period, scope and publication availability.
DO NOT EQUATEETF flow, futures volume, mine output and physical consumption.
KEEP AS CONTEXTUSD, real rates, inflation expectations, volatility and market liquidity.

07 · STOCK DEMAND, SUPPLY AND LIQUIDITY

Shares outstanding, volume and order-book supply are different analytical layers.

Structural supply relates to shares and free float. Executable demand and supply appear as bids and asks. Trades show matched activity, not every intention.

SELL / ASK 103.00 — 8,200 102.50 — 5,500 102.00 — 3,100
101.75 mid-price · spread 0.50
BUY / BID 101.50 — 4,300 101.00 — 6,800 100.50 — 9,100
A stylized order-book snapshot. Displayed depth is dynamic and is not total investor demand.
FEATURE 01

book_imbalance_l1

Relative displayed bid and ask size, aligned by venue and time.

FEATURE 02

relative_spread

Quoted spread divided by mid-price; one liquidity descriptor.

FEATURE 03

turnover_20d

Volume relative to corporate-action-adjusted free float.

FEATURE 04

realized_volatility_20d

Past return dispersion; not guaranteed future risk.

07 expanded · Separate ownership supply, displayed liquidity and executed activity.

A stock has several “supply” concepts. Shares outstanding describe issued ownership units; free float estimates shares more available to trade; the ask book shows current displayed sell orders; and executed volume records only matched trades. They belong to different fact tables and time grains.

STRUCTURAL SUPPLY

Shares and free float

Store effective_from/effective_to for shares outstanding and float. Adjust historical denominators for splits, reverse splits, rights, buybacks and new issuance.

  • turnover = adjusted_volume / effective_free_float
  • Do not divide today’s volume by a future or stale share count.
DISPLAYED LIQUIDITY

Bid, ask, depth and spread

Order-book features require venue, sequence number and event time. Orders can be cancelled or hidden; displayed depth is an observable intention, not guaranteed demand.

  • mid = (best_ask + best_bid) / 2
  • relative_spread = (best_ask − best_bid) / mid
EXECUTED ACTIVITY

Trades and volume

A trade indicates that a buyer and seller matched. It does not reveal every unexecuted intention. Aggregate by instrument–venue–interval before comparing markets.

  • VWAP = Σ(price × quantity) / Σ(quantity)
  • trade_intensity = trades / active_seconds
RISK CONTEXT

Return and realized volatility

Use corporate-action-adjusted prices and a declared return interval. Missing intervals, market closures and thin trading change the meaning of volatility.

  • r_t = ln(P_t / P_t−1)
  • realized_volatility = sqrt(Σ r_t²)

Worked order-book snapshot

best_bid = 101.50, bid_size = 4,300 best_ask = 102.00, ask_size = 3,100 mid = (101.50 + 102.00) / 2 = 101.75 relative_spread = 0.50 / 101.75 = 0.4914% book_imbalance_l1 = (4,300 - 3,100) / (4,300 + 3,100) = 0.1622 Meaning: displayed L1 size tilts toward bids at this instant. This does not prove price must rise; orders may cancel, hidden liquidity may exist, and deeper levels may reverse the picture.
POINT-IN-TIME KEYinstrument + venue + event_time + sequence
CORPORATE ACTION KEYinstrument + effective_from + adjustment_version
MODEL SAFEGUARDAlign every feature to information observable before prediction time.

08 · OTHER FINANCIAL PATTERNS

The same OLAP architecture supports many financial questions.

These are data-product and feature examples, not profit forecasts or investment recommendations.

FX & TREASURY net_exposure_7d · basis_change · cash_gap
CREDIT utilization_3m · payment_ratio · missed_count_12m
PORTFOLIO & RISK sector_weight · duration · concentration · drawdown
FRAUD & AML velocity_1h · new_device_7d · circular_flow_score
INSURANCE claim_frequency · loss_ratio · reporting_delay
MACRO DATA surprise_vs_consensus · revision_size · trend_3m

09 · SQL EXAMPLE: DAILY MARKET FEATURES

Materialize windows after declaring adjustment and availability rules.

Real data differs by venue, license, session, corporate-action convention and timestamp precision.

WITH daily AS (
 SELECT instrument_key, trading_date,
        MAX(adjusted_close) AS close,
        SUM(volume) AS volume,
        AVG((ask-bid)/NULLIF((ask+bid)/2,0)) AS relative_spread
 FROM analytics.fact_market_observation
 WHERE available_at <= trading_date + INTERVAL '1 day'
 GROUP BY instrument_key, trading_date
), returns AS (
 SELECT *, LN(close/NULLIF(LAG(close) OVER
   (PARTITION BY instrument_key ORDER BY trading_date),0)) AS return_1d
 FROM daily
)
SELECT instrument_key, trading_date AS feature_date,
       volume, relative_spread, return_1d,
       AVG(volume) OVER (PARTITION BY instrument_key ORDER BY trading_date
         ROWS BETWEEN 19 PRECEDING AND CURRENT ROW) AS avg_volume_20obs,
       STDDEV_SAMP(return_1d) OVER (PARTITION BY instrument_key ORDER BY trading_date
         ROWS BETWEEN 19 PRECEDING AND CURRENT ROW) AS volatility_20obs
FROM returns;
Before publication

Define adjustment, holidays, missing prices, listings, delistings, sessions, revisions and licensing. Valid SQL can still produce an invalid data product.

03 · BUILD A FEATURE SPINE

First define the rows the model is allowed to learn from.

The entity spine anchors one entity at one prediction time. Every feature must join to that row without using information arriving later.

ENTITY SPINE
customer_idprediction_timelabel_time
1012026-08-01 09:002026-08-08
1012026-08-15 09:002026-08-22
+
ELIGIBLE HISTORY
t − 90dfeature windowtlabel window
FEATURE VECTOR[orders_30d,
spend_90d,
days_since_last_order,
segment_v3]
The contract of one rowEntity + prediction time + feature version determine the observation. Label time belongs after prediction time and must not leak into the features.

04 · FEATURE ENGINEERING LAYERS

Separate reusable evidence from model-specific interpretation.

Not every transformation belongs in one giant feature query. Layers make ownership, testing, reuse and cost visible.

1RAW / BRONZE

Preserve source evidence

Immutable or replayable events, source timestamps and ingestion metadata.

2CLEAN / SILVER

Repair representation

Types, deduplication, keys, late-event policy and quality flags.

3BUSINESS / GOLD

Publish analytical meaning

Facts, dimensions, entity-day aggregates and reviewed business rules.

4FEATURE

Compute model evidence

Point-in-time windows, encodings, defaults and feature versions.

5STORAGE

Materialize for access

Historical offline rows, batch snapshots and latest online values.

05 · SQL LAB: MATERIALIZE FEATURES

Move expensive computation from every read into a controlled build.

A table built incrementally is one practical offline feature store pattern. The exact syntax depends on the warehouse or lakehouse platform.

CREATE TABLE IF NOT EXISTS features.customer_activity_daily (
  customer_id        BIGINT NOT NULL,
  feature_date       DATE NOT NULL,
  orders_30d         INTEGER NOT NULL,
  spend_30d          DECIMAL(14,2) NOT NULL,
  days_since_order   INTEGER,
  feature_version    VARCHAR(30) NOT NULL,
  computed_at        TIMESTAMP NOT NULL,
  PRIMARY KEY (customer_id, feature_date, feature_version)
);

-- Build only the affected feature_date partition.
INSERT INTO features.customer_activity_daily
SELECT
  c.customer_id,
  :feature_date,
  COUNT(o.order_id),
  COALESCE(SUM(o.net_amount), 0),
  DATE_PART('day', :feature_date - MAX(o.order_date)),
  'customer_activity_v1',
  CURRENT_TIMESTAMP
FROM analytics.dim_customer c
LEFT JOIN analytics.fact_order o
  ON o.customer_id = c.customer_id
 AND o.order_date >= :feature_date - INTERVAL '30 days'
 AND o.order_date <  :feature_date
 AND o.is_completed = TRUE
GROUP BY c.customer_id
ON CONFLICT (customer_id, feature_date, feature_version)
DO UPDATE SET
  orders_30d=EXCLUDED.orders_30d,
  spend_30d=EXCLUDED.spend_30d,
  days_since_order=EXCLUDED.days_since_order,
  computed_at=EXCLUDED.computed_at;
Why store feature_date?

Without observation time, a value such as spend_30d cannot be reconstructed or compared. Storing only the latest value may support online prediction, but it cannot reproduce historical training sets.

06 · WHY MATERIALIZED FEATURES CAN OUTRUN A VIEW

Compute once, read many—when the workload actually repeats.

A normal logical VIEW generally stores SQL, not results. Each query may repeat joins, filters, windows and aggregation. A materialized feature table pays that work during the build and makes later reads narrower and more predictable.

LOGICAL VIEW · COMPUTE ON READ
request 1
scan + join + window + aggregate
result
request 2
scan + join + window + aggregate
result
MATERIALIZED FEATURE · COMPUTE ON WRITE
scheduled / incremental build
stored feature rows
partitioned · indexed · compact
↗ request 1→ request 2↘ request 3
Figure 2. Materialization can reduce repeated compute and latency variance. It exchanges read speed for storage, build time, orchestration and freshness management.
DimensionLogical VIEWMaterialized feature tableOnline feature store
Work per readMay repeat full planFilter/project stored rowsKey lookup
Typical goalFresh abstractionTraining and batch throughputPredictable low latency
FreshnessReads current sourcesAs of last successful buildAs of latest sync/update
Main costRepeated computeStorage + build + backfillPlatform + sync + availability
Best fitLogic is light or reads are rareExpensive features are reusedPrediction has strict latency
A precise performance claim

Materialized features can be dramatically faster than a complex logical view when they avoid repeated scans and joins, are partitioned for the access pattern, and remain small enough to read efficiently. They are not universally faster: a database may rewrite or cache a simple view well, the materialized table may be stale or poorly clustered, and building unused features wastes more than it saves. Benchmark the actual query, data volume, concurrency and freshness target.

07 · PERFORMANCE LEVERS

Storage design determines whether materialization pays off.

Precomputation alone is not an architecture. The stored result must align with the dominant access path and operational constraints.

01

Partition pruning

Partition historical features by feature_date or another dominant time boundary so training reads skip unrelated history.

02

Clustering / sorting

Co-locate rows by entity and time to reduce scanned blocks during point-in-time retrieval.

03

Columnar projection

Read only selected feature columns; avoid very wide tables when models use disjoint feature groups.

04

Incremental build

Recompute affected dates and entities rather than full history; define how late events reopen old partitions.

05

Pre-aggregation

Maintain customer-day or entity-hour intermediates so many rolling windows reuse smaller inputs.

06

Key-value serving

For online inference, materialize only the latest approved vector keyed by entity, with atomic update and TTL where appropriate.

07

Cache deliberately

Cache helps hot repeated reads but needs invalidation, capacity and correctness rules; it is not a substitute for historical storage.

08

Compact files

Avoid many tiny files in a lakehouse; compaction improves metadata and scan efficiency.

08 · STORE EACH FEATURE ON PURPOSE

One feature may have several physical representations.

The definition should remain one governed concept even when history, batch snapshots and online values use different storage engines.

HISTORICAL OFFLINE

entity × observation_time

Full point-in-time history for training, evaluation, audit and backfill.

customer_id, feature_date, value, version
BATCH SNAPSHOT

entity × scoring_run

Frozen input for one campaign or scheduled scoring run, improving reproducibility.

run_id, customer_id, feature_vector
ONLINE LATEST

entity → latest vector

Compact, low-latency lookup for a prediction request; history remains offline.

customer_id → {features, timestamp, version}

09 · QUALITY, FRESHNESS AND COST

A fast wrong feature is merely a faster incident.

Materialization creates a new stateful data product. It must be monitored as carefully as the source and the model.

CorrectnessUniqueness · null policy · ranges · reconciliation · PIT leakage tests
FreshnessSource delay · build completion · online sync age · stale-read behavior
ParityOffline vs online values · shared code/definition · sampled comparison
PerformanceBuild duration · scan bytes · query p95/p99 · online lookup latency
CostCompute per build · storage per version · backfill cost · unused feature spend
OperationsOwner · lineage · incident runbook · rollback · deprecation date

10 · DECISION GUIDE

Materialize because evidence justifies it—not because storage feels safer.

Choose the simplest representation that satisfies correctness, reuse, freshness, latency and recovery.

KEEP AS VIEW

Light logic, occasional reads

Use when freshness matters, source plans are efficient and repeated compute is inexpensive.

MATERIALIZE OFFLINE

Expensive, reused history

Use for repeatable training, heavy windows, many models or stable batch scoring.

ADD ONLINE STORE

Strict prediction latency

Use when a request cannot wait for warehouse computation and latest values can be synchronized reliably.

PRE-AGGREGATE FIRST

Many related windows

Build reusable entity-time summaries before creating hundreds of near-duplicate features.

THE CENTRAL IDEA

OLAP makes history computable. Feature storage makes selected computations reusable and fast—at the cost of state that must be operated.