Install Docker
Install Docker Desktop or Docker Engine with Compose V2. Allocate at least 4 GB RAM.
DATA ENGINEERING · WORKFLOW ORCHESTRATION
See a data pipeline think.
Apache Airflow turns a workflow into a directed acyclic graph: each task has an observable state, dependencies decide what may run next, and the scheduler coordinates execution without hiding the evidence.
CLASSROOM LOGIN
warin/Airflow202601DAGDescribe dependencies
02SchedulerFind runnable tasks
03ExecutorRun work
04MetadataPreserve evidence
INTERACTIVE DAG SIMULATOR
Advance one scheduler decision at a time
This browser simulation does not run Airflow. It isolates the dependency logic before students open the real UI in Docker.
extractsource
→
validatequality gate
↗↘
summarizeTaskFlow
find_bestTaskFlow
↘↗
loadsink
pendingrunningsuccess
REAL AIRFLOW · DOCKER COMPOSE
From simulation to an observable system
The package contains Compose, PostgreSQL, LocalExecutor, two Airflow 3 DAGs, start/stop scripts and a bilingual-ready teaching README.
Initialize once
docker compose up airflow-initWait until the initialization container exits with code 0.
Start the lab
docker compose up -dhttp://localhost:8080 · airflow / airflow
Observe evidence
Trigger teaching_etl; inspect Graph, Grid, task logs, retries and XCom values.
Live UI and downloadable labhttps://warin.me/airflow/ · warin / Airflow2026
SYSTEM ARCHITECTURE
A DAG is code; orchestration is a conversation among services.
Parses Python files and serializes workflow structure. A syntax error can prevent a DAG from appearing.
Evaluates dependencies and state, then submits eligible task instances to the executor.
Runs multiple task processes on one host—appropriate for this teaching package, not a distributed cluster.
Stores DAG runs, task states, schedules, connections and XCom metadata—the evidence behind the UI.
Serves the web interface and API on port 8080. Do not expose the classroom credentials publicly.
Efficiently waits for deferrable work without occupying a worker slot.
MANUAL DAILY INGESTION · OPEN-METEO → POSTGRESQL
Load one chosen day—without cron
Trigger the DAG manually, set data_date, and observe Extract → Validate → Load. An UPSERT makes rerunning the same date safe.
@dag(
dag_id="daily_weather_to_postgres",
schedule=None, # manual trigger only
params={"data_date": Param("2026-08-01", format="date")},
)
def daily_weather_to_postgres():
row = validate(extract()) # download one day from Open-Meteo
load(row) # UPSERT into teaching_daily_weatherReady-to-run DAG sourceChoose a completed historical date in the Trigger DAG form.
AIRFLOW 3 · TASKFLOW API
The dependency graph is created by calling functions
Return values become XCom references; calling a decorated function defines a task instead of immediately executing ordinary Python.
from airflow.sdk import dag, task
@dag(schedule=None, start_date=..., catchup=False)
def teaching_etl():
@task
def extract() -> list[dict]: ...
@task
def summarize(rows: list[dict]) -> dict: ...
rows = extract()
summarize(rows) # XComArg wires the dependency
teaching_etl()TEACHING SEQUENCE
Six experiments—from graph reading to operations
Predict the next runnable task before pressing Step.
Change data and inspect the XCom lineage.
Force one failure and explain retries.
Change the branch score and identify skipped tasks.
Add a cron schedule and explain logical date versus wall-clock time.
Diagnose a missing DAG using parser errors and container logs.
The included credentials and secret are intentionally simple. Keep the service on localhost. Production Airflow needs proper secrets, TLS, authentication, backups, monitoring and an intentionally designed executor/deployment model.