data orchestration aws claude-curated
Airflow is a workflow orchestrator. You define pipelines as code (Python), Airflow schedules them, runs each step, retries failures, exposes a UI to inspect what’s running. MWAA (Managed Workflows for Apache Airflow) is AWS’s hosted Airflow.
DAGs
A pipeline is a Directed Acyclic Graph (DAG) of tasks. Each task is a unit of work — a Python function, a Bash command, a Glue job trigger, an SQL query.
from airflow import DAG
from airflow.operators.python import PythonOperator
from datetime import datetime
with DAG("daily_etl", start_date=datetime(2026, 1, 1), schedule="@daily") as dag:
extract = PythonOperator(task_id="extract", python_callable=extract_fn)
transform = PythonOperator(task_id="transform", python_callable=transform_fn)
load = PythonOperator(task_id="load", python_callable=load_fn)
extract >> transform >> loadThe >> operator declares dependencies. Airflow runs extract, then transform, then load. If transform fails, load doesn’t run; transform retries based on its config; the UI shows the failure.
Why DAGs over a monolithic job
A typical monolithic ETL is one Glue job that does extract, transform, load in one script. Compare against a DAG:
| Monolithic Glue job | Airflow DAG | |
|---|---|---|
| Failure handling | Whole job fails, restart from beginning | Only failed task retries; upstream tasks don’t repeat |
| Visibility | One log per run | Per-task logs, durations, history |
| Dependencies | Linear, hard-coded | Explicit graph, parallelism free |
| Reuse | Copy/paste between jobs | Operators / TaskFlow shared across DAGs |
| Complex schedules | Cron only | Sensors, triggers, dataset events |
| Cross-system orchestration | Hard | Native (Glue + Lambda + S3 + dbt + Snowflake in one DAG) |
For one-off ETL on a single source, a Glue job is fine. The moment you have:
- Multiple data sources with dependencies between them
- Steps that should retry independently
- Workflows mixing AWS services
…a DAG-based orchestrator pays for itself.
When DAGs win on cost / performance
- Granular retries — re-running a 10-min failed task beats re-running a 4-hour monolithic job.
- Parallelism — 20 independent table extractions run in parallel as 20 tasks; in a single Glue job they’d often be sequential, or require Spark gymnastics.
- Resource matching — task A is small Python (Lambda operator, ~free), task B is heavy Spark (KubernetesPodOperator with 32 cores). Each task uses what it needs; the monolith uses the max for the duration.
MWAA specifics
MWAA = managed Airflow on AWS. AWS runs the scheduler, web server, and metadata database. You provide:
- DAGs folder in S3 — Airflow loads them on a schedule.
- Requirements file — extra Python libraries.
- Plugins — custom operators / hooks.
Pricing is per environment-hour + worker-hour. A small environment is ~$0.50/hour idle; not free, not Lambda-cheap.
MWAA gotchas
- Cold scheduler — DAG parse times affect responsiveness; keep DAG files small.
- Worker autoscaling is slower than EKS; for spiky workloads consider self-hosting on EKS instead.
- Environment upgrades are AWS-driven; minor Airflow version bumps may break custom operators.
- Connections / variables stored in Secrets Manager via the Airflow secrets backend — handier than the metadata DB for cross-environment promotion.
Operators worth knowing
PythonOperator/ TaskFlow@task— most commonBashOperator— shell commandsGlueJobOperator— trigger and wait for a Glue jobEmrServerlessStartJobOperator— EMR ServerlessS3KeySensor— wait for a file to appear in S3 before startingSqlOperator(and engine-specific variants) — run SQL on Postgres/MySQL/Snowflake/etc.KubernetesPodOperator— run anything in a custom container
Alternatives to consider
- AWS Step Functions — simpler, AWS-native, no Python. Good when DAGs are small and AWS-only. Limited Python expressiveness.
- Dagster — newer, more opinionated, asset-centric instead of task-centric.
- Prefect — modern Python-first, also asset-centric.
- Self-hosted Airflow on EKS — cheaper than MWAA at scale, more setup.