AWS Glue is a serverless ETL service that runs Apache Spark jobs on managed infrastructure. You write PySpark or Scala, Glue provisions the cluster, runs the job, tears it down. No EMR cluster to manage.
Job types
| Type | Use |
|---|---|
| Spark | Batch ETL, large transformations |
| Spark Streaming | Continuous processing from Kinesis/MSK |
| Python Shell | Lightweight scripts (no Spark), small data |
| Ray | Distributed Python (newer, ML-leaning) |
For data lake ETL, Spark is the default.
DynamicFrame vs DataFrame
Glue exposes a DynamicFrame on top of Spark’s DataFrame. Differences:
- DynamicFrame is schema-flexible — handles inconsistent schemas across records (a Glue-specific concept).
- DataFrame is standard Spark — strict schema, faster for typed operations.
In practice you convert between them: read with DynamicFrame to absorb dirty source data, convert to DataFrame for transformations, convert back to write through Glue.
df = dyf.toDF()
# transform with Spark APIs
dyf2 = DynamicFrame.fromDF(df, glueContext, "result")Glue Data Catalog
Job inputs/outputs reference tables in the Glue Data Catalog (a Hive metastore). Crawlers populate the catalog by scanning S3 paths and inferring schema. Athena, Redshift Spectrum, and EMR all read the same catalog — single source of truth.
Job parameters
Pass arguments to jobs via --KEY value flags. In code:
from awsglue.utils import getResolvedOptions
args = getResolvedOptions(sys.argv, ['JOB_NAME', 'date_partition'])Useful for parameterising the same job code across daily runs (different date_partition).
Local development
Iterating on a Glue job by deploying every change is slow. Faster loops:
Option 1 — Glue Interactive Sessions
Glue exposes a Jupyter kernel that runs against the same Glue runtime. Connect from VS Code or PyCharm via the AWS plugin and iterate against real Glue without packaging.
Option 2 — PyCharm Big Data Tools / local Spark
Run Spark locally with pyspark and the Glue libs (aws-glue-libs from GitHub). The full Spark API works locally; Glue-specific bits (DynamicFrame, glueContext) need the Glue Python library installed. Good enough for transformation logic; deploy to test the job-runner integration.
Option 3 — Docker image
AWS publishes a Glue Docker image (amazon/aws-glue-libs). Mount your script, run as if you’re in Glue. Slower than native, but exactly matches the runtime.
Common pitfalls
- DPU sizing — too few DPUs = slow, too many = wasted spend. Start at 5 G.1X for medium jobs, profile, adjust.
- Small files — writing one file per partition per record creates millions of objects. Use
coalesce()orrepartition()before write. - Schema drift — a new column in the source breaks downstream readers if the catalog isn’t updated. Crawl after every load, or use schema evolution-tolerant formats.
- Cost surprises — Glue charges per DPU-hour. A misconfigured job left running burns money. Set timeouts.