data bigdata aws claude-curated
S3 is the de-facto storage layer for cloud data lakes — durable, infinitely scalable object store, with separate compute engines (Athena, EMR, Redshift Spectrum, Glue) reading directly from it. See also Data Lakes for the general concept.
Why S3 for a data lake
- Decoupled storage and compute — pay for each separately, scale independently.
- Open formats — Parquet, ORC, Avro readable by any engine. No vendor lock-in like a warehouse.
- Lifecycle policies — automatic tiering to cheaper storage for cold data.
Storage classes for lifecycle
Hot, warm, and cold data live in the same bucket with different storage classes. Lifecycle rules transition objects automatically.
| Class | Use | Retrieval |
|---|---|---|
| S3 Standard | Recent / queried daily | Immediate |
| S3 Intelligent-Tiering | Unknown access pattern | Immediate |
| S3 Standard-IA | Queried weekly/monthly | Immediate (per-GB retrieval fee) |
| S3 Glacier Instant Retrieval | Quarterly access | Milliseconds |
| S3 Glacier Flexible Retrieval | Yearly access | Minutes to hours |
| S3 Glacier Deep Archive | Compliance / never read | 12–48 hours |
Typical pattern: keep last 90 days in Standard, transition to Glacier Instant for 1 year, Deep Archive thereafter.
Partitioning
Lay data out so query engines can skip files based on the predicate. Hive-style partitioning is the convention:
s3://bucket/events/year=2026/month=04/day=29/events-001.parquet
A query with WHERE year=2026 AND month=04 reads only that prefix — partition pruning. Without partitions, Athena scans every file in the bucket.
Pick partitions by what the queries filter on most (date is almost always one). Avoid high-cardinality partitions (e.g. user_id) — they create millions of tiny files and tank performance.
File formats
| Format | Type | Best for |
|---|---|---|
| CSV / JSON | Row-based, text | Raw landing only |
| Parquet | Columnar, binary | Analytics — column pruning, compression |
| ORC | Columnar, binary | Hive ecosystem, similar to Parquet |
| Avro | Row-based, binary | Streaming, schema evolution friendly |
For analytics, Parquet is the default. It compresses well (Snappy/Zstd), supports predicate pushdown, and is read natively by Athena/Spark/DuckDB.
Schema evolution
Schema drift is inevitable — new columns get added, types change. Strategies:
- Glue Data Catalog stores the table schema; crawlers detect new columns automatically (with caveats — type changes can break queries).
- Parquet handles additive changes well — readers ignore unknown columns, missing columns return null.
- Avoid breaking changes (renames, type narrowing) — write a new table version instead.
Medallion / multi-layer pattern
Common organisational pattern even on S3:
- Raw / bronze — exact copy of source, no transformation. Audit trail, replayable.
- Cleaned / silver — deduplicated, validated, joined with reference data.
- Curated / gold — aggregated, business-ready tables consumed by BI.
Each layer is its own S3 prefix and Glue table.