data security privacy claude-curated
When production data needs to live in non-production environments — staging, CI, demo, dev sandboxes — PII has to come out. The naive “just copy prod” approach fails GDPR and most security audits.
Anonymisation vs pseudonymisation
GDPR distinguishes:
- Anonymisation — irreversible. The data is no longer personal data after the process; subject can’t be re-identified.
- Pseudonymisation — reversible (with the right key). Still personal data under GDPR, but lower risk.
In practice, most “anonymisation” pipelines actually pseudonymise. True anonymisation is hard — datasets that look anonymous can be re-identified by joining with public data (the Netflix Prize attack, AOL search log de-anonymisation).
Techniques
1. Masking
Replace values with a fixed token. john@example.com → email@example.com.
- Simple, predictable.
- Destroys distribution — every row looks identical, breaks queries that group by email domain.
2. Hashing
Replace with a hash. john@example.com → sha256(john@example.com).
- Preserves uniqueness and joins (same input → same output).
- Vulnerable to dictionary attacks for low-entropy inputs (emails, phone numbers).
- Add a salt; rotate it for each non-prod refresh.
3. Format-preserving encryption (FPE)
Encrypt while keeping the same shape. A credit card number stays a 16-digit number. Useful when downstream systems validate format.
4. Tokenisation
Replace with a token from a lookup table. The mapping lives in a separate, secured store (or is destroyed for true anonymisation).
5. Generalisation
Reduce precision. Date of birth → year of birth. Postcode → first 3 chars. Balances utility and risk.
6. Synthetic data
Generate fake data that statistically resembles real. Tools: Faker, Mostly AI, Gretel. Best for demo and load testing; less faithful for bug reproduction.
Preserving referential integrity
If users.id = 42 is changed to users.id = X, every foreign-key reference (orders.user_id = 42) must also become X. Otherwise the dataset is broken.
Strategies:
- Deterministic hash — same salt across tables means
hash(42)is identical everywhere. - Mapping table — generate
(real_id, fake_id)once, apply across all tables in one pass. - Cascade through schema — walk FKs from root tables outward.
What to anonymise
PII categories that almost always need treatment:
- Names, email addresses, phone numbers
- Addresses, postcodes, GPS coordinates
- Government IDs (SSN, NI, passport)
- Date of birth (often → year only)
- IP addresses
- Free-text fields that may contain PII (review comments, notes) — hardest case; consider redaction or removal
Plus anything domain-specific: medical records, financial accounts, biometrics.
Pipeline shape
[ Prod snapshot ] → [ Restore to scratch DB ] → [ Anonymisation job ] → [ Export to non-prod ]
Run the job inside a locked-down account/network so the unmodified prod copy never reaches dev hands. Common implementation: AWS Glue or a dedicated EC2 in a VPC with no egress, writing the sanitised result to a non-prod-accessible bucket.
Pitfalls
- Test data that’s too fake — if every user has the email
test@test.com, you’ll never repro a bug caused by Unicode in addresses. - Forgetting indirect PII — a unique combination of postcode + DOB + gender re-identifies most people. Anonymising names alone isn’t enough.
- Backups containing real data — non-prod backups must be sanitised too, or you’ve recreated the problem.
- Stale anonymisation rules — a new column gets added in prod, the anonymisation job doesn’t know about it, PII leaks. Audit periodically.