Multi-Failure Defence Patterns

dr aws security claude-curated

Disaster recovery is not one defence — it’s a stack of defences, each suited to a different failure class. A backup strategy that survives a hardware failure won’t help against ransomware; cross-region replication won’t help against accidental deletes. The discipline is mapping each plausible failure to the cheapest mechanism that defeats it.

Failure modes and their defences

Failure	Defence	Mechanism
Single instance fails	Multi-AZ deployment	RDS Multi-AZ, ASG across AZs
AZ fails	Multi-AZ + LB health checks	Cross-AZ load balancing, instance health checks
Region fails	Cross-region replication + DNS failover	CRR, Aurora Global, Route 53 health checks
Account compromise / ransomware	Backups in separate account + Vault Lock	AWS Backup cross-account copy, IAM separation
Human error / accidental delete	Versioning, soft-delete, MFA Delete, PITR	S3 versioning, DynamoDB PITR, Terraform prevent_destroy
Misconfigured IAM / privilege escalation	SCPs, least privilege, session-based access	AWS Organizations, IAM Identity Center

The pattern: every layer assumes the layer above it failed.

Single-instance and AZ failures

The cheapest tier. AWS makes these almost-free if you use managed services correctly:

• RDS Multi-AZ — synchronous standby in another AZ; automatic failover in 60–120s.
• Auto Scaling Groups spread across at least two AZs.
• Application Load Balancer (ALB) with cross-zone load balancing and target health checks.
• ECS/EKS — schedule across multiple AZs with topology constraints.

Mistakes here are usually omissions: a single-AZ RDS instance, an ASG with min=1 in one AZ, a NAT gateway in only one AZ.

Region failures

Rare but real (control-plane outages, fibre cuts, large-scale events). Defences:

• Stateful tier — cross-region replication (Aurora Global, DynamoDB Global Tables, S3 CRR).
• Stateless tier — pre-baked AMIs/container images replicated to ECR in the secondary region; IaC ready to apply.
• Routing — Route 53 health checks + failover record, or weighted records for active-active.
• Secrets and config — replicated to the DR region (Secrets Manager replication, Parameter Store rebuilt by IaC).

The biggest gotcha is dependencies: certificates issued in one region can’t be used in another (ACM is regional except for CloudFront), KMS keys are regional unless multi-region. Build a regional dependency list as part of the DR runbook.

Account compromise and ransomware

This is the failure mode that catches most teams, and a key concern for Cyber Security. Same-region, same-account backups protect against hardware and human error but not against compromised credentials. Once an attacker has admin in the account, they can delete every snapshot, every replica, every S3 object — including the backups.

The defence is account isolation:

• Backups live in a separate AWS account, in a backup-only role.
• The production account has an IAM role that can write to the backup vault but cannot delete or shorten retention.
• The backup account has its own root credentials, stored offline (cold storage, hardware token).
• Vault Lock in compliance mode prevents even the backup-account admin from shortening retention.
• Multi-region copies inside the backup account.

The blast radius of a compromised production account ends at the production account boundary. The attacker can wreck production but cannot reach the backups.

This pattern has a cost — extra account, extra policies, extra runbook complexity — but it’s the only defence against the ransomware case where the attacker has time to enumerate and destroy.

Human error and accidental delete

The most common failure. Defences are layered:

• S3 versioning — every PUT creates a new version; deletes leave a delete marker, original is recoverable.
• MFA Delete — requires hardware MFA for permanent version deletion. Annoying for operations but ironclad against fat-finger and credential theft.
• DynamoDB PITR — restore to any second in the last 35 days.
• RDS automated backups + PITR — same idea.
• Terraform prevent_destroy lifecycle — refuses to plan a destroy on critical resources. See Terraform State Drift for related risks.
• AWS Config + EventBridge — alarms on destructive API calls (DeleteBucket, DeleteDBInstance) so a mistake is caught in minutes, not weeks.
• Soft-delete patterns in application code — flag rows as deleted instead of hard-deleting.

Defence in depth here matters because each layer fails for a different reason: versioning fails if someone disables it, prevent_destroy fails if someone removes the lifecycle block, soft-delete fails if a migration runs TRUNCATE. Stack them.

Why backups in the same account is insufficient

The argument in three lines:

1. The threat model includes credential compromise.
2. A compromised credential with sufficient IAM can delete anything in the account.
3. Therefore backups must live where the compromised credential cannot reach.

“Same account but different region” defends against region failure, not against credential compromise. They are different failure modes; conflating them is a common mistake in DR design.

AWS Organizations and SCPs

Service Control Policies (SCPs) are the org-level guardrail layer:

• Deny s3:DeleteBucket and s3:PutBucketVersioning (with action Suspended) in the backup account, even for the root user.
• Deny modifications to the SCPs themselves outside of an approved change process.
• Deny disabling CloudTrail, Config, GuardDuty. See CloudTrail Auditing.
• Deny changes to KMS key policies that would lock out the backup-writing role.

SCPs are enforced by AWS regardless of IAM policy in the member account, so they’re a structural defence rather than a configurable one. Useful even for non-DR concerns (region restrictions, instance-type limits, mandatory encryption).

Putting it together

A reasonable layered design for a Tier-1 production system:

1. Multi-AZ for everything stateful.
2. ASG / managed compute across AZs for everything stateless.
3. Cross-region replication for the stateful tier (CRR for S3, Aurora Global, DynamoDB Global Tables).
4. AWS Backup writing to a vault in a separate backup account, with vault lock in compliance mode and 7-year retention.
5. SCPs preventing destructive actions outside the change-management role.
6. PITR enabled on every database.
7. S3 versioning + MFA Delete on critical buckets.
8. CloudTrail + GuardDuty + Config in every account, logs in a separate log-archive account. Use CloudWatch Alarms SLO Driven for monitoring.
9. Quarterly DR drill that actually restores into a clean environment. See RPO and RTO for the targets it must meet.

Skipping any one layer leaves a specific failure mode uncovered. The cost is real but bounded; the cost of being wrong about each layer compounds.

See also

• RPO and RTO
• AWS Backup vs PITR
• Cross-Region Replication
• Disaster Recovery Concept

References

• AWS Well-Architected — Reliability Pillar
• AWS Organizations — Service Control Policies
• S3 — Using MFA Delete
• AWS Backup — Cross-account backup