mgarden

Disaster Recovery Concept

6 min read

dr aws operations claude-curated

Disaster recovery is a discipline, not a feature. The product of DR work isn’t a backup or a replica — it’s confidence that, when something goes catastrophically wrong, the team knows what to do, has tested doing it, and can do it within a known time window (see RPO and RTO). Backups and replicas are inputs. Confidence is the output.

The full scope

DR is more than the technical mechanisms:

  • Backups and replication — the data layer. See AWS Backup vs PITR and Cross-Region Replication.
  • Infrastructure-as-code — the environment layer; without it, you can’t rebuild what you’ve lost.
  • Runbooks — the human layer; the exact sequence of steps under pressure.
  • Communication — who decides, who’s notified, who talks to customers.
  • Testing — proves all of the above actually works.

A team that has snapshots but no runbook has a recovery hope, not a recovery plan. The full plan is part of the broader Business Continuity Plan.

DR drills

Drills are the single highest-leverage DR activity. Two flavours:

  • Tabletop — walk through the runbook in a meeting room. “The primary region is unreachable. What’s our first step? Who calls AWS? Where’s the runbook?” Cheap, useful, surfaces gaps in days not hours of lead time.
  • Full restore — actually execute the runbook against a real DR environment. Measure end-to-end time-to-recover. Compare to RTO. Iterate.

Cadence: tabletop quarterly, full-restore at least annually for Tier 1 systems. The first run will overshoot RTO by 2–5x; that’s expected. Each subsequent run improves as the runbook is tightened.

Untested DR plans usually fail

The folklore is well-founded. An untested DR plan fails for a thousand small reasons:

  • The runbook references a Confluence page that’s been moved.
  • The IAM role used in the runbook had its permissions tightened a year ago. (See Least Privilages.)
  • The DR Terraform state references a module that no longer exists in the registry.
  • The on-call engineer doesn’t have credentials for the backup account.
  • The certificate used by the failover endpoint has expired.
  • The DNS TTLs are 24 hours.
  • The recovery depends on a Lambda that depends on a layer that lives only in the failed region.

None of these show up in a diagram. All of them show up in the first drill. An RTO estimate without a drill is a guess; the only evidence that a system meets its RTO is that it has met its RTO recently.

Documentation

The minimum useful DR documentation set:

  • Runbook — step-by-step recovery procedure. Specific. “Run terraform apply from the dr/ directory” is too vague — which Terraform, where’s the state, what variables. Assume the operator is sleep-deprived and unfamiliar.
  • Decision flowchart — when to declare an incident, when to invoke DR, who has authority to do so. Avoids the “is this bad enough?” debate at 03:00.
  • Contact tree — primary on-call, secondary, engineering manager, executive sponsor, AWS TAM, key customers. Phone numbers, not Slack handles, because Slack might be the thing that’s down.
  • Architecture diagram — current, dated, version-controlled. Not a beautifully rendered out-of-date PDF.
  • Recovery dependency map — what depends on what. Identifies the minimum viable recovery (auth + database + main app) versus the full recovery (everything else).

Store the documentation somewhere that doesn’t depend on the systems being recovered. A wiki hosted in the failed region is not a wiki during the incident.

Why DR makes you a better Cloud Engineer

Designing for DR forces architectural rigor in ways nothing else does:

  • Statelessness — services that hold local state can’t be rebuilt from IaC. DR pressure pushes state into managed stores (RDS, DynamoDB, S3) where it’s replicated and backed up centrally.
  • Idempotency — recovery often involves replaying events, retrying operations, partial completion. Non-idempotent code becomes obvious during a drill.
  • Infrastructure-as-code — rebuilding manually is too slow. Terraform/CloudFormation isn’t optional for any system that has an RTO. See also GitOps.
  • Coupling and dependencies — DR exposes hidden coupling. “We can’t restore the API without the auth service, but we can’t deploy auth without the secrets in this other region.” These are bugs in the architecture; recovery testing finds them.
  • Observability — you can’t recover what you can’t see. DR teams are also observability teams: dashboards for replication lag, backup age, failover-readiness; alarms on the things that would silently invalidate the DR plan (CloudWatch / CloudWatch Alarms SLO Driven / SLO).
  • Compliance underpinning — SOC 2, ISO 27001, HIPAA, PCI all require documented DR with regular testing. The work is the same; the audit is the byproduct.

The skills that DR demands — IaC, observability, immutability, runbook discipline, blast-radius thinking — are the same skills that distinguish a senior cloud engineer from a junior one. DR is therefore one of the best growth vectors available, even outside the formal reliability mandate.

Common pitfalls

  • Testing recovery to a dev environment instead of greenfield. The dev environment has been touched a thousand times; it carries assumptions that the real DR target won’t have. Always recover into a clean account or namespace.
  • Depending on resources that wouldn’t exist post-disaster. Recovery script reads from a parameter in the failed region. CI/CD that runs the recovery lives in the failed region. The runbook PDF is in the failed region’s S3 bucket. Audit dependencies; assume the failed region is gone.
  • Depending on third-party services that have their own DR boundaries. Your DR plan is bounded by the worst RTO of any third-party dependency you can’t replace. Identify them.
  • Confusing high availability with disaster recovery. Multi-AZ is HA; multi-region is DR. They protect against different failures and cost different amounts. Don’t claim DR coverage for an HA-only design. See Multi-Failure Defence Patterns.
  • Recovering the technology, not the business. A successful restore that finishes after the company has lost its largest customer is a technical success and a business failure. RTO must be set by impact, not engineering convenience.
  • Skipping post-drill writeups. The drill’s value compounds across iterations only if each iteration produces a documented list of what failed and what changed.
  • Treating DR as a one-time project. Architecture changes, dependencies drift, retention policies expire. A DR plan that was correct two years ago is probably wrong today.

See also

References

Graph View

Backlinks