mgarden

Terraform State Drift

5 min read

cloud IaC terraform state claude-curated

Drift is the gap between what Terraform state says exists and what’s actually deployed. Terraform’s whole model assumes state is the source of truth; when reality diverges, plans become misleading and applies risk surprise destruction.

What drift is

State is a JSON snapshot of every managed resource: its identifier, attributes, and dependencies. On plan, Terraform refreshes that snapshot against the cloud and computes the diff between (refreshed state) and (desired config). Drift is when the refresh step turns up changes — the cloud now reports a different value than state remembered.

Two flavours:

  • Attribute drift — the resource still exists but a property changed (a security group rule was added by hand, a tag was edited).
  • Existence drift — the resource was deleted out of band, or a new ID points to something that’s no longer the original. State references a thing that’s gone.

Common causes

  • Manual console changes. Someone fixed a production incident through the cloud UI and forgot (or didn’t know how) to fold the change back into config.
  • Out-of-band scripts. A backup script that rotates credentials, an Lambda that updates DNS records, a cron that scales something nightly.
  • Provider bugs. Rare but real — a provider version computes an attribute differently between runs and shows a phantom diff.
  • Deleted resources. A teammate ran aws ... delete on a resource Terraform owned, leaving state pointing at nothing.
  • External lifecycle. Resources whose ID changes over time — auto-renewed certs (see ACM), rotated tokens — can look like drift even when nothing is wrong.
  • Cross-tool ownership. Two Terraform configs both think they own the same resource; one’s apply is the other’s drift.

Detection

The cheapest detection is terraform plan itself. With no config changes, the plan should be empty; any output is drift.

terraform plan

For organisation-wide visibility:

  • TFC drift detection runs scheduled plans across workspaces and surfaces drift in the UI (paid feature on Plus tier). See Terraform Cloud Workspaces.
  • Atlantis and similar self-hosted runners can be configured to run periodic plans.
  • Custom CI — a nightly job that runs plan per workspace and posts non-empty diffs to chat works for smaller estates. CloudWatch alarms or EventBridge can route alerts.

The detection cadence matters. Drift discovered the morning after it happened is recoverable; drift discovered six months later is forensic archaeology.

Recovery patterns

Re-import a resource

If the resource still exists but state has the wrong identifier (resource was recreated externally with a new ID), drop it from state and re-import:

terraform state rm aws_db_instance.main
terraform import aws_db_instance.main db-prod-2026-04

Or, declaratively, use an Terraform Import Block in a PR.

terraform state rm for orphans

The resource was deleted out of band; state still references it; apply keeps trying to reconcile a ghost. Remove the state entry:

terraform state rm aws_eip.legacy

Then either delete the resource block from config (acknowledging the deletion) or restore the underlying resource through Terraform.

terraform apply to reconcile

If config still represents the desired state and the drift is something you want reverted (e.g. someone added a permissive security group rule by hand), an apply will pull reality back to match config.

This is the right answer when “the manual change was a mistake”. It’s the wrong answer when the manual change was a fix that hadn’t yet been written into config — applying then will silently undo the fix.

Always read the plan before applying drift recovery. The diff tells you which side of the drift is correct.

terraform refresh (deprecated)

refresh updated state from real infra without computing a diff. It’s deprecated as a standalone command; the same behaviour is built into plan’s refresh step. There’s a -refresh-only flag if you specifically want the refresh without the diff:

terraform plan -refresh-only
terraform apply -refresh-only

-refresh-only accepts the live state as the new truth. Use this when you’ve decided the manual change is correct and you want state to remember it without changing infra.

Stale state IDs

A particularly annoying case: the resource was recreated externally (deleted then re-created with the same name but different cloud ID). State remembers the old ID, refresh fails to find it, plan shows a destroy-and-create diff that would actually destroy the working resource if applied.

Recovery:

  1. terraform state rm the stale entry.
  2. terraform import (or import block) using the new ID.
  3. terraform plan should now show only the in-place differences between config and the recreated resource.

Drift detection automation

For a small estate, a nightly CI job per environment that runs plan -detailed-exitcode and pipes a non-zero exit to a notification is enough. Detailed exit codes:

  • 0 — no diff
  • 1 — error
  • 2 — diff present (drift or pending change)

For large estates, paying for TFC’s drift detection (or running Atlantis equivalent) is usually cheaper than maintaining the ad-hoc CI version. See CI CD for Terraform for the wider tool comparison.

Prevention

Drift recovery is straightforward; preventing drift is cultural.

  • Make config the only writeable interface. Console access is read-only or break-glass. Pair with Least Privilages IAM roles.
  • Require all changes through PR. CLI bypass is documented and reviewed (see Terraform CLI with Cloud Hybrid).
  • Use lifecycle { ignore_changes = [...] } for attributes that genuinely belong to another tool — this is “officially sanctioned drift” rather than untracked drift.
  • Run drift detection often enough that whoever caused the drift still remembers what they did. Terraform Cloud Pitfalls notes how TFC’s own drift feature can surprise.

See also

References

Graph View

Backlinks