observability aws sre alerting claude-curated
Most CloudWatch alarm setups grow organically: someone adds “CPU > 80%” because a node fell over once, then “memory > 90%”, then “queue depth > 1000”, and within a year the on-call rotation drowns in pages that nobody acts on. The fix is not better thresholds — it is a different model. Alert on user-visible failure (or its leading indicators), not on every internal metric. This is the heart of observability practice.
The anti-pattern: cause-based alerting
Alarming on every resource metric (raw CloudWatch Metrics) leads to alert fatigue. A node at 95% CPU might be a problem or might be exactly what the autoscaler intended. A queue with 10k items might be a backlog or might be peak hour. These metrics describe the system, not the user experience. Think instead in MELT terms — metrics are only one of four signal classes.
Symptoms of the anti-pattern:
- Pagers fire that resolve themselves within 5 minutes.
- On-call acknowledges and ignores most alerts.
- Real incidents get missed because they look like the noise.
- Alarm count grows monotonically; nobody deletes any.
SLO-driven design
The Site Reliability Engineering model inverts the question. Instead of “what could go wrong with my system” ask “what would the user notice.”
SLI — Service Level Indicator
A measurable signal of user experience. Examples:
- Fraction of HTTP requests that return non-5xx within 500ms.
- Fraction of background jobs that complete within their deadline.
- Fraction of writes that are durably acknowledged.
Pick one or two per service. More than that and the model collapses back into resource alerting.
SLO — Service Level Objective
The target for the SLI over a window. Examples:
- 99.9% of requests succeed within 500ms over 30 days.
- 99.5% of jobs finish within 10 minutes over 7 days.
The number itself matters less than picking it deliberately and reviewing it. 99.9% means about 43 minutes of “unsuccess” allowed per 30-day window — that is the error budget.
Error budget
If your SLO is 99.9%, your budget is 0.1%. Spending that budget faster than 1/30th of a day means you are on track to miss the SLO. This is the signal worth paging on.
Burn-rate alerts
The Google SRE Workbook formalises this as multi-window, multi-burn rate alerts:
| Alert | Window | Burn rate | Meaning |
|---|---|---|---|
| Fast burn | 5 min and 1 hour | 14.4x | Will exhaust 30-day budget in 2 days. Page now. |
| Slow burn | 30 min and 6 hour | 6x | Will exhaust budget in 5 days. Page or ticket. |
| Ticket | 6 hour and 3 day | 1x | Trending bad. Investigate during business hours. |
The two-window structure stops a brief blip from paging — both the short and long window must agree before the alarm fires. The multi-rate structure catches both fast outages and slow degradations.
In CloudWatch this is built with metric math expressions that compute the burn rate from a request-success metric, then alarm when the burn rate breaches a threshold across both windows simultaneously.
Composite alarms
CloudWatch composite alarms combine multiple underlying alarms with boolean logic. Useful for:
- AND logic: only page if both burn-rate windows are tripped.
- OR logic: page if any of N regions are unhealthy.
- Suppression: do not page on application alarms during a planned database maintenance.
Composite alarms reduce the firing surface — one actionable page instead of three correlated ones.
Alarm states and INSUFFICIENT_DATA
CloudWatch alarms have three states:
- OK — metric is within threshold.
- ALARM — metric breached threshold.
- INSUFFICIENT_DATA — no data points in the evaluation window.
The trap is INSUFFICIENT_DATA. If your service stops emitting metrics entirely (process crashed, IAM broke, agent died) the alarm goes to INSUFFICIENT_DATA, not ALARM. Whether to treat that as a problem depends on the metric:
- For health-check style metrics, treat INSUFFICIENT_DATA as ALARM. No data probably means dead.
- For sparse business metrics, treat as OK. No data might just mean nobody is using the feature right now.
The setting is TreatMissingData and the choices are breaching, notBreaching, ignore, and missing.
Anomaly detection
CloudWatch Anomaly Detection trains a model on the historical metric and alarms when current values fall outside the predicted band. Useful for metrics with daily or weekly seasonality where a static threshold would either over- or under-fire. The downsides: opaque, slow to adapt to legitimate changes, and harder to reason about during an incident than a clean threshold.
Use it for:
- Cost anomalies.
- Traffic shape (legitimate growth versus runaway client).
- Long-running batch jobs whose duration drifts.
Avoid it for:
- Hard SLOs — these have explicit budgets, not statistical bands.
- Security signals — anomalies are too forgiving here. Pair with CloudTrail Auditing for high-signal audit detections, and GuardDuty / Security Hub for managed alerts.
Routing
Not every alarm should page. A common split:
| Severity | Channel | Response |
|---|---|---|
| SEV1 (page) | SNS to PagerDuty/Opsgenie | On-call engages immediately |
| SEV2 (warn) | SNS to a Slack channel | Look at it next business day |
| Info | EventBridge to a ticket queue | Tracked, not actioned synchronously |
The rule: if it does not require human action within minutes, it does not page.
Runbooks
Every paging alarm should have a runbook URL in its description. The runbook covers:
- What this alarm means in plain English.
- The first three things to check.
- Known causes and known fixes.
- Escalation contact if the runbook does not resolve it.
CloudWatch supports Markdown in alarm descriptions and most paging integrations surface the description in the page body. A runbook link in the alarm is the difference between waking up disoriented and waking up productive. Track repeating incidents in your MTTR and MTBF reports so the runbook itself becomes a living document.