observability aws opensearch claude-curated
OpenSearch is the Apache 2.0 fork of Elasticsearch maintained by AWS and the wider community since the 2021 license change. Version upgrades — moving a domain from one minor or major version to the next — are routine but carry real risk: shard format changes, mapping deprecations, client incompatibility. The upgrade path differs sharply between AWS managed OpenSearch Service and self-managed clusters on EC2.
AWS managed OpenSearch Service
Managed OpenSearch makes upgrades a one-click operation. Behind the scenes AWS performs a blue/green deployment:
- Provisions a new cluster running the target version alongside the existing one.
- Streams a snapshot of every index across to the new cluster.
- Replays in-flight writes once the bulk migration catches up.
- Atomically swaps the domain endpoint to point at the new cluster.
- Tears down the old cluster.
The endpoint URL stays the same, so clients usually do not need to be reconfigured. Throughput on indexing typically dips during the migration window because the cluster is doing double the work, but query traffic continues to be served.
Pre-upgrade checklist
- Take a manual snapshot to S3 even though AWS keeps automated daily snapshots. Manual snapshots are required for cross-region restore and for retention beyond the managed window (think RPO and RTO).
- Read the release notes for breaking changes: removed REST endpoints, deprecated mapping types, default setting changes.
- Verify client compatibility. A common surprise is that an older client library throws on a newer cluster because of header or response-shape changes. The official OpenSearch clients are version-tolerant within reason; third-party libraries less so.
- Check plugin compatibility. Custom analysers, security plugins, and ingest processors must support the target version.
- Run a test upgrade on a non-production domain first, ideally one fed with a representative slice of production traffic.
Snapshot strategies
Snapshots are the foundation of any safe upgrade.
| Snapshot type | Owner | Use |
|---|---|---|
| Automated hourly/daily | AWS managed | Quick rollback inside the retention window |
| Manual | You | Long retention, cross-region restore, pre-upgrade backstop |
To register a custom S3 repository (see also Data Lake on S3):
PUT _snapshot/my-repo
{
"type": "s3",
"settings": {
"bucket": "opensearch-snapshots",
"region": "eu-west-1",
"role_arn": "arn:aws:iam::...:role/OpenSearchSnapshotRole"
}
}Then trigger a manual snapshot:
PUT _snapshot/my-repo/pre-upgrade-2026-04-28Major version jumps
OpenSearch will not let you skip too many majors in one go. A cluster on Elasticsearch 6.x cannot upgrade directly to OpenSearch 2.x — the typical path is 6.x to 7.x to OpenSearch 1.x to OpenSearch 2.x, with index reindexing required at each major boundary because Lucene segment formats change. This is by design: each version supports reading indices from the previous major only. Terraform modules describing the domain should pin the engine version so unintended upgrades cannot land via drift.
Plan a multi-week programme for big jumps: each step is its own blue/green cycle and each step needs verification.
Downtime expectations
Blue/green is “near-zero” downtime, not zero:
- Endpoint flip is atomic, so reads do not break.
- Indexing throughput drops during migration — sometimes by 30-50% on busy clusters.
- Long-running queries that span the cutover may fail and need retry.
- Bulk indexing pipelines should be paused or back-pressured during the window. Pair the runbook with the CloudWatch Alarms SLO Driven dashboards so the on-call sees the dip without paging.
Self-managed OpenSearch on EC2
Running OpenSearch yourself on EC2 (or Kubernetes) means doing the upgrade by hand and managing your own telemetry. The standard pattern is a rolling restart upgrade:
- Disable shard allocation:
PUT _cluster/settings { "persistent": { "cluster.routing.allocation.enable": "primaries" } }. - Stop the OpenSearch process on one node.
- Upgrade the binary in place.
- Start the node, wait for it to rejoin the cluster.
- Re-enable allocation, wait for green status.
- Repeat on the next node until the whole cluster is on the new version.
This works but is slow, error-prone, and does not skip versions. You walk the cluster through every minor release in turn. The upside is full control — you decide when to upgrade plugins, when to reindex, what node sizes to use during transit. Surrounding tooling such as Logstash, Kibana, OpenSearch Dashboards, and Grafana should be version-checked as part of the same change.
Rollback
Managed OpenSearch does not offer a true “rollback” button. If something breaks after upgrade you restore the pre-upgrade snapshot to a fresh domain on the old version and re-point clients. This is why the manual pre-upgrade snapshot matters — automated snapshots may already have rolled past the boundary. Treat upgrade rollback as a Disaster Recovery Concept exercise: test it, rehearse it, and align the snapshot retention with your RTO expectation.