By INI8 Labs · 2026-07-04 · 13 min read
Kubernetes Disaster Recovery and Backup: A Production-Ready Enterprise Guide
The most dangerous assumption in Kubernetes operations is not "we have no backups." It is "we have backups, but we've never tested restoration under realistic conditions."
Enterprise teams consistently over-invest in documenting their Kubernetes disaster recovery plan and under-invest in testing it. The gap between those two activities is where production incidents are born.
A Velero schedule running for six months, combined with a restoration process never validated in a real cluster environment, is not a tested DR capability. It is the appearance of one.
Many organisations discover this during their first serious incident. The backups exist.
The documentation exists.
What doesn't exist is organisational muscle memory for executing a restoration under time pressure, with engineers most familiar with the original cluster unavailable — because they're managing the incident itself.
What Does Kubernetes Disaster Recovery Include?
What are the components of a Kubernetes DR strategy?
A complete Kubernetes DR strategy covers four recovery domains:
- Cluster infrastructure recovery — can you recreate the cluster?
- Workload state recovery — can you restore running applications?
- Persistent data recovery — can you restore stateful data?
- Configuration and secret recovery — can you restore all application configuration?
Most teams address one or two of these. Complete DR requires all four.
The Four Things That Must Be Backed Up
1. etcd — the Most Critical Component
etcd is the Kubernetes control plane database. It stores all cluster state — every pod definition, service, secret, ConfigMap, and RBAC rule.
If etcd is lost without a backup, the cluster cannot be recovered without complete reconstruction.
etcd backup requirements:
- Frequency: Every 15—30 minutes for production clusters
- Storage: Encrypted, off-cluster, in a different availability zone or region
- Retention: Minimum 7 days of point-in-time snapshots
- Testing: Monthly restoration drill to a separate cluster
# etcd snapshot backup
ETCDCTL_API=3 etcdctl snapshot save /backup/etcd-$(date +%Y%m%d-%H%M).db --endpoints=https://127.0.0.1:2379 --cacert=/etc/kubernetes/pki/etcd/ca.crt --cert=/etc/kubernetes/pki/etcd/server.crt --key=/etc/kubernetes/pki/etcd/server.key
2. Persistent Volumes — Stateful Application Data
Persistent Volumes store application data that must survive pod restarts. For stateful workloads — databases, message queues, file stores — PV backup is the primary DR concern.
Velero with CSI snapshots is the current standard. It integrates with cloud provider snapshot APIs (EBS, Azure Disk, GCP PD) for point-in-time PV snapshots.
3. Kubernetes Manifests — Declarative Configuration
All Kubernetes resources — Deployments, Services, ConfigMaps, Ingress rules, NetworkPolicies, RBAC — should exist in Git.
GitOps architectures (ArgoCD, Flux) make this natural: the Git repository is the authoritative source of cluster state.
If you can recreate a cluster and run argocd app sync, you have recovered your workload configuration.
This is the simplest and most reliable configuration backup approach available.
4. Secrets — the Most Commonly Forgotten Target
Secrets are covered by etcd backup.
But teams using external secret managers (Vault, AWS Secrets Manager) must also have tested recovery procedures for the secret management infrastructure itself — not just the application that depends on it.
Recovery Architecture Patterns
Pattern 1: Single-Region Active-Passive
A primary cluster runs production workloads. A standby cluster in the same region exists with current cluster configuration (via GitOps) but no running workloads. Velero performs scheduled backups to object storage.
- RTO: 1—4 hours
- RPO: Equals backup frequency (15—60 minutes)
- Best for: Development and staging workloads, non-critical production
Pattern 2: Multi-Region Active-Passive
Primary cluster in Region A. Standby cluster in Region B synced via GitOps. Velero backups stored in cross-region object storage. Database replication — not Velero — handles stateful data for near-zero RPO.
- RTO: 15—60 minutes
- RPO: Near-zero for configuration, minutes for data
- Best for: Business-critical applications where 1-hour RTO is acceptable
Pattern 3: Multi-Region Active-Active
Two or more production clusters across regions serve live traffic simultaneously. Global load balancing routes to healthy regions. Data layer uses geo-distributed databases (CockroachDB, Cassandra, Spanner).
- RTO: Seconds to minutes (automated failover)
- RPO: Near-zero with synchronous replication
- Best for: Applications with 99.99%+ SLOs and direct revenue cost of downtime
Velero: The Standard Tool and Its Limits
Velero is the CNCF-graduated backup tool for Kubernetes. It handles scheduled backups, selective restoration, and cross-cluster migration reliably.
What Velero does well:
- Kubernetes resource backup (objects to object storage)
- PV backup via CSI snapshot integration
- Scheduled backups with configurable retention
- Selective restore by namespace or resource type
What Velero does NOT do — and where teams learn this the hard way:
A filesystem-level snapshot of a PostgreSQL PV taken during active write operations may be inconsistent.
Whether it is recoverable depends on the database's crash recovery capability — and depending on PostgreSQL's crash recovery as your backup strategy is a choice that should be made explicitly, not by default.
Run database-native backups alongside Velero for every stateful workload. Treat them as complementary, not competing.
Industry Applications
Healthcare
Clinical applications require documented RTO and RPO commitments as part of HIPAA Business Continuity Planning.
DR testing frequency is also a compliance requirement — a backup strategy that has never been tested does not meet the intent of HIPAA contingency planning.
Financial Services
PCI-DSS requires that backup data is protected from unauthorised access and that restoration capability is tested.
For trading and payment systems, the RPO requirement is often measured in seconds — requiring continuous database replication, not Velero snapshots.
Retail
Peak-period DR testing is essential.
A DR plan tested in March may not account for the infrastructure changes accumulated before a November peak season.
Run a full DR drill in the 60 days preceding peak trading — not just post-incident reviews.
The DR Testing Protocol
Platform teams often find that DR testing is the most consistently deferred item on the engineering calendar. The reasons are legitimate. The consequences of deferral are not.
The most valuable outcome of DR testing is not confirming that restoration works. It is discovering the specific ways it doesn't:
- The undocumented dependency that isn't in Velero scope
- The secret managed externally that isn't restored
- The network configuration requiring manual recreation
- The application needing a specific startup sequence not encoded in manifests
These discoveries, made during a planned test, are recoverable. Made during a real incident, they extend RTO by hours.
Recommended test cadence:
- Monthly: Restore a non-production namespace from backup to a test cluster. Verify all pods start, services resolve, and application functions.
- Quarterly: Full cluster restoration drill. Provision a new cluster, restore from backup, verify application functionality end-to-end.
- Annually: Region-level failover test. Simulate a complete regional outage by routing traffic to the DR region.
One practical note: the engineers who execute the quarterly drill should rotate.
A DR process that only works when the engineer who designed it is executing it is not an organisational capability — it is a single point of failure wearing an infrastructure hat.
Actionable Takeaways
- Implement GitOps as the foundation of K8s DR — if cluster config lives in Git, cluster recovery is a sync operation
- Configure Velero with scheduled backups to cross-region object storage for all production namespaces
- Back up etcd every 15—30 minutes; store backups encrypted and off-cluster
- Never use Velero as your sole database backup strategy — run database-native backup tools in parallel
- Define RTO and RPO targets before selecting a DR architecture pattern
- Schedule quarterly DR testing drills with rotating engineers — untested backups are not DR
FAQ
What is Kubernetes disaster recovery? The combination of backup configurations, restoration procedures, and architecture patterns that enable recovery of Kubernetes cluster state, workloads, persistent data, and application configuration after infrastructure failure or accidental deletion.
What is Velero in Kubernetes? A CNCF-graduated backup and restore tool for Kubernetes that backs up resources to object storage and persistent volumes via CSI snapshots. Supports scheduled backups, selective restoration, and cross-cluster migration.
How often should Kubernetes etcd be backed up? Every 15—30 minutes for production clusters with strict RPO requirements. Backups must be stored encrypted, off-cluster, and in a different availability zone or region from the cluster itself.
What is the difference between RTO and RPO? RTO (Recovery Time Objective) is how long recovery takes after a failure. RPO (Recovery Point Objective) is how much data can be lost — measured by the time between the last good backup and the failure.
Can Velero back up databases? Velero can take filesystem-level PV snapshots, which may be inconsistent for active databases.
Database-native tools (pg_dump, Percona XtraBackup) should run alongside Velero for transactionally-consistent database backups.
What is GitOps-based Kubernetes DR? Using a Git repository as the authoritative source of all Kubernetes configuration.
After a cluster failure, a new cluster is provisioned and ArgoCD or Flux syncs all workload configuration from Git — reducing cluster recovery to infrastructure provisioning plus GitOps sync.
INI8 Labs provides Kubernetes platform engineering including DR architecture design, Velero implementation, and multi-cluster management.