Unit 7.1: Resilience and Business Continuity
Introduction
Module 3 built the steering layers: balancer health checks, IP failover groups and a Cloud DNS record that your monitoring rewrites (Units 3.3 to 3.7). Module 5 built the copies: point-in-time recovery, off-site backup locations, snapshots and a locked archive (Units 5.3 and 5.7). Neither half is a recovery design on its own. A record that points at Berlin before Berlin has the data is an outage with a new address.
This unit joins the two halves. It starts from the targets the business sets, chooses a strategy for each workload, places the redundant parts so that one failure cannot hit both, and ends by building and drilling the cross-site failover that FinCorp's risk office signs off.
1. Objectives and Strategies
1.1 RTO and RPO Belong to the Business
The recovery time objective (RTO) is the longest outage a service may have. The recovery point objective (RPO) is the most recent data the business may lose. Both are business decisions: the architect prices them, the business owner chooses them, and the design then has to prove them in a drill. A target that nobody has tested is a hope, not an objective.
On this platform every mechanism contributes a known share to those numbers, and the design adds them up:
- RPO comes from the data plane. Strictly Synchronous DBaaS for PostgreSQL loses no committed transaction on an in-cluster failover, while a site loss loses the WAL not yet uploaded to the backup location, up to about five minutes (Unit 5.3).
- RTO is detection, plus the time to make the data available at the other site, plus the time clients take to follow. Clients follow a Cloud DNS record within one TTL, and the TTL floor is 60 seconds (Unit 3.7).
1.2 Three Strategies on Platform Primitives
Each strategy buys a lower RTO with more standing cost. FinCorp chooses per workload, not per company.
| Strategy | What runs at the recovery site before the incident | Recovery mechanism | RTO shape | Standing cost |
|---|---|---|---|---|
| Backup and restore | Nothing; a network plan and the runbook | Backup Service restores, a new DBaaS cluster from an off-site backup, servers rebuilt from images uploaded to the recovery location | Hours: everything is provisioned during the incident | Backups and storage only |
| Pilot light | VDC, LANs, reserved IPs, ALB, DNS records and images; servers stopped in the DCD | Start the servers, create the database from backup, rewrite the DNS record | Detection plus database restore plus one TTL | Network and storage; stopped servers have CPU and RAM billing suspended |
| Active-active (stateless tiers) | Full stateless tiers in both sites, each serving its own share of traffic | The surviving site takes the failed site's names; the data tier fails over as in pilot light | Minutes for the stateless tiers | Two full compute tiers |
Three platform facts shape the table. A server stopped through the DCD has its resources deallocated and billing suspended, while a shutdown inside the operating system keeps it billed; stopping also releases a dynamically assigned IP address, so every address the recovery site depends on must be reserved. Uploaded images and snapshots are bound to their region, so the Berlin site needs its own copies. And DBaaS replication runs between the instances of one cluster, inside one location (Unit 5.3), so no strategy here gives a second writable database in the other region. "Active-active" on this platform therefore means active-active stateless tiers over a single-writer data tier.
FinCorp's assignment follows:
| Workload | RTO / RPO (set by the business) | Strategy |
|---|---|---|
| Customer portal and partner API | 1 hour / 5 minutes | Pilot light in fc-prod-txl-dr |
Case management on fc-mongo-cases |
8 hours / 24 hours | Backup and restore from dumps |
| Internal reporting | 2 days / 24 hours | Backup and restore |
2. Placement Inside a Site
2.1 Zones Are Separate Physical Resources
Inside a data center, availability zones separate failure domains. A server or a storage device assigned to Zone 1 resides on a different physical resource than one assigned to Zone 2. Servers offer Zone 1, Zone 2 and Auto (the default, in which the system assigns a zone on provisioning). Block Storage adds Zone 3, which has no compute counterpart (Unit 5.1).
The two layers behave differently after provisioning. A server's zone can be changed later in the Inspector pane; a volume's zone is fixed at first provisioning, and moving it takes a snapshot and a new volume. A redundant pair whose servers sit in Zone 1 and Zone 2 but whose volumes share a zone has not removed its common failure point.
2.2 The Auto-Zone Trap
Auto leaves the assignment to the system at provisioning time. For a single server that is harmless. For a redundant pair it means the separation is not a recorded decision, and nobody can show an auditor that the two members do not share a physical resource. The rule is simple: every member of a redundant pair gets an explicit zone, and its volumes get the same zone as their server.
The rule applies wherever you choose the zone: portal and data servers, IP failover members (Unit 3.5), and Kubernetes node pools, where the zone is part of the immutable node template, so zone redundancy means one pool per zone (Unit 6.2). The managed edge services carry their own redundancy; the NAT Gateway, for example, deploys multiple gateways across availability zones in a region (Unit 3.5). Zones protect against the loss of a resource inside the site. They do not protect against the loss of the site, which is the job of the next section.
3. Two Planes, One Order
3.1 Steering and Continuity Are Different Planes
A failover moves two things that the platform handles with different mechanisms:
| Plane | Moves | Mechanisms | Decides |
|---|---|---|---|
| Traffic steering | Where new requests go | ALB and NLB health checks within a site; IP failover groups for your own appliances; your monitor rewriting a Cloud DNS record across sites | RTO, through detection time and TTL |
| Data continuity | Where the newest copy of the data is | In-cluster replication; PITR backups in an off-site backup location; Backup Service; dumps and replicated buckets in Object Storage | RPO, and the restore share of the RTO |
The planes fail independently and recover at different speeds. A record change takes seconds plus one TTL; creating and filling a new PostgreSQL cluster from a backup takes as long as the data size demands. For object data, remember the Object Storage boundary: cross-regional replication works for user-owned buckets, not for contract-owned ones (Unit 5.2).
3.2 Never Let Steering Run Ahead of Data
The failover sequence is fixed: declare the incident, make the data available at the recovery site, verify it, then steer traffic. Your monitor from Unit 3.7 can detect a dead site on its own, but in the pilot-light design it should raise the declaration, not rewrite the record unattended, because the database in Berlin does not exist yet. Only the stateless active-active names, whose backends need no restore, may fail over automatically.
Failback follows the same order in reverse. After the incident, Frankfurt receives the data from Berlin, for example as a new cluster created from the Berlin cluster's backup, and only then does the record move back. A failback that skips the data step loses every write made in Berlin.
4. Hybrid Estates: Private Cloud Redundancy
FinCorp's core-banking systems run on a Private Cloud cluster (Unit 4.4), which has its own redundancy model. Inside the cluster, vSAN protects data across hosts: a RAID 1 policy needs at least three hosts, so that two complete copies exist and one stays intact if a host fails. Changing storage efficiency or encryption can require an extra host to keep full redundancy; otherwise vSAN must temporarily reduce the protection level, which the Allow reduced redundancy option permits. Schedule such changes like a risk window, not like routine maintenance.
The cluster stands in one location, so host redundancy plays the role that zones play in a VDC: it protects against a failure inside the site. Protection against losing the site comes from VMware Cloud Director Availability (VCDA), which replicates virtual machines asynchronously, performs failover and reverses failovers between paired sites. The Backup Service can also be used together with Private Cloud for backups of the estate.
The two models reconcile as one table in FinCorp's continuity plan:
| Failure | VDC estate | Private Cloud estate |
|---|---|---|
| One server or host | Explicit zones, balancer health checks | vSAN copies on other hosts, at least three hosts |
| One site | Pilot light, DNS rewrite after data restore | VCDA failover of replicated VMs, then the same DNS step for any public name |
| Data corruption | PITR to a target time, dumps | Backup Service restore |
The steering plane is shared. Whichever estate fails over, a public name moves through the same monitored Cloud DNS record, so the TTL rule and the "data first" order apply to both.
DCD Implementation Walkthrough
You will verify the two-zone pair in Frankfurt, prepare the Berlin pilot light, and run a full failover drill for the portal: database restore in Berlin, then the DNS rewrite, then measurement against the RTO and RPO. This realises Sections 2.2 and 3.2: explicit zones inside the site, and steering after data.
Prerequisites: fc-prod-fra-core with fc-app-01 and fc-app-02, fc-data-01 and fc-data-02 (Units 3.1, 4.1, 4.2); the ALB fc-alb-edge with the health-checked target group fc-tg-portal (Unit 3.3); fc-pg-portal with a backup location outside Frankfurt (Unit 5.3); the zone fincorp.example with portal, portal-fra and portal-txl, and the external monitor (Unit 3.7); the VDC fc-prod-txl-dr with its LANs, a reserved public IP and an ALB; the Access and manage DBaaS and Access and manage DNS privileges.
Build goal: Wire monitored DNS failover across a two-zone pair and validate it, with the cross-site switch made by your monitoring after the data is restored.
Steps (in the Data Center Designer):
- Open
fc-prod-fra-core. Selectfc-app-01and check Availability Zone in the Inspector pane: it must read Zone 1, not Auto. Checkfc-app-02for Zone 2, and the data servers the same way. Correct a server set to Auto now; its zone can be changed after provisioning. - Select each volume of those servers and confirm that its Availability Zone matches its server. A volume in the wrong zone cannot be changed: create a snapshot and provision a new volume from it in the right zone.
- Open
fc-alb-edgeand the target groupfc-tg-portal, and confirm that both targets have Health Check Enabled. These checks are the in-site failover; the drill must not depend on them failing. - In
fc-prod-txl-dr, build the portal tier from images uploaded to the Berlin location: two servers, Zone 1 and Zone 2, behind the Berlin ALB. When they pass their health checks, stop them through Power > Stop in the Inspector pane and provision the change. They now wait with CPU and RAM billing suspended. - Start the drill in a maintenance window. Record the time, then go to Menu > Databases > PostgreSQL and select Create cluster. Enter a Cluster Name such as
fc-pg-portal-txland select the Berlin Location; it cannot be changed after creation. - In Create from backup, select
fc-pg-portalas Backup cluster and pick the Recover target time. Complete the configuration as in Unit 5.3, with the Berlin data LAN in Network configuration, and select Save. Wait until the cluster is AVAILABLE, and record the time. - Start the two Berlin portal servers, point their database configuration at the new cluster's DNS Name, and confirm that
portal-txlanswers through the Berlin ALB. - Trigger the failover action of your monitor: the
PUTthat rewritesportalto the Berlin ALB address (Unit 3.7). Measure how long test clients take to reach Berlin. - Record the drill: detection or declaration time, database restore time, DNS switchover time, and the gap between the recovery target time and the last write in Frankfurt. Compare the sum with the 1-hour RTO and the gap with the 5-minute RPO.
- Fail back in the same order: data to Frankfurt first, then the record. Stop the Berlin servers through the DCD and delete the drill cluster once the drill report is signed off.
Common mistakes:
- Leaving a redundant member on Auto. The separation is then an assumption, not a decision.
- Checking server zones and forgetting the volumes. A volume's zone is fixed at provisioning.
- Letting the monitor rewrite the record automatically for a tier whose database must first be restored.
- Shutting down the Berlin servers inside the operating system. They stay billed; stop them through the DCD.
- Relying on a dynamically assigned address in the recovery site. Stopping a server releases it; reserve every address a record or partner depends on.
- Keeping
fc-pg-portal's backup location in Frankfurt. A site loss then takes the backups with it. - Measuring the drill without the database restore. The restore is usually the largest share of the RTO.
Summary
A recovery design starts from business-owned RTO and RPO and adds up the share each mechanism contributes. Inside a site, explicit zones for every redundant member and its volumes remove common failure points; across sites, FinCorp's portal uses a pilot light in Berlin, with data restored from the off-site backup before the monitored Cloud DNS record moves. Private Cloud follows the same logic with vSAN host redundancy inside the cluster and VCDA across sites, and shares the same steering layer.
Key Points:
- RTO is detection plus data restore plus one TTL; RPO comes from the data plane, up to about five minutes of WAL for a PostgreSQL site loss.
- Choose backup-restore, pilot light or active-active per workload; on this platform the data tier stays single-writer.
- Give every redundant server and volume an explicit zone; Auto is not a recorded separation.
- Restore the data first, then steer traffic; fail back in the same order.
- Stop standby servers through the DCD and reserve every address the recovery site depends on.
Important Terminology:
- Pilot light: a recovery site whose network, addresses, balancer, records and images exist in advance, with compute stopped and data restored on demand.
- Traffic-steering plane: the mechanisms that decide where new requests go: health checks, IP failover and DNS records.
- Data-continuity plane: the mechanisms that decide where the newest usable copy of the data is: replication, backups, dumps and replicated buckets.