Disaster Recovery

1. Concept Overview & Systemic Problem

In infrastructure engineering, hardware failures of SSD drives, data center outages, ransomware attacks, and human errors (DROP DATABASE production, accidental rm -rf) are not a question of "if," but rather "when."

Most teams operate under an illusion of security: they add a line to crontab for daily dumps and consider the task solved. When a real disaster strikes, it becomes clear that:

• The last backup failed three months ago due to disk overflow.
• The dump contains corrupted binary data.
• Restoring a 100 GB database takes 36 hours, paralyzing the business (unacceptable RTO).
• An attacker, having compromised the server, deleted backups along with the working database because the S3 API key had full deletion rights (s3:DeleteObject).

Disaster Recovery is a comprehensive business continuity strategy. It encompasses not just file preservation, but mathematically calculated RTO/RPO targets, immutable storage, and automated regular drills for deploying infrastructure from scratch.

2. Architectural Taxonomy & Mental Model

The architectural matrix of disaster recovery is based on the trade-off between speed and frequency of state capture:

┌─────────────────────────────────────────────────────────────┐
│                 DISASTER RECOVERY TAXONOMY                  │
├─────────────────────────────────────────────────────────────┤
│ 1. Point-In-Time Recovery (PITR) ➔ RPO ~ seconds            │
│    • Streaming WAL (Write-Ahead Log) in real-time           │
│    • Tools: Litestream (for SQLite), pgBackRest (PG)        │
├─────────────────────────────────────────────────────────────┤
│ 2. Deduplicated & Encrypted Snapshots ➔ RPO ~ hours        │
│    • Atomic data snapshots via Restic, BorgBackup, Kopia    │
│    • Client-side encryption AES-256 / ChaCha20               │
├─────────────────────────────────────────────────────────────┤
│ 3. Offsite Immutable Storage Tier (Anti-Ransomware)         │
│    • S3 Object Lock (WORM - Write Once, Read Many)          │
│    • Geographically isolated storage (Cloudflare R2, AWS S3) │
├─────────────────────────────────────────────────────────────┤
│ 4. Cold Standby & Automated Recovery Drill (RTO validation)  │
└─────────────────────────────────────────────────────────────┘

1. Point-In-Time Recovery (PITR):
   • Instead of a single heavy dump once a day, the database continuously streams its binary change log (WAL) to the cloud. This allows "rewinding" the database state to the second before a failure or erroneous developer request (RPO < 10 seconds).
2. Deduplicated Incremental Snapshots:
   • Utilities like Restic break files into cryptographic blocks. If only 1% of a 50 GB database has changed, only the new blocks are uploaded, saving up to 95% of disk space and network traffic.
3. Immutable Storage (Immutable WORM Storage):
   • The S3 Object Lock security policy ensures that no one (not even an administrator with root access) can delete or overwrite an archived file for a specified period (e.g., 30 days).
4. Recovery Drills:
   • A regular automated process in CI that downloads the latest archive, deploys it in an isolated container, runs verification SQL queries, and confirms data integrity.

3. Technical Pipeline & Internal Mechanics

The lifecycle of reliable backup with deduplication and encryption:

1. Consistent State Freezing (Snapshot Lock): The database is put into a preparation mode for copying, or a transactional snapshot is created via COW (Copy-on-Write) mechanisms of ZFS/Btrfs file systems.
2. Client-Side Encryption at Host: Before being sent over the network, data is encrypted using a secure algorithm (AES-256-GCM) with a passphrase known only to the engineer. The S3 hosting provider receives only encrypted binary noise.
3. Parallel Upload to Isolated Cloud Storage: Blocks are transmitted via HTTPS to an independent geographic region (e.g., a server in Germany backs up to Cloudflare R2 storage in Sweden).
4. Application of Retention Policy (GFS Retention Policy): The algorithm retains: the last 24 hourly backups, 7 daily, 4 weekly, and 12 monthly (Grandfather-Father-Son), automatically deleting older interim snapshots.
5. Automated Integrity Verification: A separate background task runs weekly executing the command restic check --read-data-subset=5%, detecting potential bit degradation (Bit Rot) in the storage.

4. Production Engineering Scenarios

01. Continuous SQLite Replication Using Litestream

For applications based on SQLite/Turso:

• Litestream operates as a background system process, intercepting changes in the WAL file.
• Every 10 seconds, new frames are encrypted and uploaded to a Cloudflare R2 bucket.
• In the event of a complete server failure, a new machine can be brought up with a single command litestream restore -o /var/data/app.db, restoring the system state with virtually no data loss (RPO < 10 s, RTO < 60 s).

02. Protection Against Malicious Actors via S3 Object Lock

Preventing infrastructure destruction by hackers:

• Even if an attacker gains full root access to the server and finds the configuration with the AWS_ACCESS_KEY_ID keys, attempting to execute aws s3 rm --recursive is blocked by the AWS Compliance Lock policy at the Amazon data center level.
• The company is guaranteed access to immutable backups.

03. Cold Standby Recovery Plan via Terraform (IaC Standby)

An OVH-level disaster (fire in the data center in Strasbourg):

• The primary server becomes physically inaccessible.
• The engineer triggers a CI pipeline: Terraform rents a new instance in Hetzner in 3 minutes, Ansible applies the base configuration, a script uploads the latest Restic backup, and Cloudflare DNS records switch to the new IP in 60 seconds.

5. Pitfalls, Common Mistakes & Security

• Storing the decryption password alongside the backup: If the password for the Restic repository is stored in an open file /root/.backup_pass, server compromise simultaneously grants attackers access to read the entire client database in the backups.
• Attempting to copy live database files without locks (Dirty Reads): Simply copying the PostgreSQL data directory (cp -r /var/lib/postgresql) during active queries creates an inconsistent binary state (Torn Pages) that cannot be started after recovery.
• Insufficient memory for decompressing the dump: If the new server has a smaller disk than the unpacked database, the recovery procedure will fail midway with a "no space left on device" error.
• Ignoring configuration files and environment variables: Backing up only the database without preserving Nginx configurations, SSL certificates, and .env files increases RTO from 15 minutes to several days of manual configuration recall.