Classification: Business Continuity & Disaster Recovery (BC/DR)
Status: Implemented & Verified

Executive Summary#

This report details the implementation, cryptographic architecture, and verification lifecycle of Bastion’s automated disaster recovery pipeline. Moving beyond passive snapshots, the architecture enforces client-side AES-256 encryption, offsite repository replication, and regular cold-restore validation drills to guarantee deterministic recovery in the event of catastrophic storage or hardware loss.

Threat Model & Failure Scenarios#

A backup strategy that has not been restored is merely a hypothesis. The backup architecture was designed to mitigate four specific failure modes:

  1. Catastrophic Hardware/Host Failure: Total loss of the primary host or local drive media requiring a bare-metal reconstruction from raw configuration files and storage volumes.
  2. Silent Data Corruption & Bit Rot: Undetected degradation of persistent volume states over time that, if replicated blindly, corrupts snapshot histories.
  3. Ransomware / Malicious Host Compromise: Host-level breach where an adversary attempts to delete, corrupt, or encrypt local backups.
  4. Credential & Data Exposure at Rest: Unencrypted snapshots exposing sensitive database dumps, password manager secrets, and SSL/TLS keys if the offsite storage provider is compromised.

Technical Architecture & Backup Principles#

1. Client-Side Cryptographic Isolation#

All volume snapshots, database dumps, and compose configurations are encrypted locally with authenticated AES-256-GCM before transmitting over the network. The remote storage backend possesses zero decryption keys or plaintext visibility.

2. Deduplication & Snapshot Retention#

Utilizing content-defined chunking (via restic), snapshots achieve high storage efficiency through deduplication:

  • Daily snapshots retained for 7 days.
  • Weekly snapshots retained for 4 weeks.
  • Monthly snapshots retained for 6 months.

3. Database State Consistency#

Relational database engines (MariaDB/MySQL, SQLite, PostgreSQL) are never backed up via raw volume copying while live. The pipeline executes consistent transactional dumps (mysqldump / .backup) prior to snapshot creation to ensure table lock consistency and avoid half-written transactions.

The Recovery Validation Drill#

To satisfy the operational requirement that backups are only real after a verified restore, regular recovery drills are executed against an isolated staging target:

Drill Execution Workflow#

  1. Repository Integrity Check: Run integrity verification across the encrypted repository chunks:
restic check --read-data-subset=10%
  1. Cold Pull & Snapshot Extraction: Pull the latest snapshot into an isolated sandbox directory on an alternate machine:
restic restore latest --target /tmp/dr-drill/
  1. Database Spin-Up & Table Validation:
  • Spin up a detached database container pointing to the restored data dump, query row counts, and check schema validity against production metrics.
  1. Application Sanity Verification: Verify service configuration parsing and startup logs without errors.

Operational Verification#

  • Cryptographic Verification: Confirmed snapshots cannot be decrypted or indexed without the primary repository key and password.

  • Restore Reliability: Cold-restore drills consistently achieve recovery time objectives (RTO) under 15 minutes for core services.

  • Integrity Audit: Snapshot trees pass integrity hash audits with zero missing or orphaned blob references.

Key Takeaway#

Hope is not a recovery plan. Storage media will eventually fail, and automation will eventually encounter edge cases. Resilient operations treat backups as an active pipeline requiring regular restore validation, client-side encryption, and strict offsite isolation.