Classification: Operational Lifecycle & Database Resilience
Status: Implemented & Verified

Executive Summary#

This report documents the failure modes, triage, and safe maintenance patterns discovered while updating multi-container microservice stacks with tightly coupled database dependencies. It analyzes race conditions during parallel image layer pulls, container recreations vs. image caching, and ensuring zero data loss during automated backend schema migrations.

The Problem Statement & Failure Modes#

Updating production-grade container stacks (such as Firefly III, its importer, cron sidecar, and MariaDB/MySQL backend) presents subtle operational risks if treated like simple stateless containers:

  1. Parallel Layer Pull Collisions & Image Skew: When executing docker compose pull on multi-container stacks sharing base layers, parallel engine routines can skip pulling downstream services (e.g., reporting Image is already being pulled by cron). If unverified, executing docker compose up -d recreates only a subset of services, leaving dependent containers running on disparate, out-of-sync image layers.
  2. Database Schema Locking & Partial Migrations: Large framework updates (such as Laravel migrations) alter table schemas, backfill ledger columns, and generate new encryption keys upon boot. Forcibly interrupting or rebooting the container while migrations execute corrupts the database state.
  3. Compose Service vs. Container Name Ambiguity: Service definitions inside docker-compose.yml (e.g., app) often differ from runtime container names (container_name: firefly_iii), leading to silent targeting failures during isolated CLI management.

Diagnostic & Architectural Analysis#

  • Layer Drift Detection: Observed that while database and importer images successfully updated, the primary application container remained in a Running (0.0s) state, failing to ingest the newly pulled base image due to compose target skipping.
  • Migration Timing: Monitored container startup latency (17.5s bootstrap cycle) confirming active execution of php artisan migrate, route cache clearance, and OAuth Passport key re-validations.

Safe Maintenance & Remediation Procedures#

1. Synchronized Image Pulling#

Force a full pull across all stack definitions and audit skipped services:

doas docker compose pull

2. Targeted Force-Recreation#

Explicitly recreate services on updated image layers using the internal compose service name rather than the runtime container name:

doas docker compose up -d --force-recreate app cron

3. Non-Destructive Startup & Migration Monitoring#

Never interrupt the initial boot sequence. Monitor the live log stream to ensure migrations complete cleanly:

doas docker logs -f firefly_iii

Verify:

  • Migrating: … batch tables settle cleanly.

  • Configuration cache cleared! and Routes cached successfully! complete.

  • HTTP daemon outputs ready status and begins serving requests.

4. Storage Reclamation#

Reclaim gigabytes of orphaned, superseded layer hashes across the host filesystem once containers are verified healthy:

doas docker image prune -f

Verification & Operational State

  • Schema Integrity: Confirmed MariaDB schema alignment with zero PDO or foreign key mismatch errors.

  • Synchronized Services: Verified via lazydocker that app, cron, importer, and db are operating on identical upstream release tags.

  • Clean Ingress: Validated zero dropped packets or 502 Bad Gateway responses at the reverse proxy layer during service handoff.

Key Takeaway#

Pulling an image is not deploying an update. Multi-tier application stacks require deterministic recreation commands, runtime migration patience, and log verification before declaring a maintenance window closed.