Skip to content

Operations Playbook: Database Incident

Trigger Conditions

  • migration failures
  • DB connectivity errors
  • query-time runtime failures
  • schema mismatch symptoms

Severity Assessment

Indicator Severity Suggestion
migration failure on deploy path high
transient DB timeout with auto-recovery medium
persistent query failure on critical path high

Response Workflow

  1. Check service and log health.
  2. Validate migration state.
  3. Verify DB credentials/connectivity.
  4. Isolate failing migration/query.
  5. Apply controlled remediation.
  6. Re-run checks and smoke paths.
docker compose ps
docker compose logs -f db app
docker compose run --rm app python manage.py showmigrations
docker compose run --rm app python manage.py check

Post-Remediation Validation

  • critical domain pages load
  • migrations are consistent
  • no new DB errors in logs
Risk control

Never apply ad hoc production schema changes outside controlled migration workflows.