RunbooksDocker volumes

Dirty Docker volumes

Applies to: all

Symptom

After a previous run left bad state on disk, services refuse to come up clean:

hivecfm-postgres  | FATAL:  database files are incompatible with server
hivecfm-postgres  | DETAIL:  The data directory was initialized by PostgreSQL version 15, which is not compatible with this version 16.x.
hivecfm-minio     | FATAL Unable to initialize backend: Access key invalid
hivecfm-superset  | ERROR - superset.extensions.metastore_cache: IntegrityError ... UNIQUE constraint failed

Or simply: containers enter a crash loop, docker compose ps shows Restarting (1) on repeat.

Likely cause

Docker ComposeDocker ComposeThe YAML file and CLI that spin up Postgres, Redis, MinIO, MailHog locally with one command. persists state in named volumes. hivecfm-core declares these in docker-compose.yml:

Compose volumeHost nameContents
postgres_datahivecfm-postgres-dataPostgres cluster (PGDATA)
redis_datahivecfm-redis-dataRedisRedisIn-memory key-value store used for caching, sessions, and rate limiting. Runs on port 6380 locally. append-only file
minio_datahivecfm-minio-dataS3 object buckets
hivecfm_uploadshivecfm-uploadsUploaded user files
hivecfm_samlhivecfm-samlSAMLSAMLThe XML-based enterprise SSO protocol HiveCFM supports for customers using Okta, Azure AD, or similar IdPs. connection store
superset_datahivecfm-superset-dataSuperset SQLite metadata DB

A previous run left data that is incompatible with the current image version, has stale credentials, or is half-migrated. The container fails to initialise because the volume is not empty.

Fix

Identify which volume is dirty

docker compose ps
docker compose logs --tail=200 postgres
docker volume ls | grep hivecfm

Read the failing container’s logs first — it will usually name the path or the complaint (version mismatch, key mismatch, schema drift).

Stop the stack

cd /home/ubuntu/AG-DEV/hivecfm-core
docker compose down

down alone stops containers and removes the network but keeps volumes.

Remove just the bad volume (preferred)

⚠️

docker volume rm permanently deletes the volume’s contents. Back up anything you need first — especially hivecfm-uploads and hivecfm-postgres-data.

# Example: reset only Postgres
docker volume rm hivecfm-postgres-data
 
# Example: reset only MinIO credentials / buckets
docker volume rm hivecfm-minio-data

Start the stack again:

docker compose -f docker-compose.yml -f docker-compose.dev-ports.yml up -d

The missing volume is recreated empty and re-initialised from the container’s entrypoint.

Nuke everything (last resort)

⚠️

docker compose down -v removes all named volumes in the compose project at once — Postgres, Redis, MinIO, uploads, SAML, Superset. Every row of local data is destroyed. Do not run this against anything you care about.

docker compose down -v
docker compose -f docker-compose.yml -f docker-compose.dev-ports.yml up -d
# After the stack is up, re-run migrations and seed:
pnpm --filter @hivecfm/database db:setup

Verify

docker compose ps              # all services "running" / "healthy"
docker compose logs postgres | tail -20   # "database system is ready to accept connections"
curl -s http://localhost:3001  # HTML response

Prevent

  • When you upgrade the Postgres or Redis image version, delete the matching volume in the same commit — mixed versions will not coexist.
  • If you rotate MinIO credentials in .env, also delete hivecfm-minio-data so MinIO re-initialises with the new keys.
  • Prefer docker volume rm <specific-volume> over down -v — it lets you reset one service without wiping uploads.
  • For a full wipe-and-restart from a clean slate, use the documented reset: ./scripts/deploy-local.sh --reset.