When a container can't read its own secret — Docker secrets & non-root users

The notes site was down after a wake: its MongoDB container was crashlooping, and the app logged getaddrinfo ENOTFOUND mongo — it couldn't even find its database. The database's own logs told the real story.

The symptom

notes-mongo-1   Restarting (1) 33 seconds ago     # 17+ restarts
# docker logs notes-mongo-1:
/usr/local/bin/docker-entrypoint.sh: line 83: /run/secrets/mongo_password: Permission denied

The app's ENOTFOUND mongo was a downstream symptom: because the mongo container kept dying, it never joined the compose network, so its service name mongo never resolved. Fix the database and the app heals itself.

Root cause: the entrypoint runs as uid 999, the secret is root-only

The compose file feeds the root password as a bind-mounted secret file:

environment:
  MONGO_INITDB_ROOT_PASSWORD_FILE: /run/secrets/mongo_password
volumes:
  - ./data/secrets/mongo.password:/run/secrets/mongo_password:ro

The mongo:7 entrypoint starts as root, then drops privileges to the mongodb user (uid 999) via gosu and re-execs itself. That second pass is what reads the password file (val="$(< "$fileVar")"). But on the host the file was:

-rw------- root:root  mongo.password      # 0600, owner-only

A bind mount preserves host ownership/mode verbatim, so inside the container the file is 0600 root:root too — and uid 999 cannot read it. The redirect fails, the entrypoint exits 1 before mongod ever starts, and Docker restarts it forever.

The DB was already initialised (data owned by uid 999 from a prior run), so the password wasn't even needed to create the user — but the entrypoint reads the _FILE unconditionally and dies on the read. The permission denial blocks startup regardless of whether the value is used.

The fix: make it readable by the runtime uid — safely

The file lives inside data/secrets/, which is drwx------ root (0700). That directory is the real host-side guard: no non-root user can even traverse into it. So loosening the file to 0644 is safe on the host, and lets uid 999 read it inside the container:

chmod 0644 data/secrets/mongo.password
# verify the container's runtime user can now read it:
docker run --rm -u 999:999 \
  -v /root/apps/notes/data/secrets/mongo.password:/run/secrets/mongo_password:ro \
  mongo:7 sh -c 'cat /run/secrets/mongo_password >/dev/null && echo READABLE'

Then docker compose up -d mongo → healthy, up -d app → "Nest application successfully started", and https://notes.oaoisme.top/ returns 200.

Why not the alternatives

chown 999:999 — works, but on this host uid 999 is the dnsmasq service account, so it hands a host service nominal ownership of the secret. chmod 0644 inside the 0700 dir keeps it root-owned and is simpler.
Run mongo as root (user: "0:0") — mongod warns and it's discouraged; the data dir is already uid-999-owned anyway.
Pass the password as a plain MONGO_INITDB_ROOT_PASSWORD env — visible in docker inspect; worse than a file.

Generalise it

Any image that drops to a non-root user (mongo→999, postgres→999, redis, nginx workers, …) must have its bind-mounted secrets/keys readable by that runtime uid — a 0600 root:root file mounted in will deny the very process that needs it. Protect the directory (0700) and let the file be group/other readable, or chown it to the container's uid. When a container crashloops, always read its own logs before the app's — the app's error (ENOTFOUND, connection refused) is usually just an echo of the dependency that won't stay up.