It built fine and still crashed: Docker file ownership vs runtime user

A Docker build runs as root. Your container, if you have configured it properly, does not. Every file the build creates belongs to root, and every write your application makes at runtime is attempted by somebody else. A green build tells you the artifacts exist. It says nothing about whether the process that has to use them is allowed to.

I have now hit this in four stacks that share no code: a Node SSR app, a PHP framework, a search engine, and a log aggregator. It presents differently each time, which is why it kept catching me.

The shape of it

BUILD                                RUNTIME
docker build (uid 0)                 USER app (uid 1001)
  writes  /app/.next/cache             wants to write /app/.next/cache
  owner:  root:root                    permission: denied
      │                                        │
      └──────────► same files ◄────────────────┘
                   different identity

Nothing is wrong with either half. The build succeeded honestly and the runtime failed honestly. They just never agreed about who owns the output.

Four symptoms, one cause

Same class of bug in every row. The differences are cosmetic: whether the process crashes loudly, fails silently, or comes up and does nothing.

The last two are the worse variants, because a mounted volume arrives with the ownership the host gave it, so the image can be flawless and the pod still fails on a different node.

The fix, in the image

Create the user, create the directories the runtime needs, and hand over ownership as part of the copy rather than as a chown -R afterwards:

RUN addgroup --system --gid 1001 nodejs \
 && adduser  --system --uid 1001 nextjs

RUN mkdir -p /app/.next/cache \
 && chown -R nextjs:nodejs /app/.next

COPY --from=builder --chown=nextjs:nodejs /app/.next/standalone ./
COPY --from=builder --chown=nextjs:nodejs /app/.next/static     ./.next/static
COPY --from=builder --chown=nextjs:nodejs /app/public           ./public

USER nextjs

--chown on the COPY matters more than it looks. Doing it in a later RUN chown -R duplicates the whole layer, which on a large build output is a real cost in image size and pull time.

For mounted volumes, the image cannot solve it alone. The orchestrator has to set the group that owns the mount:

securityContext:
  runAsNonRoot: true
  runAsUser:  1001
  runAsGroup: 1001
  fsGroup:    1001    # only needed when a volume is mounted

fsGroup is the one that fixes the search engine and log aggregator rows in that table. It tells the kubelet to set group ownership on the volume before the container starts. Without it, a perfectly built image fails on storage it did not create.

The fix I keep seeing instead

RUN chmod -R 777 /app

This works. It also makes every file in the image world-writable, which means any process that gets execution inside the container can rewrite your application code, and you have converted a permissions bug into a permanent weakening of the container. It does not fix the mounted-volume case either, because the mount arrives after the image is built.

The reason it is tempting is that it is the only fix that requires no understanding of which directory the failure was about. That is exactly the reason not to use it. Finding out which path needs writing is the useful half of the work.

Finding the path in ten seconds

The error usually names it, but not always, and PHP-style frameworks are notorious for failing to log the failure to write the log:

# what does the process think it is?
kubectl exec -it <pod> -- id

# what does it own?
kubectl exec -it <pod> -- ls -ld /app/.next/cache /app/storage

# what did it try, if the app was quiet about it?
kubectl logs <pod> --previous

id and ls -ld next to each other answer the question in one screen. If the uid from the first does not appear in the owner or group of the second, and the directory is not group-writable, you have found it.

What generalises

The container case is the loud one, but the rule underneath is older than containers, and it is worth stating on its own:

A process that creates a resource and a process that uses it are two different processes. Whenever their identities differ, ownership is a thing you have to specify, not a thing you can assume.

That covers build user versus runtime user, a CI job writing an artifact that a deploy user later reads, a cron job creating a file the web server needs, and a volume created by a host that a pod later mounts. In every one of those, the creating side succeeds and the using side fails, which is why the failure always looks like it belongs to the innocent party.

The checklist I run on any new image now is short. Which uid does this run as in production. Which paths does it write at runtime, including caches and temporary directories that never appear in the source tree. Are those paths created and owned before USER is set. And if any of them is a mounted volume, is fsGroup set to match.

Four questions. They would have saved me four separate afternoons.