How to Have a Host and Container Read/Write the Same Files with Docker

How can I have a host and container read/write the same files with Docker?

There are multiple ways to do this, but the central issue is that bind mounts do not include any UID mapping capability, the UID on the host is what appears inside the container and vice versa. If those two UID's do not match, you will read/write files with different UID's and likely experience permission issues.

Option 1: get a Mac or deploy docker inside of VirtualBox. Both of these environments have a filesystem integration that dynamically updates the UID's. For Mac, that is implemented with OSXFS. Be aware that this convenience comes with a performance penalty.

Option 2: Change your host. If the UID on the host matches the UID inside the container, you won't experience any issues. You'd just run a usermod on your user on the host to change your UID there, and things will happen to work, at least until you run a different image with a different UID inside the container.

Option 3: Change your image. Some will modify the image to a static UID that matches their environment, often to match a UID in production. Others will pass a build arg with something like --build-arg UID=$(id -u) as part of the build command, and then the Dockerfile with something like:

FROM alpine
ARG UID=1000
RUN adduser -u ${UID} app

The downside of this is each developer may need a different image, so they are either building locally on each workstation, or you centrally build multiple images, one for each UID that exists among your developers. Neither of these are ideal.

Option 4: Change the container UID. This can be done in the compose file, or on a one off container with something like docker run -u $(id -u) your_image. The container will now be running with the new UID, and files in the volume will be accessible. However, the username inside the container will not necessarily map to your UID which may look strange to any commands you run inside the container. More importantly, any files own by the user inside the container that you have not hidden with your volume will have the original UID and may not be accessible.

Option 5: Give up, run everything as root, or change permissions to 777 allowing everyone to access the directory with no restrictions. This won't map to how you should run things in production, and the container may still write new files with limited permissions making them inaccessible to you outside the container. This also creates security risks of running code as root or leaving filesystems open to both read and write from any user on the host.

Option 6: Setup an entrypoint that dynamically updates your container. Despite not wanting to change your image, this is my preferred solution for completeness. Your container does need to start as root, but only in development, and the app will still be run as the user, matching the production environment. However, the first step of that entrypoint will be to change the user's UID/GID inside the container to match your volume's UID/GID. This is similar to option 4, but now files inside the image that were not replaced by the volume have the right UID's, and the user inside the container will now show with the changed UID so commands like ls show the username inside the container, not a UID to may map to another user or no one at all. While this is a change to your image, the code only runs in development, and only as a brief entrypoint to setup the container for that developer, after which the process inside the container will look identical to that in a production environment.

To implement this I make the following changes. First the Dockerfile now includes a fix-perms script and gosu from a base image I've pushed to the hub (this is a Java example, but the changes are portable to other environments):

FROM openjdk:jdk as build
# add this copy to include fix-perms and gosu or install them directly
COPY --from=sudobmitch/base:scratch / /
RUN  apt-get update \
 &&  apt-get install -y maven \
 &&  useradd -m app
COPY code /code
RUN  mvn build
# add an entrypoint to call fix-perms
COPY entrypoint.sh /usr/bin/
ENTRYPOINT ["/usr/bin/entrypoint.sh"]
CMD ["java", "-jar", "/code/app.jar"]
USER app

The entrypoint.sh script calls fix-perms and then exec and gosu to drop from root to the app user:

#!/bin/sh
if [ "$(id -u)" = "0" ]; then
  # running on a developer laptop as root
  fix-perms -r -u app -g app /code
  exec gosu app "$@"
else
  # running in production as a user
  exec "$@"
fi

The developer compose file mounts the volume and starts as root:

version: '3.7'
volumes:
  m2:
services:
  app:
    build:
      context: .
      target: build
    image: registry:5000/app/app:dev
    command: "/bin/sh -c 'mvn build && java -jar /code/app.jar'"
    user: "0:0"
    volumes:
    - m2:/home/app/.m2
    - ./code:/code

This example is taken from my presentation available here: https://sudo-bmitch.github.io/presentations/dc2019/tips-and-tricks-of-the-captains.html#fix-perms

Code for fix-perms and other examples are available in my base image repo: https://github.com/sudo-bmitch/docker-base

I can share files between Docker containers, how do I share the same files with the host

Solution

I got some clues from here: https://forums.docker.com/t/how-to-access-docker-volume-data-from-host-machine/88063
Meyay’s answer “Only with named volumes existing data is copied from the container target folder into the volume.”

This code works. Please note: /tmp/data must pre-exist on the host, with the same permissions (and owner I think) as the docker container.

celery_worker:
    image: celery_worker:latest
    container_name: celery_worker
    volumes:
      - data:/srv/project/hardware/hardware_data/

hardware:
    image: hardware:latest
    container_name: hardware
    volumes:
      - data:/home/hardware/data/

volumes:
  data:
    driver: local
    driver_opts:
      o: bind
      type: none
      device: /tmp/data

Dockerized executable read/write on host filesystem

You're correct, you need to pass in /this/is as a volume and the executable will write to that location.

If you want to constrain the thing even more, you can pass /this/is/b.file as a volume. You need to create it (simply via touch) beforehand, otherwise Docker will consider it a directory and create it as such for you, but you'll know that the thing won't be able to create /this/is/c.file or any other thing.

The same file on host and inside container is different, it is not in sync with its original

When you mount an individual file into a container, what you are mounting is the inode of the file on the linux filesystem. If you replace the file (which many editors do), then the inode changes. That's not normally an issue if you mount an entire directory because the directory's pointer to the inode gets updated. But when only mounting a single file, writes to the file after starting the container will not be reflected between the inside and outside of the container.

For more details, see this issue: https://github.com/moby/moby/issues/6011

Edit: If your goal is to modify a file outside of the container, and have that modify a single file in a directory of files that can't be in the volume mount, then I'd recommend changing the container to have a symlink to a file in a different directory where you can mount the entire directory instead of a single file. That would look like:

In your Dockerfile:

RUN mkdir -p /var/docker/composer \
 && ln -s /var/docker/composer/composer.json /var/www/d8-default/composer.json

And your volume mount would then be:

docker run -v /home/a_user/projects/drupal/d8-default:/var/docker/composer ...

Allow Docker Container & Host User To Write on Bind Mounted Host Directory

Problem: if I set "ubuntu" as owner, container can't write (using php to write), if I set "nobody" as owner, VSCode SSH can't write. I am finding a way to allow both to write without changing directory owner user again and again, or similar ease.

First, I'd recommend the container image should create a new username for the files inside the container, rather than reusing nobody since that user may also be used for other OS tasks that shouldn't have any special access.

Next, as Triet suggests, an entrypoint that adjusts the container's user/group to match the volume is preferred. My own version of these scripts can be found in this base image that includes a fix-perms script that makes the user id and group id of the container user match the id's of a mounted volume. In particular, the following lines of that script where $opt_u is the container username, $opt_g is the container group name, and $1 is the volume mount location:

# update the uid
if [ -n "$opt_u" ]; then
  OLD_UID=$(getent passwd "${opt_u}" | cut -f3 -d:)
  NEW_UID=$(stat -c "%u" "$1")
  if [ "$OLD_UID" != "$NEW_UID" ]; then
    echo "Changing UID of $opt_u from $OLD_UID to $NEW_UID"
    usermod -u "$NEW_UID" -o "$opt_u"
    if [ -n "$opt_r" ]; then
      find / -xdev -user "$OLD_UID" -exec chown -h "$opt_u" {} \;
    fi
  fi
fi

# update the gid
if [ -n "$opt_g" ]; then
  OLD_GID=$(getent group "${opt_g}" | cut -f3 -d:)
  NEW_GID=$(stat -c "%g" "$1")
  if [ "$OLD_GID" != "$NEW_GID" ]; then
    echo "Changing GID of $opt_g from $OLD_GID to $NEW_GID"
    groupmod -g "$NEW_GID" -o "$opt_g"
    if [ -n "$opt_r" ]; then
      find / -xdev -group "$OLD_GID" -exec chgrp -h "$opt_g" {} \;
    fi
  fi
fi

Then I start the container as root, and the container runs the fix-perms script from the entrypoint, followed by a command similar to:

exec gosu ${container_user} ${orig_command}

This replaces the entrypoint that's running as root with the application running as the specified user. I've got more examples of this in:

DockerCon presentation
Similar SO questions

What I tried: In Container, I added user "nobody" to group "ubuntu".
On host, directory (used as mount) was set "sudo chown -R
ubuntu:ubuntu directory", user "ubuntu" was already added to group
"ubuntu". VSCode did edit, container was unable to edit.

I'd avoid this and create a new user. Nobody is designed to be as unprivileged as possible, so there could be unintended consequences with giving it more access.

Edit: the container already created without Dockerfile also ran and
maybe edited with important changes, so maybe I can't use Dockerfile
or entrypoint.sh way to solve problem. Can It be achieved through
running commands inside container or without creating container again?
This container can be stopped.

This is a pretty big code smell in containers. They should be designed to be ephemeral. If you can't easily replace them, you're missing the ability to upgrade to a newer image, and creating a lot of state drift that you'll eventually need to cleanup. Your changes that should be preserved need to be in a volume. If there are other changes that would be lost when the container is deleted, they will be visible in docker diff and I'd recommend fixing this now rather than increasing the size of the technical debt.

Edit: I am wondering, in Triet Doan's answer, an option is to modify
UID and GID of already created user in the container, will doing this
for the user and group "nobody" can cause any problems inside
container, I am wondering because probably many commands for settings
already executed inside container, files are already edited by php on
mounted directory & container is running for days

I would build a newer image that doesn't depend on this username. Within the container, if there's data you need to preserve, it should be in a volume.

Edit: I found that alpine has no usermod & groupmod.

I use the following in the entrypoint script to install it on the fly, but the shadow package should be included in the image you build rather than doing this on the fly for every new container:

if ! type usermod >/dev/null 2>&1 || \
   ! type groupmod >/dev/null 2>&1; then
  if type apk /dev/null 2>&1; then
    echo "Warning: installing shadow, this should be included in your image"
    apk add --no-cache shadow
  else
    echo "Commands usermod and groupmod are required."
    exit 1
  fi
fi

How to copy files from host to Docker container?

The cp command can be used to copy files.

One specific file can be copied TO the container like:

docker cp foo.txt container_id:/foo.txt

One specific file can be copied FROM the container like:

docker cp container_id:/foo.txt foo.txt

For emphasis, container_id is a container ID, not an image ID. (Use docker ps to view listing which includes container_ids.)

Multiple files contained by the folder src can be copied into the target folder using:

docker cp src/. container_id:/target
docker cp container_id:/src/. target

Reference: Docker CLI docs for cp

In Docker versions prior to 1.8 it was only possible to copy files from a container to the host. Not from the host to a container.