How to Point a Docker Image to My .M2 Directory for Running Maven in Docker on a MAC

How do I point a docker image to my .m2 directory for running maven in docker on a mac?

How do I point a docker image to my .m2 directory for running maven in docker on a mac?

You rather point a host folder (like /Users/myname/.m2) to a container folder (not an image)

See "Mount a host directory as a data volume":

In addition to creating a volume using the -v flag you can also mount a directory from your Docker daemon’s host into a container.

$ docker run -d -P --name web -v /Users/myname/.m2:/root/.m2 training/webapp python app.py

This command mounts the host directory, /Users/myname/.m2, into the container at /root/.m2.

If the path /root/.m2 already exists inside the container’s image, the /Users/myname/.m2 mount overlays but does not remove the pre-existing content.

Once the mount is removed, the content is accessible again.

This is consistent with the expected behavior of the mount command.

Multi Module Maven Project and Docker: Cannot find artifact?

Dependency com.company.parent:jee6:pom:1.0.1-SNAPSHOT seems to be private, your Maven command inside Docker build needs to be able to either download it from private repository or have it readily accessible.

I assume this issue because when trying to run the command in the maven docker image it cannot see my local .m2 folder?

Yes, it then cannot see your settings.xml with private repository config, or local dependency if it's already available locally.

Would also copying my maven settings.xml help?

It's better not to: your settings.xml (and eventual secrets within) may be available to anyone using your image later. Using a secret mount with BuildKit would be a better solution (see below)

---

You have multiple solutions:

Mount settings.xml as secret during build

This solution assumes you have a settings.xml configured with proper credentials to access private registry.

Use Docker BuildKit with --mount=secret to load settings.xml as secret with a Dockerfile such as:

# syntax=docker/dockerfile:1.2
# Required comment at top of Dockerfile for using BuildKit

FROM maven:3.5-jdk-8 AS build  

COPY module1 /usr/src/app/src  
COPY module2 /usr/src/app/src  
COPY module3 /usr/src/app/src  
COPY pom.xml /usr/src/app  

# Use your secret settings.xml
RUN --mount=type=secret,id=mvnsettings,target=/root/.m2/settings.xml \
  mvn -f /usr/src/app/pom.xml clean install

And build command such as:

DOCKER_BUILDKIT=1 docker build --secret id=mvnsettings,src=$HOME/.m2/settings.xml . 

Maven should then be able to download parent dependency during build.

Note: this is NOT COPYing the settings.xml in image, as the secret settings.xml will only be made available for the specified build step and won't be persisted in final image.

Copy com.company.parent:jee6 pom.xml during build

This solution is less practical and may not solve problem entirely:

• It would require to have com.company.parent:jee6:pom:1.0.1-SNAPSHOT pom.xml file available in build context
• Your parent pom.xml may refer to other private dependencies. You would have to include them the same way.

... But it still may be worth a try.

You can do something like:

FROM maven:3.5-jdk-8 AS build  

# Copy and install parent pom
COPY parent-pom.xml /tmp/parent/pom.xml
RUN mvn -f /tmp/parent/pom.xml clean install

COPY module1 /usr/src/app/src  
COPY module2 /usr/src/app/src  
COPY module3 /usr/src/app/src  
COPY pom.xml /usr/src/app  
RUN mvn -f /usr/src/app/pom.xml clean install

Where to store settings.xml for maven in docker based GitLab CI setup

I mount the host volume via the config.toml as the following example: -

concurrent = 1
check_interval = 30

[[runners]]
  name = "some-name"
  url = "url/to/gitlab/"
  token = "some-token"
  executor = "docker"
  [runners.docker]
    tls_verify = false
    image = "some-maven-image"
    privileged = false
    disable_cache = false
    volumes = ["...", "path/to/host/dir:/some/name:rw"]
    pull_policy = "if-not-present"
    shm_size = 0
  [runners.cache]

While the path/to/host/dir is a path on host machine which contains many files including with settings.xml, settings-security.xml and so on, based on project requirement. On the other hand the /some/name is a directory inside the docker.

At the .gitlab-ci.yml, I provide the before_script as the following example: -

before_script:
  - cp -f /some/name/settings.xml  $HOME/.m2/settings.xml
  - cp -f /some/name/settings-security.xml $HOME/.m2/settings-security.xml
  - ...
after_script:
  - rm -f $HOME/.m2/settings.xml
  - rm -f $HOME/.m2/settings-security.xml
  - ...

Please visit Advanced configuration for further information about the config.toml.

docker build with maven - how to prevent re-downloading dependencies

Before to tell you how I would process, I will explain the issue that you encounter.

Your Dockerfile relies on the build multi-stage feature.

Here stages are considered as intermediary layers that are not kept as layers in the final image. To keep files/folders between layers you have to explicit copy them as you done.

So concretely, it means that in the below instructions : maven resolves all dependencies specified in your pom.xml and it stores them in the local repository located on the layer of that stage :

FROM maven:3.5-jdk-8 as mavenDeps
COPY pom.xml pom.xml
RUN mvn dependency:resolve

But as said, the stage content is not kept by default. So all downloaded dependencies in the local maven repo are lost since you never copy that in the next stage :

FROM mavenDeps as mavenBuild
RUN mvn install

Since the local repo of that image is empty : mvn install re-download all dependencies.

---

How to process ?

You have really many many ways.

The best choice depends on your requirement.

But whatever the way, the build strategy in terms of docker layers looks like :

Build stage (Maven image) :

• pom copy to the image
• dependencies and plugins downloads.
  
  About that, mvn dependency:resolve-plugins chained to mvn dependency:resolve may do the job but not always.
  
  Why ? Because these plugins and the package execution may rely on different artifacts/plugins and even for a same artifact/plugin, these may still pull a different version.
  So a safer approach while potentially slower is resolving dependencies by executing exactly the mvn package command (which will pull exactly dependencies that you are need) but by skipping the source compilation and by deleting the target folder to make the processing faster and to prevent any undesirable layer change detection for that step.
• source code copy to the image
• package the application

Run stage (JDK or JRE image) :

• copy the jar from the previous stage

1) No explicit cache for maven dependencies : straight but annoying when pom changes frequently

If re-downloading all dependencies at every pom.xml change is acceptable.

Example by starting from your script :

########build stage########
FROM maven:3.5-jdk-8 as maven_build
WORKDIR /app

COPY pom.xml .
# To resolve dependencies in a safe way (no re-download when the source code changes)
RUN mvn clean package -Dmaven.main.skip -Dmaven.test.skip && rm -r target

# To package the application
COPY src ./src
RUN mvn clean package -Dmaven.test.skip

########run stage########
FROM java:8
WORKDIR /app

COPY --from=maven_build /app/target/*.jar

#run the app
ENV JAVA_OPTS ""
CMD [ "bash", "-c", "java ${JAVA_OPTS} -jar *.jar -v"]

Drawback of that solution ?
Any changes in the pom.xml means re-create the whole layer that download and stores the maven dependencies.

That is generally not acceptable for applications with many dependencies, overall if you don't use a maven repository manager during the image build.

2) Explicit cache for maven dependencies : require more configurations and use of buildkit but that is more efficient because only required dependencies are downloaded

The only thing that changes here is that maven dependencies download are cached in the docker builder cache :

# syntax=docker/dockerfile:experimental
########build stage########
FROM maven:3.5-jdk-8 as maven_build
WORKDIR /app

COPY pom.xml .    
COPY src ./src

RUN --mount=type=cache,target=/root/.m2 mvn clean package  -Dmaven.test.skip

########run stage########
FROM java:8
WORKDIR /app

COPY --from=maven_build /app/target/*.jar

#run the app
ENV JAVA_OPTS ""
CMD [ "bash", "-c", "java ${JAVA_OPTS} -jar *.jar -v"]

To enable buildkit, the env variable DOCKER_BUILDKIT=1 has to be set (you can do that where you want : bashrc, command line, docker daemon json file...)

Docker - failed to compute cache key: not found - runs fine in Visual Studio

The way Visual Studio does it is a little bit odd.

Instead of launching docker build in the folder with the Dockerfile, it launches in the parent folder and specifies the Dockerfile with the -f option.

I was using the demo project (trying to create a minimal solution for another question) and struck the same situation.

Setup for my demo project is

\WorkerService2  ("solution" folder)
   +- WorkerService2.sln
   +- WorkserService2  ("project" folder)
       +- DockerFile
       +- WorkerService2.csproj
       +- ... other program files

So I would expect to go

cd \Workerservice2\WorkerService2
docker build .

But I get your error message.

 => ERROR [build 3/7] COPY [WorkerService2/WorkerService2.csproj, WorkerService2/]                                                                                                                        0.0s
------
 > [build 3/7] COPY [WorkerService2/WorkerService2.csproj, WorkerService2/]:
------
failed to compute cache key: "/WorkerService2/WorkerService2.csproj" not found: not found

Instead, go to the parent directory, with the .sln file and use the docker -f option to specify the Dockerfile to use in the subfolder:

cd \Workerservice2
docker build -f WorkerService2\Dockerfile --force-rm -t worker2/try7 .

docker run -it worker2/try7    

Edit (Thanks Mike Loux, tblev & Goku):

Note the final dot on the docker build command.

For docker the final part of the command is the location of the files that Docker will work with. Usually this is the folder with the Dockerfile in, but that's what's different about how VS does it. In this case the dockerfile is specified with the -f. Any paths (such as with the COPY instruction in the dockerfile) are relative to the location specified. The . means "current directory", which in my example is \WorkerService2.

I got to this stage by inspecting the output of the build process, with verbosity set to Detailed.
If you choose Tools / Options / Projects and Solutions / Build and Run you can adjust the build output verbosity, I made mine Detailed.

Edit #2 I think I've worked out why Visual Studio does it this way.
It allows the project references in the same solution to be copied in.

If it was set up to do docker build from the project folder, docker would not be able to COPY any of the other projects in the solution in. But the way this is set up, with current directory being the solution folder, you can copy referenced projects (subfolders) into your docker build process.

How to disable maven blocking external HTTP repositories?

I found a solution to do this by inspecting the commit in the Maven git repository that is responsible for the default HTTP blocking: https://github.com/apache/maven/commit/907d53ad3264718f66ff15e1363d76b07dd0c05f

My solution is as follows:

In the Maven settings (located in ${maven.home}/conf/settings.xml or ${user.home}/.m2/settings.xml), the following entry must be removed:

<mirror>
  <id>maven-default-http-blocker</id>
  <mirrorOf>external:http:*</mirrorOf>
  <name>Pseudo repository to mirror external repositories initially using HTTP.</name>
  <url>http://0.0.0.0/</url>
</mirror>

If you work in a project and cannot make sure the Maven settings are always like that, e.g. because you share code with other people or want to use CI/CD with automated testing, you may do the following: Add a directory named .mvn in the project. In the .mvn directory, add a file named maven.config with the content --settings ./.mvn/local-settings.xml. In the .mvn directory, add a file named local-settings.xml. This file should look like this:

<settings xmlns="http://maven.apache.org/SETTINGS/1.2.0"
          xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
          xsi:schemaLocation="http://maven.apache.org/SETTINGS/1.2.0 http://maven.apache.org/xsd/settings-1.2.0.xsd">
    <mirrors>
        <mirror>
            <id>my-repository-http-unblocker</id>
            <mirrorOf>my-blocked-http-repository</mirrorOf>
            <name></name>
            <url>http://........</url>
        </mirror>
    </mirrors>
</settings>

Where inside the <mirrorOf> tag, you need to specify the id of the blocked repository, and in the <url> tag, you specify the original url of the repository again. You need to create this unblocker mirror for every repository you have that is blocked.

Example:

If you have the following HTTP repositories defined in the pom.xml:

<repositories>
    <repository>
        <snapshots>
            <enabled>false</enabled>
        </snapshots>
        <id>central</id>
        <name>libs-release</name>
        <url>http://my-url/libs-release</url>
    </repository>
    <repository>
        <id>snapshots</id>
        <name>libs-snapshot</name>
        <url>http://my-url/libs-snapshot</url>
    </repository>
</repositories>

Then you need in the .mvn/local-settings.xml:

<settings>
    <mirrors>
        <mirror>
            <id>release-http-unblocker</id>
            <mirrorOf>central</mirrorOf>
            <name></name>
            <url>http://my-url/libs-release</url>
        </mirror>
        <mirror>
            <id>snapshot-http-unblocker</id>
            <mirrorOf>snapshots</mirrorOf>
            <name></name>
            <url>http://my-url/libs-snapshot</url>
        </mirror>
    </mirrors>
</settings>

I hope my work can help other people who stumble upon this. However, if you have a more elegant or better solution, please share!

---

Related Topics