Cmake: Don't Set Rpath for a Single Library Used in Link

cmake: how to set rpath in a shared library with only target_link_directories (without target_link_libraries)?

Here's a working build. You just need to adjust the BUILD_RPATH property appropriately and, if you write install() rules, adjust your INSTALL_RPATH to be similar to what I wrote in this answer. The following build is robust and adjusts the BUILD_RPATH:

cmake_minimum_required(VERSION 3.23)
project(test)

add_library(bar SHARED libs/bar.c)

add_library(foo MODULE libs/foo.c)
target_link_libraries(foo PRIVATE bar)

add_executable(main main.c)
target_link_libraries(main PRIVATE ${CMAKE_DL_LIBS})
set_property(TARGET main APPEND PROPERTY BUILD_RPATH "$<TARGET_FILE_DIR:foo>")

The last two lines are important. You must link to CMAKE_DL_LIBS to portably call dlopen and friends. The second line makes sure the directory containing libfoo, which you know main will load, is in the RPATH.

Here's the console output:

$ cmake -G Ninja -S . -B build -DCMAKE_BUILD_TYPE=RelWithDebInfo
-- The C compiler identification is GNU 9.4.0
-- The CXX compiler identification is GNU 9.4.0
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: /usr/bin/cc - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: /usr/bin/c++ - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Configuring done
-- Generating done
-- Build files have been written to: /home/alex/test/build

$ cmake --build build/ --verbose
[1/6] /usr/bin/cc -Dbar_EXPORTS  -O2 -g -DNDEBUG -fPIC -MD -MT CMakeFiles/bar.dir/libs/bar.c.o -MF CMakeFiles/bar.dir/libs/bar.c.o.d -o CMakeFiles/bar.dir/libs/bar.c.o -c /home/alex/test/libs/bar.c
[2/6] /usr/bin/cc -Dfoo_EXPORTS  -O2 -g -DNDEBUG -fPIC -MD -MT CMakeFiles/foo.dir/libs/foo.c.o -MF CMakeFiles/foo.dir/libs/foo.c.o.d -o CMakeFiles/foo.dir/libs/foo.c.o -c /home/alex/test/libs/foo.c
[3/6] /usr/bin/cc   -O2 -g -DNDEBUG -MD -MT CMakeFiles/main.dir/main.c.o -MF CMakeFiles/main.dir/main.c.o.d -o CMakeFiles/main.dir/main.c.o -c /home/alex/test/main.c
[4/6] : && /usr/bin/cc -fPIC -O2 -g -DNDEBUG   -shared -Wl,-soname,libbar.so -o libbar.so CMakeFiles/bar.dir/libs/bar.c.o   && :
[5/6] : && /usr/bin/cc -O2 -g -DNDEBUG  CMakeFiles/main.dir/main.c.o -o main  -Wl,-rpath,/home/alex/test/build  -ldl && :
[6/6] : && /usr/bin/cc -fPIC -O2 -g -DNDEBUG   -shared  -o libfoo.so CMakeFiles/foo.dir/libs/foo.c.o  -Wl,-rpath,/home/alex/test/build  libbar.so && :

$ ./build/main
here in libfoo!
here in libbar!

Answering some follow-up questions from the comments:

[W]hat is $<TARGET_FILE_DIR:foo>? Is TARGET_FILE_DIR a CMake variable (visible in CMakeLists.txts)?

It is not a variable, it is a generator expression. The values of these expressions are determined after the whole configuration step has been executed. In this way, we can be sure that this expression will expand to the actual directory containing libfoo.so, not merely the one we expect will contain it.

In general, I prefer to use generator expressions rather than variables whenever I can. They tend to lend CMake programming a more declarative rather than imperative feel and have fewer edge cases. For example, a user might set the value of CMAKE_RUNTIME_OUTPUT_DIRECTORY to something unexpected. This breaks your build if you compute the RPATH from CMAKE_CURRENT_BINARY_DIR or something.

[C]an you speak to the difference between target_link_libraries(main dl) (my version) and target_link_libraries(main PRIVATE ${CMAKE_DL_LIBS}) (your version)?

There are two differences here, both important:

Using target_link_libraries without a visibility specifier puts it into a weird limbo state that is sort of like PRIVATE, maybe, depending on the policy settings and whether or not any other calls to target_link_libraries have a visibility specifier. Both for clarity and to avoid these pitfalls, you should always specify one of PRIVATE, INTERFACE, or PUBLIC.
Using CMAKE_DL_LIBS is the correct way to link to whichever library contains the dlopen family of library functions. Did you know that HP-UX uses -ldld? Or that AIX uses -lld? Or that BSDs (including macOS) don't have a separate library for that? Well, passing -ldl is broken for those platforms.

Using target_link_libraries without a visibility specifier or passing it raw link flags are both serious code smells in the modern (post CMake ~3.5) era. Try to avoid them and ask questions when you don't think you can.

How to link a shared library with CMake with relative path

From the documentation:

By default if you don't change any RPATH related settings, CMake will link the executables and shared libraries with full RPATH to all used libraries in the build tree.

This is the behaviour you are seeing.

However, there are a number of ways to change this to match the behaviour you require.

Some examples from the above linked docs:

# use, i.e. don't skip the full RPATH for the build tree
SET(CMAKE_SKIP_BUILD_RPATH  FALSE)
    
# when building, don't use the install RPATH already
# (but later on when installing)
SET(CMAKE_BUILD_WITH_INSTALL_RPATH FALSE) 
    
# the RPATH to be used when installing
SET(CMAKE_INSTALL_RPATH "")
    
# don't add the automatically determined parts of the RPATH
# which point to directories outside the build tree to the install RPATH
SET(CMAKE_INSTALL_RPATH_USE_LINK_PATH FALSE)

Using the above you'll likely want to set CMAKE_INSTALL_RPATH and then distribute the installed binary.

If you want to distribute from the binary in your build tree, it is also possible to bypass CMake's rpath handling and modify the rpath directly using linker flags:

set_target_properties(game PROPERTIES LINK_FLAGS "-Wl,-rpath,./")

How to set RPATH in CMAKE?

I am using the following two lines in the CMakefile

set(CMAKE_MACOSX_RPATH 1)
set(CMAKE_INSTALL_RPATH "${CMAKE_INSTALL_PREFIX}/lib")

(the first one is required only if you use MacOSX)

Cannot get Cmake to bake rpath into executable if prefix used

It is not so obvious from the article, but variables CMAKE_INSTALL_RPATH and CMAKE_INSTALL_RPATH_USE_LINK_PATH should be set before creation of the target:

cmake_minimum_required( VERSION 3.14 )
project( cmakeprogram VERSION 1.0 )

# All further targets on installation will have given RPATH.
set( CMAKE_INSTALL_RPATH "${CMAKE_INSTALL_PREFIX}/lib" )

add_executable( program program.cxx )

add_subdirectory( lib )

Funny thing that even documentation of CMAKE_INSTALL_RPATH variable is misleading:

This is used to initialize the target property INSTALL_RPATH for all targets.

But description of the INSTALL_RPATH correctly specifies the situation:

This property is initialized by the value of the variable CMAKE_INSTALL_RPATH if it is set when a target is created.

Actually, in CMake initialization of the most target properties from variables follows the clause bolded above.

CMake: add packages to rpath

The observed behavior is described in the CMake wiki (thanks, Tsyvarev!)

By default if you don't change any RPATH related settings, CMake will
link the executables and shared libraries with full RPATH to all used
libraries in the build tree. When installing, it will clear the RPATH of
these targets so they are installed with an empty RPATH.

This means you need to tell CMake to add the automatically determined parts of the RPATH which point to directories outside the build tree to the install RPATH

SET(CMAKE_INSTALL_RPATH_USE_LINK_PATH TRUE)

In other words, 1) yes, CMake automatically sets the rpath, but only for the build tree by default. 2) CMake can set the rpath for the installed executable as well, it just has to be told to do so, as shown above.