Ld: Using -Rpath,$Origin Inside a Shared Library (Recursive)

How to include correctly -Wl,-rpath,$ORIGIN linker argument in a Makefile?

No, you're misunderstanding. You need to pass the literal string $ORIGIN/../lib as an argument to your linker. The $ORIGIN token is kept inside your program after it's created and when the runtime linker starts to run your program it will replace $ORIGIN with the current path that your program was invoked from. This is true even if you've copied your program somewhere else. So if you run your program as /usr/local/bin/myprogram then the runtime linker will replace $ORIGIN with /usr/local/bin. If you copy it to /opt/mystuff/libexec/myprogram then the runtime linker will replace $ORIGIN with /opt/mystuff/libexec.

In order to pass a literal $ to the command invoked by a make recipe, you have to escape the $ by doubling it: $$. Otherwise, make will see the $ as introducing a make variable or function. Remember, it's perfectly legal for a make variable to avoid the parentheses etc., if it's a single character (note, $@, $<, etc.)

So when you write -Wl,-rpath,$ORIGIN/../lib make will interpret the $O in $ORIGIN as expanding a variable named O, which is empty, giving you -Wl,-rpath,RIGIN/../lib.

Also you have to escape the $ from the shell, otherwise it will try to expand $ORIGIN as a shell variable which you don't want.

You want to do something like this:

LDFLAGS = '-Wl,-rpath,$$ORIGIN/../lib' -L/usr/local/lib
LDLIBS = -lPocoFoundation -lPocoNet -lPocoUtil

$(TARGET): $(OBJECTS)
        @echo " Linking..."
        $(CC) $^ -o $@ $(LDFLAGS) $(LDLIBS)

(I don't know why you use @ to hide the command, then echo the command... why not just take out the @ and the echo and let make show you the command?)

How to set the path that a .so library will search for other .so libraries?

The bottom line is that your final executable must know where your library resides. You can accomplish that 2 ways (1) exporting the LD_LIBRARY_PATH that includes the path to your library, or (2) using rpath so your executable knows where to find your library. Exporting the LD_LIBRARY_PATH generally looks something like this:

LD_LIBRARY_PATH=/path/to/your/lib:${LD_LIBRARY_PATH}
export LD_LIBRARY_PATH

I prefer to use rpath. To use rpath compile your library as normal (example below for my extended test function library libetf.so)

gcc -fPIC -Wall -W -Werror -Wno-unused -c -o lib_etf.o lib_etf.c
gcc -o libetf.so.1.0 lib_etf.o -shared -Wl,-soname,libetf.so.1

Then to compile an executable making use of this library, you compile to object, then link the object with rpath given as a linker option. You would provide a path for both libA.so and libB.so in your case. Building my testso executable:

gcc -O2 -Wall -W -Wno-unused -c -o testetf.o testetf.c
gcc -o testso testetf.o -L/home/david/dev/src-c/lib/etf -Wl,-rpath=/home/david/dev/src-c/lib/etf -letf

Use ldd to confirm that the executable has correctly located your library:

$ ldd testso
        linux-vdso.so.1 (0x00007fffd79fe000)
        libetf.so.1 => /home/david/dev/src-c/lib/etf/libetf.so.1 (0x00007f4d1ef23000)
        libc.so.6 => /lib64/libc.so.6 (0x00007f4d1eb75000)
        /lib64/ld-linux-x86-64.so.2 (0x00007f4d1f126000)

Note: libetf.so.1 points to /home/david/dev/src-c/lib/etf/libetf.so.1.

Why does ld need -rpath-link when linking an executable against a so that needs another so?

Why is it, that ld MUST be able to locate liba.so when linking test? Because to me it doesn't seem like ld is doing much else than confirming liba.so's existence. For instance, running readelf --dynamic ./test only lists libb.so as needed, so I guess the dynamic linker must discover the libb.so -> liba.so dependency on its own, and make it's own search for liba.so.

Well if I understand linking process correctly, ld actually does not need to locate even libb.so. It could just ignore all unresolved references in test hoping that dynamic linker would resolve them when loading libb.so at runtime. But if ld were doing in this way, many "undefined reference" errors would not be detected at link time, instead they would be found when trying to load test in runtime. So ld just does additional checking that all symbols not found in test itself can be really found in shared libraries that test depend on. So if test program has "undefined reference" error (some variable or function not found in test itself and neither in libb.so), this becomes obvious at link time, not just at runtime. Thus such behavior is just an additional sanity check.

But ld goes even further. When you link test, ld also checks that all unresolved references in libb.so are found in the shared libraries that libb.so depends on (in our case libb.so depends on liba.so, so it requires liba.so to be located at link time). Well, actually ld has already done this checking, when it was linking libb.so. Why does it do this checking second time... Maybe developers of ld found this double checking useful to detect broken dependencies when you try to link your program against outdated library that could be loaded in the times when it was linked, but now it can't be loaded because the libraries it depends on are updated (for example, liba.so was later reworked and some of the function was removed from it).

UPD

Just did few experiments. It seems my assumption "actually ld has already done this checking, when it was linking libb.so" is wrong.

Let us suppose the liba.c has the following content:

int liba_func(int i)
{
    return i + 1;
}

and libb.c has the next:

int liba_func(int i);
int liba_nonexistent_func(int i);

int libb_func(int i)
{
    return liba_func(i + 1) + liba_nonexistent_func(i + 2);
}

and test.c

#include <stdio.h>

int libb_func(int i);

int main(int argc, char *argv[])
{
    fprintf(stdout, "%d\n", libb_func(argc));
    return 0;
}

When linking libb.so:

gcc -o libb.so -fPIC -shared libb.c liba.so

linker doesn't generate any error messages that liba_nonexistent_func cannot be resolved, instead it just silently generate broken shared library libb.so. The behavior is the same as you would make a static library (libb.a) with ar which doesn't resolve symbols of the generated library too.

But when you try to link test:

gcc -o test -Wl,-rpath-link=./ test.c libb.so

you get the error:

libb.so: undefined reference to `liba_nonexistent_func'
collect2: ld returned 1 exit status

Detecting such error would not be possible if ld didn't scan recursively all the shared libraries. So it seems that the answer to the question is the same as I told above: ld needs -rpath-link in order to make sure that the linked executable can be loaded later by dynamic loaded. Just a sanity check.

UPD2

It would make sense to check for unresolved references as early as possible (when linking libb.so), but ld for some reasons doesn't do this. It's probably for allowing to make cyclic dependencies for shared libraries.

liba.c can have the following implementation:

int libb_func(int i);

int liba_func(int i)
{
    int (*func_ptr)(int) = libb_func;
    return i + (int)func_ptr;
}

So liba.so uses libb.so and libb.so uses liba.so (better never do such a thing). This successfully compiles and works:

$ gcc -o liba.so -fPIC -shared liba.c
$ gcc -o libb.so -fPIC -shared libb.c liba.so
$ gcc -o test test.c -Wl,-rpath=./ libb.so
$ ./test
-1217026998

Though readelf says that liba.so doesn't need libb.so:

$ readelf -d liba.so | grep NEEDED
 0x00000001 (NEEDED)                     Shared library: [libc.so.6]
$ readelf -d libb.so | grep NEEDED
 0x00000001 (NEEDED)                     Shared library: [liba.so]
 0x00000001 (NEEDED)                     Shared library: [libc.so.6]

If ld checked for unresolved symbols during the linking of a shared library, the linking of liba.so would not be possible.

Note that I used -rpath key instead of -rpath-link. The difference is that -rpath-link is used at linking time only for checking that all symbols in the final executable can be resolved, whereas -rpath actually embeds the path you specify as parameter into the ELF:

$ readelf -d test | grep RPATH
 0x0000000f (RPATH)                      Library rpath: [./]

So it's now possible to run test if the shared libraries (liba.so and libb.so) are located at your current working directory (./). If you just used -rpath-link there would be no such entry in test ELF, and you would have to add the path to the shared libraries to the /etc/ld.so.conf file or to the LD_LIBRARY_PATH environment variable.

UPD3

It is actually possible to check for unresolved symbols during linking shared library, --no-undefined option must be used for doing that:

$ gcc -Wl,--no-undefined -o libb.so -fPIC -shared libb.c liba.so
/tmp/cc1D6uiS.o: In function `libb_func':
libb.c:(.text+0x2d): undefined reference to `liba_nonexistent_func'
collect2: ld returned 1 exit status

Also I found a good article that clarifies many aspects of linking shared libraries that depend on other shared libraries:
Better understanding Linux secondary dependencies solving with examples.

Recursive RPATH in macOS

Yes, you are right.

If you use otool -l <dylib>, you will find (in my case):

Load command 22
          cmd LC_RPATH
      cmdsize 32
         path @loader_path/../lib (offset 12)

@loader_path will be resolved to path to folder which contains <dylib>, so each dylib will have itself @loader_path.

As above, LC_RPATH means @rpath_path will have different path in different dylib, it won't be inherited from another.

So, when you load fw, @loader_path of fw is its dirname, load a.dylib, @loader_path of a.dylib is its dirname.

Building a simple (hello-world-esque) example of using ld's option -rpath with $ORIGIN

(I'd rather have [foo.sh] than [lib/foo.sh] but I'll fix that later).

There's most of your problem: the / in the name stops the dynamic linker from doing the rpath magic.

(Your rpath is wrong too. Think about it: from the shell, if you were currently in the directory where your executable is, how would you get to the directory where your library is? Here, you'd need to cd ../lib. So your rpath should be $ORIGIN/../lib.)

If you built your object as libfoo.so and linked with -Llib -lfoo, the linker would work out what you were intending, and do the right thing. But if you're going to use unusual naming conventions, you'll have to help it out:

1. Change the link line for the library to explicitly set the SONAME for your library to just foo.sh:

   g++ -shared -Wl,-soname,foo.sh -o lib/foo.sh obj/foo.o

2. Fix the rpath:

   g++ -o run/main.run obj/main.o -Wl,-rpath,'$ORIGIN/../lib' -Llib -l:foo.sh

It's useful to run ldd main/main.run to see what's going on. In your original failing case, you'll see something like:

    lib/foo.sh (0xNNNNNNNN)

(the lack of any => /some/resolved/path showing that it's not done any path resolution). In the fixed case, you'll see something like:

    foo.sh => /your/path/to/run/../lib/foo.sh (0xNNNNNNNN)

How to force .so dependency to be in same directory as library

The .dynamic section of an ELF file (.so libraries on Linux use ELF format) contains information to help the library find its dependencies. .dynamic entries with type DT_NEEDED contain the names of other .so files for the dynamic linker to find, but they do not contain any information on where to find those files. For that, as you mentioned, you can use LD_LIBRARY_PATH, but the ELF format also provides a way to specify it in the file itself.

A .dynamic entry with type DT_RUNPATH gives the dynamic linker a path to a directory where the dynamic linker should look for DT_NEEDED files. DT_RUNPATH allows a special variable, $ORIGIN, which refers to the file's current directory. This allows you to use relative paths, without requiring the user to invoke an executable from a specific working directory.

You use the -rpath linker flag to specify a DT_RUNPATH entry. In order to pass the literal string $ORIGIN, however, you must wrap it in single quotes to prevent your shell from interpreting it as an environment variable.

Assuming you are using gcc, you should use add this argument to the link step:

-Wl,-rpath,'$ORIGIN'

---

Related Topics