Why Is My Linux Application Pulling in The Wrong .So Library

Why is my Linux application pulling in the wrong .so library?

I'm not exactly sure of all the details of how -rpath and LD_LIBRARY_PATH work, and their precedence, but I did find some useful environment variables:

• LD_DEBUG=all - This env variable turns on verbose dynamic linker debugging. Now doing an ldd on your app will spew output about the details of how all its dependencies find their dependencies.
• LD_DEBUG_OUTPUT=<filename_prefix> - Used in conjunction with LD_DEBUG to specify output files to log the debugging info to.

The LD_DEBUG env variable helped me track down that /apps1/gdal-1.8.0-jasper/lib/libgdal.so.1 was compiled with an -rpath option that was pulling the old (wrong) versions of my libraries. It gave this helpful debug output:

search path=/pathXYZ/lib/tls/x86_64:/pathXYZ/lib/tls:/pathXYZ/lib/x86_64:
  /pathABC/jasper/lib:/pathABC/hdf5/lib/tls/x86_64:/pathABC/hdf5/lib/tls:
  /pathABC/hdf5/lib/x86_64:/pathABC/hdf5/lib:/pathABC/netcdf/lib/tls/x86_64:
  /pathABC/netcdf/lib/tls:/pathABC/netcdf/lib/x86_64:/pathABC/netcdf/lib

          (RPATH from file /apps1/gdal-1.8.0-jasper/lib/libgdal.so.1)

So the rpath of how the GDAL library was compiled seemed to be making an end-run around my LD_LIBRAR_PATH. Until I can get my lab team to rebuild libgdal correctly, I found this env var, which helped me load the "right" library versions that I wanted:

• LD_PRELOAD=<path/to/libName.so> - Point this to the location of a library (or a space-separated list of libraries) that should be loaded before all others. See the ld.so man page.

Why wrong function was linked by the linker?

When the linker is linking static libraries it looks there for undefined symbols only and if it finds a definition, then it links it in and resolves the symbol. sem_init happens to come from B which is linked before -lc (the GNU C standard library which also implements POSIX sem_init) is linked, and hence the linker picks up sem_init from B and then never looks for sem_init again, this is why you don't get multiple symbol definition error.

B library probably doesn't intend to implement POSIX sem_init with a different parameter list. It probably needs to mark its sem_init as static so that it is not visible to other translation units. And/or use a completely different name for it.

Linux Program can't find Shared Library at run-time

Symlinks on libraries work fine, as long as the final target they trace to exists and is accessible.

You have built a dynamically-linked executable, that wishes to be linked against libid3-3.8.so.3 at execution time. This was likely linked during the build phase with something like -L/path/to/libid3/directory -lid3.

You have a few options to make libid3 available, in generally decreasing order of preference (since you didn't mention where the file was, I can only be general):

• Create a symlink to libid3* in a directory listed in /etc/ld.so.conf (or /lib or /usr/lib)
• Copy libid3* to a directory listed in /etc/ld.so.conf (or /lib or /usr/lib) (defaults)
• Add the directory containing libid3* to /etc/ld.so.conf
• Set LD_LIBRARY_PATH=/directory/path/to/libid3* before running your id3v2 executable.
• Recompile id3v2 statically. (It will work, but don't bother.)

After any of the first 3, rerun ldconfig so the linker cache is updated. (You can then run ldconfig -v to verify it's resolvable.)

Note those aren't steps, they're options. You only need to do 1 of them.

Glad you updated the title. #include directives have nothing to do with linking.

Linux C++ trys to load one specific library using an absolute path, while all otheres are linked using a relative one

I set rpath to . so all libraries are linked using the relative file path

Using . in rpath is a poor idea:

• Usability: the application must be run from a specific working directory.
• Security: an attacker may place modified .so files in another directory and run your application from there.

The correct way is to use -rpath=$ORIGIN feature. See man ld.so:

$ORIGIN (or equivalently ${ORIGIN})
This expands to the directory containing the program or shared object. Thus, an application located in somedir/app could be compiled with

gcc -Wl,-rpath,'$ORIGIN/../lib'

so that it finds an associated shared object in somedir/lib no matter where somedir is located in the directory hierarchy. This facilitates the creation of "turn-key" applications that do not need to be installed into special directories, but can instead be unpacked into any directory and still find their own shared objects.

$ORIGIN syntax is a bit unfortunate because it gets expanded as a variable by both make and bash, so you may need to quote it appropriately.

---

what could lead to linux trying to find the library at the original location instead of at the relative one like all the others

When linking, the library may be specified as -lmylib or -l:libmylib.so or -l<path>/libmylib.so. In the latter case the runtime linker looks for the library in that particular path <path>/libmylib.so only. See man ld, option -l for full details. You may like to review your build system linker commands.

Dependency Hell: linux .so plugin dynamic loading

This answer will be specific to my use case, but many of the commands and settings can be generalized to other applications and configurations. I also have a more detailed discussion of my debugging of this specific problem on the vrep forum.

First I changed vrep.sh to completely clear LD_LIBRARY_PATH, so it doesn't interfere with what's loading.

unset LD_LIBRARY_PATH

Thanks to @fireant for the pointer, though unfortunately that link alone didn't solve my problem.

My first hints towards solving the problem were from this stackoverflow post on rpath following commands from that post that let you see the current rpath:

objdump -x binary-or-library |grep RPATH

# Maybe an even better way to do it is the following:
readelf -d binary-or-library |head -20

# This second command above also lists the direct dependencies on other libraries followed by rpath.

# On ubuntu 15.04 someone had to use:
objdump -x binary-or-library |grep RUNPATH

The current 3.2.2 release of vrep has a very strange rpath, which I believe is because it may not be explicitly configured when vrep is built for release:

$ objdump -x vrep |grep RPATH
  RPATH                /home/marc/Qt5.2.0/5.2.0/gcc_64:/home/marc/Qt5.2.0/5.2.0/

To solve the problem in my case I created a symlink from $VREPDIR/lib to where I wanted my libraries to be found in ~/.linuxbrew/lib, then I changed the rpath of the vrep executable with the following commands:

VREPDIR=/path/to/vrep/executable
cd $VREPDIR
ln -s ~/.linuxbrew/lib $VREPDIR/lib
patchelf --set-rpath '$ORIGIN/lib' vrep

Note that the ld/rpath/patchelf systems understand $ORIGIN to be the directory where the vrep executable is located. Once I've run the above code, I have a new RUNPATH as follows (RPATH vs runpath may be due to version differences?):

$ objdump -x vrep |grep RUNPATH
  RUNPATH              $ORIGIN/lib

This solution isn't ideal and I'll probably need to fix it up because it is picking up the linuxbrew Qt installation as follows:

libQt5Core.so.5 => /usr/lib/x86_64-linux-gnu/libQt5Core.so.5 (0x00007fece8993000)

This may conflict with the API used by V-REP in the Qt shared library provided in the directory alongside vrep, especially as linuxbrew is updated. Nonetheless this solution works and vrep is loading 2 out of 3 .so libraries without crashing! One improvement will be a few minor changes like those outlined below so that the libraries provided with vrep are still loaded.

Actually, the ideal solution for me would be one where vrep loads the system libraries, and the plugin loads the .linuxbrew libraries. I'm not sure that is possible and if it is it may require modifications to the vrep source/build.

While the application works with this solution there is some potential problems and I'm not sure if it is the ideal solution. Nonetheless I think it may be useful to others who encounter a similar issue with the applications they use.

Why changing LD_LIBRARY_PATH has no effect in Ubuntu?

If you look at the output of your ls -al

These are soft links that you have. Your softlink libQt5MultimediaWidgets.so.5 points to libQt5MultimediaWidgets.so.5.9.2 in the same directory and the file is not there at all. So you need to either set the correct softlink path or have the file in same directory

Difference between shared objects (.so), static libraries (.a), and DLL's (.so)?

I've always thought that DLLs and shared objects are just different terms for the same thing - Windows calls them DLLs, while on UNIX systems they're shared objects, with the general term - dynamically linked library - covering both (even the function to open a .so on UNIX is called dlopen() after 'dynamic library').

They are indeed only linked at application startup, however your notion of verification against the header file is incorrect. The header file defines prototypes which are required in order to compile the code which uses the library, but at link time the linker looks inside the library itself to make sure the functions it needs are actually there. The linker has to find the function bodies somewhere at link time or it'll raise an error. It ALSO does that at runtime, because as you rightly point out the library itself might have changed since the program was compiled. This is why ABI stability is so important in platform libraries, as the ABI changing is what breaks existing programs compiled against older versions.

Static libraries are just bundles of object files straight out of the compiler, just like the ones that you are building yourself as part of your project's compilation, so they get pulled in and fed to the linker in exactly the same way, and unused bits are dropped in exactly the same way.

---

Related Topics