What Is Ld_Library_Path and How to Use It

What is LD_LIBRARY_PATH and how to use it?

Typically you must set java.library.path on the JVM's command line:

java -Djava.library.path=/path/to/my/dll -cp /my/classpath/goes/here MainClass

Difference between using java.library.path and LD_LIBRARY_PATH

The first form

-Djava.library.path=/path

will be handled in java bytecode level, System.loadLibrary will call Runtime.loadLibary, then will call java/lang/ClassLoader.loadLibrary. In the function call ClassLoader.loadLibrary, the system property java.library.path will be checked to get the full path of the library and pass this full path to native code to call system api dlopen/dlsym, eventually make the library loaded. You can browse the source from OpenJDK repository. The following code snippet is the segment I copy from the link.

The advantage of this form is that you will get error or warning or exception in Java code, if there are some problems with your library path.

// Invoked in the java.lang.Runtime class to implement load and loadLibrary.
static void loadLibrary(Class fromClass, String name,
boolean isAbsolute) {
ClassLoader loader =
(fromClass == null) ? null : fromClass.getClassLoader();
if (sys_paths == null) {
usr_paths = initializePath("java.library.path");
sys_paths = initializePath("sun.boot.library.path");
}
if (isAbsolute) {
if (loadLibrary0(fromClass, new File(name))) {
return;
}
throw new UnsatisfiedLinkError("Can't load library: " + name);
}
// ....

The second form

export LD_LIBRARY_PATH=/path

will be handled in native, according to the document of dlopen/dlsym

 dlopen()
The function dlopen() loads the dynamic library file named by the null-terminated string filename and returns an opaque "handle" for the
dynamic library. If filename is NULL, then the returned handle is for the main program. If filename contains a slash ("/"), then it is
interpreted as a (relative or absolute) pathname. Otherwise, the dynamic linker searches for the library as follows (see ld.so(8) for fur‐
ther details):

o (ELF only) If the executable file for the calling program contains a DT_RPATH tag, and does not contain a DT_RUNPATH tag, then the
directories listed in the DT_RPATH tag are searched.

o If, at the time that the program was started, the environment variable LD_LIBRARY_PATH was defined to contain a colon-separated list of
directories, then these are searched. (As a security measure this variable is ignored for set-user-ID and set-group-ID programs.)

In this manner, if there are some problems with your library path and the system can't load your library, the system won't give too much clue what happen and will fail silently (I guess). It depends whether or not to implement LD_LIBRARY_PATH, Android didn't use LD_LIBRARY_PATH to determine the library location, you can see Android's implementation from here.

What LD stand for on LD_LIBRARY_PATH variable on *unix?

Linker. The *nix linker is called ld. When a program with dynamic libraries is linked, the linker adds additional code to look for dynamic libraries to resolve symbols not statically linked. Usually this code looks in /lib and /usr/lib. LD_LIBRARY_PATH is a colon separated list of other directories to search.

"ldd" is a handy program to see where the libraries are: try "ldd /bin/ls", for example.

It could also mean "Loader", though. ;-)

Editorial:

As a (semi) interesting side-note: I think dynamic libraries will go away someday. They were needed when disk space and system memory was scarce. There is a performance hit to using them (i.e. the symbols need to be resolved and the object code edited). In these modern days of 3GB memory and 7 second bootup times, it might be appropriate to go back to static linking.

Except for the fact that every C++ program would magically grow to 3MB. ;-)

LD_LIBRARY_PATH vs LIBRARY_PATH

LIBRARY_PATH is used by gcc before compilation to search directories containing static and shared libraries that need to be linked to your program.

LD_LIBRARY_PATH is used by your program to search directories containing shared libraries after it has been successfully compiled and linked.

EDIT:
As pointed below, your libraries can be static or shared. If it is static then the code is copied over into your program and you don't need to search for the library after your program is compiled and linked. If your library is shared then it needs to be dynamically linked to your program and that's when LD_LIBRARY_PATH comes into play.

Why do I have to define LD_LIBRARY_PATH with an export every time I run my application?

Use

export LD_LIBRARY_PATH="/path/to/library/"

in your .bashrc otherwise, it'll only be available to bash and not any programs you start.

Try -R/path/to/library/ flag when you're linking, it'll make the program look in that directory and you won't need to set any environment variables.

EDIT: Looks like -R is Solaris only, and you're on Linux.

An alternate way would be to add the path to /etc/ld.so.conf and run ldconfig. Note that this is a global change that will apply to all dynamically linked binaries.

simple quest about LD_LIBRARY_PATH and new entries

Read about how ${LD_LIBRARY_PATH} is used like this:

$ man ld.so

You already know that ${PATH} is a list of directories that are polled when just an application name is given:

$ foo

and ${LD_LIBRARY_PATH} works exactly the same for the ld.so(1) dynamic library linker.

The tricky bit is that ld.so(1) uses a cache of the shared libraries found the last time the ${LD_LIBRARY_PATH} was used. This speeds up the program starting, just as any cache is supposed to do. Lots of examples show just mentioning a single directory to pick up an application-specific library:

$ export LD_LIBRARY_PATH=/my/wonderful/library
$ foo

but that is depending that the existing cache already any other needed library because the ld.so(1) will know to search only that "/my/wonderful/library/" location.

A better solution is to add the new directory like this:

$ export LD_LIBRARY_PATH=/my/wonderful/library:${LD_LIBRARY_PATH}
$ foo

which will work even if the ld.so(1) cache gets deleted out from under you and ldconfig(1) has to rebuild that cache.

This idiom that exports the ${LD_LIBRARY_PATH} permanently is a bad, bad idea:

$ export LD_LIBRARY_PATH=/my/wonderful/library:${LD_LIBRARY_PATH}
$ foo

because the custom ${LD_LIBRARY_PATH} will be used for the "foo" application plus any child program in this process tree. Instead, always limit the setting to process tree here instead of anything run from this shell ever again. Keep control of where the path is set like this:

$ LD_LIBRARY_PATH=/my/wonderful/library:${LD_LIBRARY_PATH} foo

limits visibility of the ${LD_LIBRARY_PATH} to only "foo" and any process it spawns.

What is the difference between LD_LIBRARY_PATH and -L at link time?

The settings of LD_LIBRARY_PATH has the highest precedence, so when it is set, the set of directories mentioned by LD_LIBRARY_PATH are searched first even before the standard set

of directories. So in your case setting of LD_LIBRARY_PATH is influencing the lookup of

the libraries mentioned with -l option. Without LD_LIBRARY_PATH some of the dependencies

might have been resolved from the standard set of directories.

Though setting of LD_LIBRARY_PATH help with debugging and to try out a newer version of

a library its usage in the general development environment setup and deployment is considered bad.

Also refer this HOWTO from Linux Documentation for more details on Shared Libraries

Setting LD_LIBRARY_PATH from inside R

With help from Hans Lub, the way to solve the problem is by using the dyn.load() function and supplying the full path to the library:

dyn.load('path_to_library')

and then, loading via library should work.

Why do I need to set LD_LIBRARY_PATH after installing my binary?

Why do I need to set LD_LIBRARY_PATH after installing my binary?

The simple answer is, it is a failure of the system architects and the toolchain. On Linux they are the folks who maintain Binutils and glibc. Collectively the maintainers think it is OK to compile and link against one version of a library, and then runtime link against the wrong version of the library or lose the library. They have determined it is the number one use case (q.v.). Things don't work out of the box, and you have to do something special to get into a good state.

Consider, I have some Build Scripts that build and install about 70 common Linux utilities and libraries, like cURL, Git, SSH and Wget. The scripts build the program and all of its dependencies. Sometimes I attempt to use them for security testing and evaluation, like Asan and UBsan. The idea is to run the instrumented program and libraries in /usr/local and detect undefined behavior, memory errors, etc. Then, report the error to the project.

Trying to run instrumented builds on a Linux system is impossible. For example, once Bzip, iConv, Unicode, IDN, PRCE, and several other key libraries are built with instrumentation the system no longer functions. Programs in /usr/bin, like /usr/bin/bash, will load instrumented libraries in /usr/local/lib. The programs and libraries have so many faults the programs are constantly killed. You can no longer run a script because the programs in /usr/bin are using wrong libraries (and programs like bash have so many faults).

And of course there are problems like yours, where programs and libraries can't find its own libraries. The web is full of the carnage due to the maintainers' decisions.


To fix the issue for the program you are building, add the following to your LDFLAGS:

-Wl,-R,/usr/local/lib -Wl,--enable-new-dtags

-R,/usr/local/lib tells the runtime linker to look in /usr/local/lib for its libraries. --enable-new-dtags tells the linker to use RUNPATH as opposed to a RPATH.

RUNPATH's are overrideable with LD_LIBRARY_PATH, while RPATH's are not. Omitting -Wl,--enable-new-dtags is usually a bad idea. Also see Use RPATH but not RUNPATH?.

You can also use a relative runtime path:

-Wl,-R,'$$ORIGIN/../lib' -Wl,--enable-new-dtags

-R,'$$ORIGIN/../lib' tells the runtime linker to look in ../lib/ for its libraries. Presumably your binary is located in .../bin/, so ../lib/ is up-and-over to the the lib/ directory.

ORIGIN-based linker paths are good when you have a filesystem layout like:

My_App
|
+-- bin
| |
| +- my_app.exe
|
+-- lib
| |
| +- my_lib.so
|
+-- share

You can move My_App folder around the filesystem, and my_app.exe will always be able to find its library my_lib.so. And if the My_App is the root directory /, then it is a standard Linux file system used by distros (i.e., --prefix=/). And if the My_App is /usr/local, then it is a standard Linux file system used by the user (i.e., --prefix=/usr/local).

If you configure, make and make test, then you can use LD_LIBRARY_PATH to set the temporary path to the library for testing (like .../My_App/lib). In fact, a well written make test should do that for you during testing.


You should not have to add -Wl,-R,/usr/local/lib -Wl,--enable-new-dtags since the program was compiled and linked against the library in /usr/local/lib (presumably using -L<path> and -l<lib>). The reason you have to do it is because the maintainers keep making the same mistake over and over again. It is a complete engineering failure.

The Linux architects were warned about these path problems about 25 years ago when they were deciding what to do out of the box. There were two camps - one who wanted things to work out of the box, and one who wanted users to do something special to get things to work. The later won the debate. They chose the "let's link against the wrong library" and "let's lose the library" patterns. I wish I could find the Usenet posting where it was discussed so you could read it for yourself.



Related Topics



Leave a reply



Submit