Dealing with Library Dependencies on Linux

Dealing with library dependencies on linux

I am using libcurl in my project and it depends on openssl and bunch
of other .so in runtime. This dependency is kind of pain in the ass,
since different distributives/versions may contain different openssl
versions.

For example i am experiencing problems running on Ubuntu 11.10 if i
compilled my app on Ubuntu 9.10.

First up, what is the problem with this? You shouldn't have problems if you're moving up from an older version of Ubuntu to a newer one. If I'm not mistaken, you only need to specify which minimum version of a library you need and the package manager should be able to install a suitable version. Newer versions of libraries should not break existing apps unless you're using deprecated features.

My app is really tiny and package/maintain it would be overkill. Plus,
one of the requirements is that it should be download-and-run'able.
So, (1) is not an opton for me.

For Linux (especially Ubuntu, Fedora and other top distros), packaging is really the way to distribute your application. Download-install-run is a Windows thing and it's not the way people on Linux install software (well, people new to Linux might...)

You should also try for distro acceptance which will reduce your burden over time. The first step towards this, atleast on Ubuntu, is to create your own PPA (https://help.launchpad.net/Packaging/PPA).

Static link (2) would be not-bad solution but it seems that there is
no static binary distributions of libopenssl, libcrypto and other
transitive dependencies that come with libcurl.

This is usually a very very bad thing to do. Static linking or just bundling the library with your app puts the burden of updating it on you and there are implications if you don't update those. So, I don't recommend this approach. See here for more details: http://www.dwheeler.com/blog/2012/04/03/#insecure-libraries

Here is Fedora's policy: http://fedoraproject.org/wiki/Packaging:No_Bundled_Libraries

So, here is the question - am I missing something? Is there a less
painful way to do what I want in linux world (Ubuntu to be specific)
with less pain? Any suggestions are welcomed.

There really are two things to do here:

1. Packaging: Ideally, this'll be deb for Ubuntu/Debian and rpm for Fedora/Suse. The other popular alternative is to use autotools (autoconf/automake) so that the user can build your application with the required pre-reqs. The last option is to provide just a Makefile and a README and expect your users to do the right thing.
2. Distribution: Ideally, this is with the distro repositories. Ubuntu PPA is a good starting point. Alternative is to host the binaries/packages on your own site.

Most popular applications provide both a .deb/.rpm for the popular Linux distros as well as .tar.gz with autotools for building on distros that have a different packaging system.

In the end, let me ask you this: is your focus on making it less painful for you to provide your application, or making it less painful for your users to obtain your application?

Resolving Linux Library Dependencies?

Having your package install "extra" versions of libicu to /usr/lib64 would be very surprising to most users. Don't do that.

The standard practice here is to build different .deb and .rpm files per target platform. The same as you need to build separately for 32-bit vs 64, you need to build separately for, say, CentOS 6 vs 7. You can do this by running a set of virtual machines to do the release builds--one for each target platform you want to support.

If you truly need to make a single .deb or .rpm that works on many platforms, one way is to eliminate some of your dependencies. In this case, all those libraries are part of the same project (ICU), so if you don't really need to depend on that project, don't.

Managing secondary dependencies of shared libraries

By setting DT_RUNPATH you are telling the loader that each of your binaries is linked with all of its dependencies.

But that isn't true for libC.so -- it (apparently) doesn't have DT_RUNPATH of its own.

I can link the libraries C and D statically, ... But is there a better way?

Yes: link libC.so with correct DT_RUNPATH (if libC.so and libD.so are in the same directory, then -Wl,-rpath,\$ORIGIN will work for libC.so as well).

Update:

The problem is that I may not have the source or the properly compiled object files of libC

In that cause you should use RPATH instead of RUNPATH. Unlike the latter, the former applies to the object itself and to all dependencies of that object.

In other words, using --enable-new-dtags is wrong in this case -- you want the opposite.

In this case, there is no solution (besides static linking); correct?

The other solution (besides using RPATH) is to set LD_LIBRARY_PATH in the environment.

Update 2:

Difference between RPATH and RUNPATH

The difference is explained in the ld.so man page:

       If a shared object dependency does not contain a slash, then it
       is searched for in the following order:

       o  Using the directories specified in the DT_RPATH dynamic
          section attribute of the binary if present and DT_RUNPATH
          attribute does not exist.  Use of DT_RPATH is deprecated.
...
       o  Using the directories specified in the DT_RUNPATH dynamic
          section attribute of the binary if present.  Such directories
          are searched only to find those objects required by DT_NEEDED
          (direct dependencies) entries and do not apply to those
          objects' children, which must themselves have their own
          DT_RUNPATH entries.  This is unlike DT_RPATH, which is applied
          to searches for all children in the dependency tree.

How to deal with dependencies in shared libraries, unix

One alternative is to put an interface within the shared library you made, which allows it to load dependencies at runtime. So, as an example you can have separate
init functions for different components:

init_memcached();
init_sqlite();

You implement these initialization functions using dlopen() and friends.

How to handle library dependencies with automake?

The build and install work well but I doubt I do it properly. The main program only uses functions defined in lib (not in lib_dep), so it looks strange that I have to also diretly link the main program to lib_dep. I would like your feedbacks on this please.

It is system-dependent whether indirect shared-library dependencies need to be included in the link. On Linux, they do. Compare to the situation with static library dependencies, which absolutely need to be included in the final link, on any system.

However, since you are using libtool, in a context where you can rely on its generated .la files to be available, you shouldn't need to provide explicitly for linking indirect dependencies. Libtool can handle that for you.

Is there a way to build/compile lib so that I don't have to add the LDAPP and CPPFLAGS related to lib_dep when I build y main program (src/Makefile.am) ?

That's actually two separate questions, one for LDADD and one for CPPFLAGS.

1. The CPPFLAGS are relevant only for compilation, not linking. Roughly speaking, if the main program's sources #include any headers from lib_dep/, whether directly or indirectly, then it is necessary for its CPPFLAGS to contain an -I flag for that directory. Otherwise, not.

   An indirect inclusion would be if main.c includes library1.h and library1.h includes library1_dep.h.

2. Since you are using libtool to build both libraries, and consistently declaring link dependencies by putting the appropriate libtool archives in the LIBADD / LDADD variables, you do not need to specify the indirect library dependency in the main program's LDADD. libtool can and will figure out the indirect library dependency and add it to the main program's link if needed, and at the correct location in the link command.

Thus, if the main program has neither a compile-time dependency nor a direct link dependency on libdep, then its Makefile.am could look like this:

bin_PROGRAMS = main

main_SOURCES = main.c

main_LDADD = $(top_builddir)/lib/lib_lib.la
main_CPPFLAGS = -I$(top_srcdir)/lib

NOTE: it is not wrong to add $(AM_LDADD) to main_LDADD or to add $(AM_CPPFLAGS) to main_CPPFLAGS, as in the version of this Makefile.am presented in the question, but that is useful only if you've actually defined values for those AM_* variables.

NOTE2: Your -I flags should specify directories to search for header files, not the names of specific header files.

How to force .so dependency to be in same directory as library

The .dynamic section of an ELF file (.so libraries on Linux use ELF format) contains information to help the library find its dependencies. .dynamic entries with type DT_NEEDED contain the names of other .so files for the dynamic linker to find, but they do not contain any information on where to find those files. For that, as you mentioned, you can use LD_LIBRARY_PATH, but the ELF format also provides a way to specify it in the file itself.

A .dynamic entry with type DT_RUNPATH gives the dynamic linker a path to a directory where the dynamic linker should look for DT_NEEDED files. DT_RUNPATH allows a special variable, $ORIGIN, which refers to the file's current directory. This allows you to use relative paths, without requiring the user to invoke an executable from a specific working directory.

You use the -rpath linker flag to specify a DT_RUNPATH entry. In order to pass the literal string $ORIGIN, however, you must wrap it in single quotes to prevent your shell from interpreting it as an environment variable.

Assuming you are using gcc, you should use add this argument to the link step:

-Wl,-rpath,'$ORIGIN'

How to handle dependency source version issues

Here is how you can both have and eat your cake at the same time (see quantum physics):

1. Maintain your dependencies (libraries) in global locations with their own distributed version control system (git, hg, bazaar, etc.). (See note 1)
2. Maintain a dependency version database (a text file) inside the project folder and update it on each build. (Format: MyLibrary commit-hash-or-version)

Project will always use the latest version of the dependencies. If compilation fails at some point (when you try to compile the project 3 years later, for example),

1. Go and get which version of a dependency was used at the last successful build point
2. Create a checkout of the dependency with that hash in a temporary location.
3. Point the temporary location as your dependency path in your build process.

Obviously this "return-to-successful-point" process can be easily automated.

Note 1: Dependencies are not meant to be only source codes, they might be simply some tools in binary format. In this case the dependency manager hook should invoke the tool with --version parameter (or what is appropriate for that tool), get the version, append to the dependencies.txt file. Obviously you should be prepared to be able to get the older version of the tool in case you'll need it at some point in the future.

---

Related Topics