How Does the Linux Kernel Determine Ld.So's Load Address

How does the Linux kernel determine ld.so's load address?

Read more about ELF, in particular elf(5), and about the execve(2) syscall.

An ELF file may contain an interpreter. elf(5) mentions:

PT_INTERP The array element specifies the location and
size of a null-terminated pathname to invoke
as an interpreter. This segment type is
meaningful only for executable files (though
it may occur for shared objects). However it
may not occur more than once in a file. If
it is present, it must precede any loadable
segment entry.

That interpreter is practically almost always ld-linux(8) (e.g. with GNU glibc), more precisely (on my Debian/Sid) /lib64/ld-linux-x86-64.so.2. If you compile musl-libc then build some software with it you'll get a different interpreter, /lib/ld-musl-x86_64.so.1. That ELF interpreter is the dynamic linker.

The execve(2) syscall is using that interpreter:

If the executable is a dynamically linked ELF executable, the
interpreter named in the PT_INTERP segment is used to load the needed
shared libraries. This interpreter is typically /lib/ld-linux.so.2
for binaries linked with glibc.

See also Levine's book on Linkers and loaders, and Drepper's paper: How To Write Shared Libraries

Notice that execve is also handling the shebang (i.e. first line starting with #!); see the Interpreter scripts section of execve(2). BTW, for ELF binaries, execve is doing the equivalent of mmap(2) on some segments.

Read also about vdso(7), proc(5) & ASLR. Type cat /proc/self/maps in your shell.

^{(I guess, but I am not sure, that the 0x555555554000 address is in the ELF program header of your executable, or perhaps of ld-linux.so; it might also come from the kernel, since 0x55555555 seems to appear in the kernel source code)}

Linux minimum Load Address with LD

You're running into kernel limitations on where pages can be mapped by userspace applications; these restrictions are intended to prevent certain kernel exploits from working. The minimum mappable address is set by the vm.mmap_min_addr sysctl value, and is usually at least 4096 (i.e, 0x1000).

For details, see: https://wiki.debian.org/mmap_min_addr (The situation is not unique to Debian; their documentation is just the first I found.)

The Mechanism Used to Determine Library Load Address in Perf

Shared libraries load address are stored inside the perf.data file recorded during perf record command. You can use perf script -D command to dump the data from perf.data in partially decoded format. When your program is loaded by ld-linux*.so.2 (or when required with dlopen), loader will search for library and load its segments using mmap syscall. These mmap events are recorded by kernel and have type PERF_RECORD_MMAP or PERF_RECORD_MMAP2 in perf.data file. And perf report (and perf script) will reconstruct memory offsets to decode symbol names.

$ perf record  echo 1
$ perf script -D|grep MMAP -c
7
$ perf script -D|less
PERF_RECORD_MMAP2 ... r-xp /bin/echo
...
PERF_RECORD_MMAP2 ... r-xp /lib/x86_64-linux-gnu/libc-2.27.so

Basic ideas of perf are described in https://github.com/torvalds/linux/blob/master/tools/perf/design.txt file. To start profiling there is perf_event_open syscall which has perf_event_attr *attr argument. Man page describes mmap-related fields of attr:

   The perf_event_attr structure provides detailed configuration
   information for the event being created.

                 mmap           : 1,   /* include mmap data */
                 mmap_data      : 1,   /* non-exec mmap data */
                 mmap2          :  1,  /* include mmap with inode data */

Linux kernel in its perf_events subsystem (kernel/events) will record required events for profiled processes and export the data with fd and mmap to the profiler. perf record usually dumps this data from kernel into perf.data file without heavy processing (check "Woken up 1 times to write data" prints of your perf record output). Mmap events in kernel are recorded by perf_event_mmap_output called from perf_event_mmap_event which is called from perf_event_mmap. mmap syscall implementation in mm/mmap.c has some unconditional calls to perf_event_mmap.

perf's design.txt mentions munmap, but current implementation has no munmap field or event, event code 2 was reused to PERF_RECORD_LOST. There were ideas that munmap can be helpful https://www.spinics.net/lists/netdev/msg524414.html with links to https://lkml.org/lkml/2016/12/10/1 and https://lkml.org/lkml/2017/1/27/452

perf tool is part of linux kernel sources and can be viewed online with LXR/elixir website: https://elixir.bootlin.com/linux/v5.4/source/tools/perf/
Processing code for mmap/mmap2 events is in perf/util/machine.c machine__process_mmap_event and machine__process_mmap2_event; logged mmap arguments, returned address, offset and file name are recorded with help of map__new and thread__insert_map for the process (pid/tid) and used later to convert sample event address into symbol name.

PS: Your perf.data has size of 300+ MB, this is huge and processing can be slow. For long running programs you may want to lower perf record event sampling frequency with -F freq option of perf record: perf record -F40 or with -c option.

When / How does Linux load shared libraries into address space?

Libraries are loaded by ld.so (dynamic linker or run-time linker aka rtld, ld-linux.so.2 or ld-linux.so.* in case of Linux; part of glibc). It is declared as "interpreter" (INTERP; .interp section) of all dynamic linked ELF binaries. So, when you start program, Linux will start an ld.so (load into memory and jump to its entry point), then ld.so will load your program into memory, prepare it and then run it. You can also start dynamic program with

 /lib/ld-linux.so.2 ./your_program your_prog_params

ld.so does an actual open and mmap of all needed ELF files, both ELF file of your program and ELF files of all neeeded libraries. Also, it fills GOT and PLT tables and does relocations resolving (it writes addresses of functions from libraries to call sites, in many cases with indirect calls).

The typical load address of some library you can get with ldd utility. It is actually a bash script, which sets a debug environment variable of ld.so (actually LD_TRACE_LOADED_OBJECTS=1 in case of glibc's rtld) and starts a program. You even can also do it yourself without needs of the script, e.g. with using bash easy changing of environment variables for single run:

 LD_TRACE_LOADED_OBJECTS=1 /bin/echo

The ld.so will see this variable and will resolve all needed libraries and print load addresses of them. But with this variable set, ld.so will not actually start a program (not sure about static constructors of program or libraries). If the ASLR feature is disabled, load address will be the same most times. Modern Linuxes often has ASLR enabled, so to disable it, use echo 0 | sudo tee /proc/sys/kernel/randomize_va_space.

You can find offset of system function inside the libc.so with nm utility from binutils. I think, you should use nm -D /lib/libc.so or objdump -T /lib/libc.so and grep output.

how is ld-linux.so* itself linked and loaded?

1. ld-linux.so* doesn't depends any other libraries. It is runable by itself when loaded to memory.

2. ldd is a script, it loads the object file via the loader, and the loader checks whether the object is dynamically or statically linked, try this:

LD_TRACE_LOADED_OBJECTS=1 /lib64/ld-linux-x86-64.so.2 /lib64/ld-linux-x86-64.so.2

3. file reads the magic number or elf header to figure whether the object is dynamically or statically linked, it may output different value from ldd

IMO, ld-linux.so is static linked, because it doesn't have an .interp section which all dynamically linked object must have.

what's the relationship between dlfcn.c, ld-linux.so and libdl.so?

ld-linux.so

... is what I call "the dynamic linker":

This file is loaded by the Linux kernel together with an ELF file when the ELF file requires dynamic libraries.

The file ld-linux.so contains the code that loads the dynamic libraries (for example libc.so) needed by the ELF file from the disk to memory.

libdl.so

This file is a dynamic library that contains functions like dlopen() or dlsym():

These functions allow a program to "dynamically" load dynamic libraries - this means the program can call a function to load a dynamic library.

One of many use-cases are plug-ins that the user may configure in some configuration dialog (so these plug-ins do not appear in the list of required files stored inside the executable file).

dlfcn.c

I'm not absolutely sure, but this file seems to be part of the source code of libdl.so.

Why is ld.so a shared object?

Can a dynamic linker be a simple ELF executable (ET_EXEC) ?

Yes, it can.

However, an ET_EXEC must be loaded at the address it was linked at, and that address may conflict with the address at which a.out itself is linked. If such conflict happens, the kernel will either kill the process before it even starts, or it will mmap a.out "on top of" ld.so, and the resulting binary will crash.

You can move ld.so out of the way of usual a.out link address, but someone can always link a.out at un-usual address.

If instead you link ld.so as ET_DYN, with load address of zero, then none of the above problems could happen.

Shared library load address under Linux

The shared library loader ld.so may change the virtual addresses at which a shared library is loaded depending on the needs of a binary, since the size of code, data and other sections may vary from one binary to the next. The process of rearranging the address space is called relocation.

Relocation is also the reason why you have to compile shared libraries as position-independent code with gcc -fPIC.

---

Related Topics