Linking a Dynamically Linked Executable with Ld

Linking a dynamically linked executable with ld

Lose the --entry main. main isn't your entry point, _start is. Try this:

$ gcc -c hello.c
$ ld -o hello -dynamic-linker /lib/ld-linux.so.2 /usr/lib/crt1.o /usr/lib/crti.o hello.o -lc /usr/lib/crtn.o
$ ./hello
hello, world
$ 

What does ld do when linking against dynamic shared library?

While you hit the most obvious things ld needs to do when linking to ELF shared libraries, there are a few more you missed. I'll re-state the ones you mentioned and add some more:

1. Ensuring that all undefined symbols are resolved (unless the output is a shared library itself, in which case undefined symbols are valid).

2. Storing a reference to the library in a DT_NEEDED record of the _DYNAMIC object of the output file.

3. If the output is not position-independent and references objects (in the sense of data, as opposed to functions) in the shared library, generating a copy relocation to copy the original image of the object into the main program's data segment at load time, and the proper symbol table entry so that references to the object in the shared library itself get resolved to the new copy in the main program, rather than the original copy in the library.

4. Generating PLT thunks for the destination of each function call in the output that's not resolved at ld-time to a definition in the output.

These are the tasks I can think of that are specific to use of shared libraries, and of course don't include all the work that the linker already does which would be the same as for static linking. One way to think of what ld does with dynamic linking is that it takes object files with a huge repertoire of relocation types (representing anything the compiler or assembler can produce) and resolves all but a small number of them (for static linking, that number would be zero), where all of the remaining relocations fit into a much more limited set of types resolvable by the dynamic linker at load time.

How dynamic-linking know where to find the linked files?

It is well described in the man 8 ld-linux.so man page (links to proper upstream, Linux man-pages project).

In short, simplified a bit (ignoring preloaded libraries, ELF DT_RPATH/DT_RUNPATH, and various options in the binary itself that needs those dynamic libraries):

ld-linux.so looks the library up in the directories specified in LD_LIBRARY_PATH environment variable if defined.

If not defined, or not found there, ld-linux.so checks the /etc/ld.so.cache file: a binary cache, updated by ldconfig administration command (that is automatically run by your package manager whenever necessary), containing the paths to (most) known dynamic libraries.

If not found there, ld-linux.so checks if the library is found in the standard library directories.

---

Linux uses the ELF file format for binaries and dynamic libraries. This is a very structured format.

Whenever you execute a new ELF binary, at the very end, in Linux it boils down to an execve or execveat syscall (or exec_with_loader syscall on some architectures).

The Linux kernel opens the file, checking for proper permissions, and maps relevant parts of the ELF file into memory. (There is a module, binfmt_misc, for extending the types of files the kernel will execute. In addition to ELF files, the kernel recognizes #! at the very beginning of a file to indicate a script, followed by the path to the script interpreter that will be executed instead.)

If the ELF file was statically linked, the kernel lets the userspace continue execution at the ELF file start point. (Note that this is not the standard C library main(); the standard C library actually links in proper initialization and exit code.)

If the ELF file was dynamically linked, it has a DT_INTERP program header specifying the absolute path to the dynamic linker. (Note that there can be several; typically one for 64-bit binaries, and one for 32-bit binaries.) The kernel will map that into memory, and hand off execution to it instead.

The dynamic linker will stay in memory for the lifetime of the process. It provides useful features exposed by including <dlfcn.h> (see man 3 dl_iterate_phdr and man 3 dlsym in particular). For example, you can dynamically load and unload new ELF libraries at any time. This is commonly used for plugins and plugin-type functionality.

Not only does the dynamic linker find and map in memory all dynamically loaded libraries, and handle their relocation records and symbol tables, it also does some very useful things before handing execution off to the starting point of the original binary. For example, both the Linux dynamic linker and static linkers provide a way to execute functions after all dynamic libraries have been loaded, but before main() is executed (by simply marking the functions __attribute__((constructor)); and similarly for executing functions after main() returns or exit() is called (but not if the process dies due to a signal, or uses _exit()/_Exit(), by marking those functions __attribute__((destructor)).

Note that I above say "map" instead of "load". This is because the Linux kernel memory-maps the data from the storage to memory, instead of "loading" it in the traditional sense. Because of the page cache, this also means that no matter how many copies of a specific program or library you have running, only one copy actually resides in RAM (unless you do certain odd shenanigans, that is).

Finally, the Linux dynamic linker is actually a part of the C library, not the Linux kernel. For further details, go read the glibc runtime dynamic linker sources.

Name of dynamic linker on Linux

Link with gcc -nostartfiles or gcc -nostdlib instead of using ld directly.

(With -no-pie or -pie if you want to make that choice explicit).

The system gcc knows the right path for the ELF interpreter.

Or to find out what the right path is, file /bin/ls and parse the output. On my Arch Linux system, it includes ... dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2,

Other programs include readelf -p /bin/ls, and ldd /bin/ls also includes the right path.

(But note that ldd includes the right path even if you use it on an ELF executable that has the wrong path; /bin/ldd is a shell script that works by running the ELF interpreter on an executable with special args, so the shell script contains paths to try, and the runtime dynamic linker doesn't look for itself because it's already running. You can use file or readelf -a to inspect executables to check for the right path, but not ldd.)

Why does ld need -rpath-link when linking an executable against a so that needs another so?

Why is it, that ld MUST be able to locate liba.so when linking test? Because to me it doesn't seem like ld is doing much else than confirming liba.so's existence. For instance, running readelf --dynamic ./test only lists libb.so as needed, so I guess the dynamic linker must discover the libb.so -> liba.so dependency on its own, and make it's own search for liba.so.

Well if I understand linking process correctly, ld actually does not need to locate even libb.so. It could just ignore all unresolved references in test hoping that dynamic linker would resolve them when loading libb.so at runtime. But if ld were doing in this way, many "undefined reference" errors would not be detected at link time, instead they would be found when trying to load test in runtime. So ld just does additional checking that all symbols not found in test itself can be really found in shared libraries that test depend on. So if test program has "undefined reference" error (some variable or function not found in test itself and neither in libb.so), this becomes obvious at link time, not just at runtime. Thus such behavior is just an additional sanity check.

But ld goes even further. When you link test, ld also checks that all unresolved references in libb.so are found in the shared libraries that libb.so depends on (in our case libb.so depends on liba.so, so it requires liba.so to be located at link time). Well, actually ld has already done this checking, when it was linking libb.so. Why does it do this checking second time... Maybe developers of ld found this double checking useful to detect broken dependencies when you try to link your program against outdated library that could be loaded in the times when it was linked, but now it can't be loaded because the libraries it depends on are updated (for example, liba.so was later reworked and some of the function was removed from it).

UPD

Just did few experiments. It seems my assumption "actually ld has already done this checking, when it was linking libb.so" is wrong.

Let us suppose the liba.c has the following content:

int liba_func(int i)
{
    return i + 1;
}

and libb.c has the next:

int liba_func(int i);
int liba_nonexistent_func(int i);

int libb_func(int i)
{
    return liba_func(i + 1) + liba_nonexistent_func(i + 2);
}

and test.c

#include <stdio.h>

int libb_func(int i);

int main(int argc, char *argv[])
{
    fprintf(stdout, "%d\n", libb_func(argc));
    return 0;
}

When linking libb.so:

gcc -o libb.so -fPIC -shared libb.c liba.so

linker doesn't generate any error messages that liba_nonexistent_func cannot be resolved, instead it just silently generate broken shared library libb.so. The behavior is the same as you would make a static library (libb.a) with ar which doesn't resolve symbols of the generated library too.

But when you try to link test:

gcc -o test -Wl,-rpath-link=./ test.c libb.so

you get the error:

libb.so: undefined reference to `liba_nonexistent_func'
collect2: ld returned 1 exit status

Detecting such error would not be possible if ld didn't scan recursively all the shared libraries. So it seems that the answer to the question is the same as I told above: ld needs -rpath-link in order to make sure that the linked executable can be loaded later by dynamic loaded. Just a sanity check.

UPD2

It would make sense to check for unresolved references as early as possible (when linking libb.so), but ld for some reasons doesn't do this. It's probably for allowing to make cyclic dependencies for shared libraries.

liba.c can have the following implementation:

int libb_func(int i);

int liba_func(int i)
{
    int (*func_ptr)(int) = libb_func;
    return i + (int)func_ptr;
}

So liba.so uses libb.so and libb.so uses liba.so (better never do such a thing). This successfully compiles and works:

$ gcc -o liba.so -fPIC -shared liba.c
$ gcc -o libb.so -fPIC -shared libb.c liba.so
$ gcc -o test test.c -Wl,-rpath=./ libb.so
$ ./test
-1217026998

Though readelf says that liba.so doesn't need libb.so:

$ readelf -d liba.so | grep NEEDED
 0x00000001 (NEEDED)                     Shared library: [libc.so.6]
$ readelf -d libb.so | grep NEEDED
 0x00000001 (NEEDED)                     Shared library: [liba.so]
 0x00000001 (NEEDED)                     Shared library: [libc.so.6]

If ld checked for unresolved symbols during the linking of a shared library, the linking of liba.so would not be possible.

Note that I used -rpath key instead of -rpath-link. The difference is that -rpath-link is used at linking time only for checking that all symbols in the final executable can be resolved, whereas -rpath actually embeds the path you specify as parameter into the ELF:

$ readelf -d test | grep RPATH
 0x0000000f (RPATH)                      Library rpath: [./]

So it's now possible to run test if the shared libraries (liba.so and libb.so) are located at your current working directory (./). If you just used -rpath-link there would be no such entry in test ELF, and you would have to add the path to the shared libraries to the /etc/ld.so.conf file or to the LD_LIBRARY_PATH environment variable.

UPD3

It is actually possible to check for unresolved symbols during linking shared library, --no-undefined option must be used for doing that:

$ gcc -Wl,--no-undefined -o libb.so -fPIC -shared libb.c liba.so
/tmp/cc1D6uiS.o: In function `libb_func':
libb.c:(.text+0x2d): undefined reference to `liba_nonexistent_func'
collect2: ld returned 1 exit status

Also I found a good article that clarifies many aspects of linking shared libraries that depend on other shared libraries:
Better understanding Linux secondary dependencies solving with examples.

What's the difference between statically linked and not a dynamic executable from Linux ldd?

There are two separate things here:

• Requesting an ELF interpreter (ld.so) or not.
  
  Like #!/bin/sh but for binaries, runs before your _start.
  
  This is the difference between a static vs. dynamic executable.
• The list of dynamically linked libraries for ld.so to load happens to be empty.
  
  This is apparently what ldd calls "statically linked", i.e. that any libraries you might have linked at build time were static libraries.

Other tools like file and readelf give more information and use terminology that matches what you'd expect.

---

Your GCC is configured so -pie is the default, and gcc doesn't make a static-pie for the special case of no dynamic libraries.

• gcc -nostdlib just makes a PIE that happens not to link to any libraries but is otherwise identical to a normal PIE, specifying an ELF interpreter.
  
  ldd confusingly calls this "statically linked".
  file : ELF 64-bit LSB pie executable, x86-64, version 1 (SYSV), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2 ...
• gcc -nostdlib -static overrides the -pie default and makes a true static executable.
  
  file : ELF 64-bit LSB executable, x86-64, version 1 (SYSV), statically linked ...
• gcc -nostdlib -no-pie also chooses to make a static executable as an optimization for the case where there are no dynamic libraries at all. Since a non-PIE executable couldn't have been ASLRed anyway, this makes sense. Byte-for-byte identical to the -static case.

• gcc -nostdlib -static-pie makes an ASLRable executable that doesn't need an ELF interpreter. GCC doesn't do this by default for gcc -pie -nostdlib, unlike the no-pie case where it chooses to sidestep ld.so when no dynamically-linked libraries are involved.
  
  file : ELF 64-bit LSB pie executable, x86-64, version 1 (SYSV), statically linked ...

  -static-pie is obscure, rarely used, and older file doesn't identify it as statically linked.

-nostdlib doesn't imply -no-pie or -static, and -static-pie has to be explicitly specified to get that.

gcc -static-pie invokes ld -static -pie, so ld has to know what that means. Unlike with the non-PIE case where you don't have to ask for a dynamic executable explicitly, you just get one if you pass ld any .so libraries. I think that's why you happen to get a static executable from gcc -nostdlib -no-pie - GCC doesn't have to do anything special, it's just ld doing that optimization.

But ld doesn't enable -static implicitly when -pie is specified, even when there are no shared libraries to link.

---

Details

Examples generated with gcc --version gcc (Arch Linux 9.3.0-1) 9.3.0

ld --version GNU ld (GNU Binutils) 2.34 (also readelf is binutils)

ldd --version ldd (GNU libc) 2.31

file --version file-5.38 - note that static-pie detection has changed in recent patches, with Ubuntu cherry-picking an unreleased patch. (Thanks @Joseph for the detective work) - this in 2019 detected dynamic = having a PT_INTERP to handle static-pie, but it was reverted to detect based on PT_DYNAMIC so shared libraries count as dynamic. debian bug #948269. static-pie is an obscure rarely-used feature.

GCC ends up running ld -pie exit.o with a dynamic linker path specified, and no libraries. (And a boatload of other options to support possible LTO link-time optimization, but the keys here are -dynamic-linker /lib64/ld-linux-x86-64.so.2 -pie. collect2 is just a wrapper around ld.)

$ gcc -nostdlib exit.s -v      # output manually line wrapped with \ for readability
...
COLLECT_GCC_OPTIONS='-nostdlib' '-v' '-mtune=generic' '-march=x86-64'
 /usr/lib/gcc/x86_64-pc-linux-gnu/9.3.0/collect2  \
-plugin /usr/lib/gcc/x86_64-pc-linux-gnu/9.3.0/liblto_plugin.so \
-plugin-opt=/usr/lib/gcc/x86_64-pc-linux-gnu/9.3.0/lto-wrapper \
-plugin-opt=-fresolution=/tmp/ccoNx1IR.res \
--build-id --eh-frame-hdr --hash-style=gnu \
-m elf_x86_64 -dynamic-linker /lib64/ld-linux-x86-64.so.2 -pie \
-L/usr/lib/gcc/x86_64-pc-linux-gnu/9.3.0 \
-L/usr/lib/gcc/x86_64-pc-linux-gnu/9.3.0/../../../../lib -L/lib/../lib \
-L/usr/lib/../lib \
-L/usr/lib/gcc/x86_64-pc-linux-gnu/9.3.0/../../.. \
/tmp/cctm2fSS.o

You get a dynamic PIE with no dependencies on other libraries. Running it still invokes the "ELF interpreter" /lib64/ld-linux-x86-64.so.2 on it which runs before jumping to your _start. (Although the kernel has already mapped the executable's ELF segments to ASLRed virtual addresses, along with ld.so's text / data / bss).

file and readelf are more descriptive.

PIE non-static executable from gcc -nostdlib

$ gcc -nostdlib exit.s -o exit-default
$ ls -l exit-default 
-rwxr-xr-x 1 peter peter 13536 May  2 02:15 exit-default 
$ ldd exit-default 
        statically linked
$ file exit-default
exit-default: ELF 64-bit LSB pie executable, x86-64, version 1 (SYSV), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, BuildID[sha1]=05a4d1bdbc94d6f91cca1c9c26314e1aa227a3a5, not stripped

$ readelf -a exit-default
...
  Type:                              DYN (Shared object file)
  Machine:                           Advanced Micro Devices X86-64
  Version:                           0x1
  Entry point address:               0x1000
...
Program Headers:
  Type           Offset             VirtAddr           PhysAddr
                 FileSiz            MemSiz              Flags  Align
  PHDR           0x0000000000000040 0x0000000000000040 0x0000000000000040
                 0x00000000000001f8 0x00000000000001f8  R      0x8
  INTERP         0x0000000000000238 0x0000000000000238 0x0000000000000238
                 0x000000000000001c 0x000000000000001c  R      0x1
      [Requesting program interpreter: /lib64/ld-linux-x86-64.so.2]
  LOAD           0x0000000000000000 0x0000000000000000 0x0000000000000000
                 0x00000000000002b1 0x00000000000002b1  R      0x1000
  LOAD           0x0000000000001000 0x0000000000001000 0x0000000000001000
                 0x0000000000000009 0x0000000000000009  R E    0x1000
  ...   (the Read+Exec segment to be mapped at virt addr 0x1000 is where your text section was linked.)

If you strace it you can also see the differences:

$ gcc -nostdlib exit.s -o exit-default
$ strace ./exit-default
execve("./exit-default", ["./exit-default"], 0x7ffe1f526040 /* 51 vars */) = 0
brk(NULL)                               = 0x5617eb1e4000
arch_prctl(0x3001 /* ARCH_??? */, 0x7ffcea703380) = -1 EINVAL (Invalid argument)
access("/etc/ld.so.preload", R_OK)      = -1 ENOENT (No such file or directory)
mmap(NULL, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f9ff5b3e000
arch_prctl(ARCH_SET_FS, 0x7f9ff5b3ea80) = 0
mprotect(0x5617eabac000, 4096, PROT_READ) = 0
exit(0)                                 = ?
+++ exited with 0 +++

vs. -static and -static-pie the first instruction executed in user-space is your _start (which you can also check with GDB using starti).

$ strace ./exit-static-pie 
execve("./exit-static-pie", ["./exit-static-pie"], 0x7ffcdac96dd0 /* 51 vars */) = 0
exit(0)                                 = ?
+++ exited with 0 +++

---

gcc -nostdlib -static-pie

$ gcc -nostdlib -static-pie exit.s -o exit-static-pie
$ ls -l exit-static-pie
-rwxr-xr-x 1 peter peter 13440 May  2 02:18 exit-static-pie
peter@volta:/tmp$ ldd exit-static-pie
        statically linked
peter@volta:/tmp$ file exit-static-pie
exit-static-pie: ELF 64-bit LSB pie executable, x86-64, version 1 (SYSV), statically linked, BuildID[sha1]=daeb4a8f11bec1bb1aaa13cd48d24b5795af638e, not stripped

$ readelf -a exit-static-pie 
...
  Type:                              DYN (Shared object file)
  Machine:                           Advanced Micro Devices X86-64
  Version:                           0x1
  Entry point address:               0x1000
...

Program Headers:
  Type           Offset             VirtAddr           PhysAddr
                 FileSiz            MemSiz              Flags  Align
  LOAD           0x0000000000000000 0x0000000000000000 0x0000000000000000
                 0x0000000000000229 0x0000000000000229  R      0x1000
  LOAD           0x0000000000001000 0x0000000000001000 0x0000000000001000
                 0x0000000000000009 0x0000000000000009  R E    0x1000
  ... (no Interp header, but still a read+exec text segment)

Notice that the addresses are still relative to the image base, leaving ASLR up to the kernel.

Surprisingly, ldd doesn't say that it's not a dynamic executable. That might be a bug, or a side effect of some implementation detail.

---

gcc -nostdlib -static traditional non-PIE old-school static executable

$ gcc -nostdlib -static exit.s -o exit-static
$ ls -l exit-static
-rwxr-xr-x 1 peter peter 4744 May  2 02:26 exit-static
peter@volta:/tmp$ ldd exit-static
        not a dynamic executable
peter@volta:/tmp$ file exit-static
exit-static: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), statically linked, BuildID[sha1]=1b03e3d05709b7288fe3006b4696fd0c11fb1cb2, not stripped
peter@volta:/tmp$ readelf -a exit-static
ELF Header:
...
  Type:                              EXEC (Executable file)
  Machine:                           Advanced Micro Devices X86-64
  Version:                           0x1
  Entry point address:               0x401000
...   (Note the absolute entry-point address nailed down at link time)
      (And that the ELF type is EXEC, not DYN)

Program Headers:
  Type           Offset             VirtAddr           PhysAddr
                 FileSiz            MemSiz              Flags  Align
  LOAD           0x0000000000000000 0x0000000000400000 0x0000000000400000
                 0x000000000000010c 0x000000000000010c  R      0x1000
  LOAD           0x0000000000001000 0x0000000000401000 0x0000000000401000
                 0x0000000000000009 0x0000000000000009  R E    0x1000
  NOTE           0x00000000000000e8 0x00000000004000e8 0x00000000004000e8
                 0x0000000000000024 0x0000000000000024  R      0x4

 Section to Segment mapping:
  Segment Sections...
   00     .note.gnu.build-id 
   01     .text 
   02     .note.gnu.build-id 
   ...

Those are all the program headers; unlike pie / static-pie I'm not leaving any out, just other whole parts of the readelf -a output.

Also note the absolute virtual addresses in the program headers that don't give the kernel a choice where in virtual address space to map the file. This is the difference between EXEC and DYN types of ELF objects. PIE executables are shared objects with an entry point, allowing us to get ASLR for the main executable. Actual EXEC executables have a link-time-chosen memory layout.

---

ldd apparently only reports "not a dynamic executable" when both:

• no ELF interpreter (dynamic linker) path
• ELF type = EXEC

---

Related Topics