Why Does The -R Option (Relocatable) Make Ld Not Find Any Libraries

Why does the -r option (relocatable) make ld not find any libraries?

MacOS X

On MacOS X, the man page for ld is quite explicit about the -r option:

-r Merges object files to produce another mach-o object file with file type MH_OBJECT.

So, if you are on MacOS X, the trouble is that -lm is not a Mach-O object file, and neither is -lc. However, in theory, if you have object files main.o, obj1.o and obj2.o and you do:

cp obj1.o ./-lm
cp obj2.o ./-lc
ld -r -o main1.o main.o -lm -lc

then it might work. In practice, it doesn't, and amongst the errors you get:

ld: warning: unexpected dylib (/usr/lib/libm.dylib) on link line
ld: warning: unexpected dylib (/usr/lib/libc.dylib) on link line

However, running:

ld -r -o main1.o -arch x86_64 main.o obj1.o obj2.o

worked without any whingeing from the loader.

Linux

On Linux the man page for ld is less explicit, but says:

-i Perform an incremental link (same as option -r).

-r

--relocatable

Generate relocatable output---i.e., generate an output file that can in turn serve as input to ld. This is often called partial linking. As a side effect, in environments that support standard Unix magic numbers, this option also sets the output file’s magic number to "OMAGIC". If this option is not specified, an absolute file is produced. When linking C++ programs, this option will not resolve references to constructors; to do that, use -Ur.

When an input file does not have the same format as the output file, partial linking is only supported if that input file does not contain any relocations. Different output formats can have further restrictions; for example some "a.out"-based formats do not support partial linking with input files in other formats at all.

This option does the same thing as -i.

Reading between the lines, this also takes object files and converts them to object files; it does not add libraries into the mix. If you think about it, object files are not created containing references to libraries.

So, although there might be platforms where it is possible to specify libraries to the linker (loader) when using the -r option, there are others where it is not.

Workarounds

The original problem is to establish whether the libraries are present. Why not mimic what autoconf does, and create a main.c that would, for preference, contain a reference to a symbol defined in the library, but which could simply contain:

int main(void){return 0;}

and compile and link it with the C compiler:

cc -o main main.c -lm -lc

If it doesn't work, then one of the libraries is missing. If you've already checked that -lc is present, then you can infer that -lm is missing.

Making relocatable object with gcc causes cannot find -lgcc_s error

To solve this problem, the -nostdlib option must also be passed to gcc:

$ gcc -r -nostdlib a.o b.o -o c.o

I don't know it for sure, but it seems without this option gcc tries to link standard libraries into output relocatable object.

If an object file defines _start and doesn't use any libraries, why do I still need to link it before I can execute it?

ELF files have different types, like ELFTYPE_EXEC (traditional non-PIE executable) or ELFTYPE_REL (relocatable object file, normally with a .o filename).

as doesn't have a special-case mode that outputs an executable instead of an object file. There are other assemblers, or at least one: FASM, that do have a special mode to output an ELF executable directly.

Given the ELF object file that as produces, you could:

• link it into a simple static executable like you're doing
• link it into a PIE executable
• link it into a dynamic executable, possibly even one that links some .so shared libraries; those could have static constructors (init functions) that run before your _start. (For example glibc's libc.so does this, which is why it happens to work to call libc functions from _start on Linux without manually calling glibc init functions, if you dynamically link.)

The .o needs to be linked because no absolute address has been chosen for it to be loaded at, to fill in things like your 64-bit absolute immediate in mov $message, %rsi.

(If you'd use lea message(%rip), %rsi the code would be position-independent but the distance between the .text and .rodata sections wouldn't be known yet. Although you put your string right in .text so that would get resolved at assemble time if you hadn't chosen the least efficient way to get an address into a register, so that would give you a stand-alone block of code+data. But the most efficient way, mov $message, %esi, would also need an absolute (32-bit) address.)

as doesn't know what you want to do, and GNU Binutils was primarily written for use by compiler back-ends, so there was no point making as more complicated to be able to write an ELF-type EXEC file directly since that's what ld is for. This is the Unix philosophy of making small separate tools that do one thing well.

If you want to assemble + link with one command, make a shell script, or use a compiler front-end:

gcc -nostdlib -static -no-pie start.s -o static_executable

what is the difference between ranlib, ar, and ld for making libraries

ar

In Linux, ar is the GNU general purpose archiver.
(There are non-GNU variants of ar in other Unix-like OSes). With the option c

ar c... archive-name file...

It creates an archive containing copies of file.... The archive-name conventionally
but not necessarily has the extension .a (for archive). Each file... may be
any kind of file whatever, not necessarily an object file.

When the archived files are all object files it is usually the intention to use the
archive for delivering that selection of object files into the linkage of programs
or DSOs (Dynamic Shared Objects). In this case archive-name will also conventionally be given the prefix lib, e.g.
libfoo.a, so that it can be discovered as a candidate linker input file via the linker option -lfoo.

Used as a linker input file, libfoo.a is normally called a static library. This
usage is a perpetual source of confusion to inexpert programmers, because it leads them
to think that an archive libfoo.a is much the same kind of thing as a DSO, libfoo.so,
normally called a dynamic/shared library, and to build false expectations on this
basis. In fact a "static library" and a "dynamic library" are not at all similar things
and are used in linkage in utterly different ways.

A conspicuous difference is that a static library is not produced by the linker,
but by ar. So no linkage happens, no symbol resolution happens. The archived
object files are unchanged: they're just put in a bag.

When an archive is input in the linkage of something that is produced by the
linker - such as a program or DSO - the linker looks in the bag to see if there
are any object files in it that provide definitions for unresolved symbol references
that have accrued earlier in the linkage. If it finds any, it extracts those object files from
the bag and links them into the output file, exactly as if they were named individually
in the linker commandline and the archive not mentioned at all. So the entire
role of an archive in linkage is as bag of object files from which the linker can
select the ones it needs to carry on the linkage.

By default, GNU ar makes its output archives ready for use as linker inputs. It adds a phony "file"
to the archive, with a magic phony filename, and in this phony file it writes content that
the linker is able to read as a lookup table from the global symbols that are defined
by any object files in the archive to the names and positions of those object
files in the archive. This lookup table is what enables the linker to look in
the archive and identify any object files that define any unresolved symbol references
it has got in hand.

You can suppress the creation or updating of this lookup table with the q ( =
quick) option - which in fact you've used in your own ar example - and also
with the (capital) S ( = no symbol table) option. And if you invoke ar to create or update
an archive that hasn't got (an uptodate) symbol table for any reason, then you
can give it one with the s option.

ranlib

ranlib doesn't
create libraries at all. In Linux, ranlib is a legacy program that adds an (uptodate)
symbol table to an ar archive if it doesn't have one. It's effect is exactly the
same as ar s, with GNU ar. Historically, before ar was equipped to generate
a symbol table itself, ranlib was the kludge that injected the magic phony file
into an archive to enable the linker to pick object files out of it. In non-GNU
Unix-like OSes, ranlib might still be needed for this purpose. Your example:

ar qc libgraphics.a *.o
ranlib libgraphics.a

says:

• Create libgraphics.a by appending to an archive all *.o files in the current
  directory, with no symbol table.
• Then add a symbol table to libgraphics.a

In linux, this has the same net effect as:

ar cr libgraphics.a *.o

By itself, ar qc libgraphics.a *.o, creates an archive that the linker
can't use, because it has no symbol table.

ld

Your example:

ld -r -o libgraphics.a *.o

is actually quite unorthodox. This illustrates the fairly rare use of the linker,
ld, to produce a merged object file by linking multiple input files into
a single output object file, in which symbol resolution has been done as far as is possible,
given the input files. The -r ( = relocatable) option
directs the linker to produce an object file target (rather than a program, or DSO) by
linking the inputs as far as possible and not to fail the linkaqe if undefined symbol references
remain in the output file. This usage is called partial linking.

The output file of ld -r ... is an object file, not an ar archive, and
specifying an output filename that looks like that of an ar archive doesn't make it one.
So your example illustrates a deception. This:

ld -r -o graphics.o *.o

would be truthful. It's unclear to me what the purpose of a such a deception could be,
because even if an ELF object file is called libgraphics.a, and is input to a linkage either by that name,
or by -lgraphics, the linker will correctly identify it as an ELF object file, not an ar archive, and will consume
it the way it consumes any object file in the commandline: it links it unconditionally
into the output file, whereas the point of inputting a genuine archive is to link
archive members only on condition that they are referenced. Perhaps you just have
an example of ill-informed linking here.

Wrapping up...

We've actually only seen one way of producing something that
is conventionally called a library, and that's the production of a so-called static library,
by archiving some object files and putting a symbol table in the archive.

And we haven't seen at all how to produce the other and most important kind of thing that's conventionally called
a library, namely a Dynamic Shared Object/shared library/dynamic library.

Like a program, a DSO is produced by the linker. A program and a DSO
are variants of ELF binary that the OS loader understands and can use to assemble
a running process. Usually we invoke the linker via one one of the GCC frontends (gcc, g++, gfortran, etc):

Linking a program:

gcc -o prog file.o ... -Ldir ... -lfoo ...

Linking a DSO:

gcc -shared -o libbar.so file.o ... -Ldir ... -lfoo ...

Both shared libraries and static libraries can be offered to the linker
by the uniform -lfoo protocol, when you are linking some other program or DSO.
That option directs the linker to scan its specified or default search directrories to find either
libfoo.so or libfoo.a. By default, once it finds either one of them it will input that file to the linkage, and
if it finds both in the same search directory, it will prefer libfoo.so.
If libfoo.so, is selected then the linker adds that DSO to the runtime dependency list
of whatever program or DSO you are making. If libfoo.a is selected
then the linker uses the archive as a selection of object files for linkage
into the output file, if needed, right there and then. No runtime dependency on
libfoo.a itself is possible; it cannot be mapped into a process; it means nothing to the OS loader.

Why do I have to define LD_LIBRARY_PATH with an export every time I run my application?

Use

export LD_LIBRARY_PATH="/path/to/library/"

in your .bashrc otherwise, it'll only be available to bash and not any programs you start.

Try -R/path/to/library/ flag when you're linking, it'll make the program look in that directory and you won't need to set any environment variables.

EDIT: Looks like -R is Solaris only, and you're on Linux.

An alternate way would be to add the path to /etc/ld.so.conf and run ldconfig. Note that this is a global change that will apply to all dynamically linked binaries.

How do I get rid of LD_LIBRARY_PATH at run-time?

As suggested by Richard Pennington, the missing library is not used directly by my application, but it is used by the shared libraries I use. Since I cannot recompile IPP, the solution to my problem is to add -liomp5 when compiling, using the -R option for the linker. This actually adds the rpath for libiomp5.so fixing the problem!

---

Related Topics