How to Include All Objects of an Archive in a Shared Object

How to include all objects of an archive in a shared object?

You could try (ld(2)):

   --whole-archive
       For each archive mentioned on the command line after the --whole-archive option, include every object file in the
       archive in the link, rather than searching the archive for the required object files.  This is normally used to turn
       an archive file into a shared library, forcing every object to be included in the resulting shared library.  This
       option may be used more than once.

(gcc -Wl,--whole-archive)

Plus, you should put -Wl,--no-whole-archive at the end of the library list. (as said by Dmitry Yudakov in the comment below)

Including objects to a shared library from a C++ archive (.a)

TL;DR

Add -Wl,-Bsymbolic to the gcc linkage options for your shared library.

Why?

You're testing the PICness of h264_cabac.o with:

readelf --relocs h264_cabac.o | egrep '(GOT|PLT|JU?MP_SLOT)

and concluding the the object file was compiled with -fPIC if you get any
hits. Presumably you got this test from the favourite answer
to How can I tell, with something like objdump, if an object file has been built with -fPIC?

You got some hits, and I can reproduce that more than one way:

From the source code

$ git clone https://github.com/FFmpeg/FFmpeg.git
$ cd FFmpeg
$ ./configure --enable-shared
$ make

Then:

$ cd libavcodec
$ readelf --relocs h264_cabac.o | egrep '(GOT|PLT|JU?MP_SLOT)'
00000000175d  003100000004 R_X86_64_PLT32    0000000000000000 __stack_chk_fail - 4
000000001926  003100000004 R_X86_64_PLT32    0000000000000000 __stack_chk_fail - 4
00000000259f  003100000004 R_X86_64_PLT32    0000000000000000 __stack_chk_fail - 4
000000002f0d  003100000004 R_X86_64_PLT32    0000000000000000 __stack_chk_fail - 4
000000003216  003200000004 R_X86_64_PLT32    0000000000000000 av_log - 4
000000003460  00330000002a R_X86_64_REX_GOTP 0000000000000000 ff_h264_chroma422_dc_s - 4
000000003afc  003100000004 R_X86_64_PLT32    0000000000000000 __stack_chk_fail - 4
000000003fb6  00360000002a R_X86_64_REX_GOTP 0000000000000000 ff_h264_i_mb_type_info - 4
000000004031  00370000002a R_X86_64_REX_GOTP 0000000000000000 ff_h264_mb_sizes - 4
00000000409a  003800000004 R_X86_64_PLT32    0000000000000000 ff_init_cabac_decoder - 4
000000004248  00390000002a R_X86_64_REX_GOTP 0000000000000000 ff_h264_b_mb_type_info - 4
000000004299  003a00000004 R_X86_64_PLT32    0000000000000000 ff_h264_pred_direct_mo - 4
000000004a31  003b00000004 R_X86_64_PLT32    0000000000000000 ff_h264_check_intra4x4 - 4
000000004bd5  003200000004 R_X86_64_PLT32    0000000000000000 av_log - 4
000000004f85  003c0000002a R_X86_64_REX_GOTP 0000000000000000 ff_h264_p_mb_type_info - 4
0000000050fd  003d0000002a R_X86_64_REX_GOTP 0000000000000000 ff_h264_b_sub_mb_type_ - 4
000000005233  003a00000004 R_X86_64_PLT32    0000000000000000 ff_h264_pred_direct_mo - 4
00000000544a  003200000004 R_X86_64_PLT32    0000000000000000 av_log - 4
000000005bef  003a00000004 R_X86_64_PLT32    0000000000000000 ff_h264_pred_direct_mo - 4
000000006db5  003e00000004 R_X86_64_PLT32    0000000000000000 ff_h264_check_intra_pr - 4
000000006de9  003f0000002a R_X86_64_REX_GOTP 0000000000000000 ff_h264_p_sub_mb_type_ - 4
000000007171  003200000004 R_X86_64_PLT32    0000000000000000 av_log - 4
000000008b1b  003e00000004 R_X86_64_PLT32    0000000000000000 ff_h264_check_intra_pr - 4
00000000ad41  004000000009 R_X86_64_GOTPCREL 0000000000000000 ff_h264_chroma_dc_scan - 4
00000000ad84  004000000009 R_X86_64_GOTPCREL 0000000000000000 ff_h264_chroma_dc_scan - 4
00000000b758  003100000004 R_X86_64_PLT32    0000000000000000 __stack_chk_fail - 4

From the Ubuntu 16.04 dev package

$ sudo apt-get install libavcodec-dev
$ dpkg -S libavcodec.a
libavcodec-dev:amd64: /usr/lib/x86_64-linux-gnu/libavcodec.a
$ mkdir ~/deleteme
$ cd ~/deleteme
$ ar x /usr/lib/x86_64-linux-gnu/libavcodec.a h264_cabac.o
$ readelf --relocs h264_cabac.o | egrep '(GOT|PLT|JU?MP_SLOT)'
0000000000c7  002e00000004 R_X86_64_PLT32    0000000000000000 __stack_chk_fail - 4
0000000002fa  002e00000004 R_X86_64_PLT32    0000000000000000 __stack_chk_fail - 4
00000000179d  002e00000004 R_X86_64_PLT32    0000000000000000 __stack_chk_fail - 4
000000001966  002e00000004 R_X86_64_PLT32    0000000000000000 __stack_chk_fail - 4
000000001b09  002e00000004 R_X86_64_PLT32    0000000000000000 __stack_chk_fail - 4
000000001d4a  002e00000004 R_X86_64_PLT32    0000000000000000 __stack_chk_fail - 4
000000001ee5  002e00000004 R_X86_64_PLT32    0000000000000000 __stack_chk_fail - 4
00000000265f  002e00000004 R_X86_64_PLT32    0000000000000000 __stack_chk_fail - 4
000000002fcd  002e00000004 R_X86_64_PLT32    0000000000000000 __stack_chk_fail - 4
0000000032f6  002f00000004 R_X86_64_PLT32    0000000000000000 av_log - 4
000000003305  002e00000004 R_X86_64_PLT32    0000000000000000 __stack_chk_fail - 4
000000003bdc  002e00000004 R_X86_64_PLT32    0000000000000000 __stack_chk_fail - 4
000000003cb5  002e00000004 R_X86_64_PLT32    0000000000000000 __stack_chk_fail - 4
000000004121  00320000002a R_X86_64_REX_GOTP 0000000000000000 ff_h264_mb_sizes - 4
000000004187  003300000004 R_X86_64_PLT32    0000000000000000 ff_init_cabac_decoder - 4
000000004381  003400000004 R_X86_64_PLT32    0000000000000000 ff_h264_pred_direct_mo - 4
000000004afe  003500000004 R_X86_64_PLT32    0000000000000000 ff_h264_check_intra4x4 - 4
000000005556  003400000004 R_X86_64_PLT32    0000000000000000 ff_h264_pred_direct_mo - 4
00000000576a  002f00000004 R_X86_64_PLT32    0000000000000000 av_log - 4
000000005acf  003400000004 R_X86_64_PLT32    0000000000000000 ff_h264_pred_direct_mo - 4
000000006e31  002f00000004 R_X86_64_PLT32    0000000000000000 av_log - 4
000000006e58  003600000004 R_X86_64_PLT32    0000000000000000 ff_h264_check_intra_pr - 4
000000009c20  003600000004 R_X86_64_PLT32    0000000000000000 ff_h264_check_intra_pr - 4
00000000b425  002f00000004 R_X86_64_PLT32    0000000000000000 av_log - 4
00000000b5ab  002e00000004 R_X86_64_PLT32    0000000000000000 __stack_chk_fail - 4

The results aren't identical, and I get 26 relocations the first way, 25 the second way. But there are plenty of PIC-safe relocations either way and
I'm happy to believe both compilations of h264_cabac.o had -fPIC, whatever other options they had.

I'll state the obvious: The symbol ff_h264_cabac_tables, about which your linkage
complains:

relocation R_X86_64_PC32 against symbol 'ff_h264_cabac_tables' can not be used when making a shared object

isn't in either of those lists. That means this object file - from both provenances - contains both PIC-safe and PIC-unsafe relocations.
How could GCC get that wrong, without anybody noticing till now? And if it did, how
did I just run the shared library build of FFmpeg and link libavcodec.so successfully?

Let's have a look at the PIC-unsafe relocations:

$ readelf --relocs h264_cabac.o | egrep -v '(GOT|PLT|JU?MP_SLOT)'
000000000017  002c00000002 R_X86_64_PC32     0000000000000000 ff_h264_cabac_tables - 4
...
...

Well I'll elide ~160 lines of that, but they all describe PC-relative type R_X86_64_PC32
relocations and the only symbol mentioned, discounting section names and local labels,
is our friend ff_h264_cabac_tables, about which the symbol table says:

$ readelf -s h264_cabac.o | grep ff_h264_cabac_tables
    44: 0000000000000000     0 NOTYPE  GLOBAL DEFAULT  UND ff_h264_cabac_tables
    

It's a global variable, not defined in this object file.

GCC's -fPIC isn't broken. However, knowing that an object file was compiled with
-fPIC can't absolutely guarantee that it contains no PC-relative type
R_X86_64_PC32 relocations that reference undefined global symbols. relocation R_X86_64_PC32 against symbol 'ff_h264_cabac_tables' is
such a relocation. The R_X86_64_PC32 type relocation employs a 32-bit PC-relative addressing mode,
which is efficient but has a critical limitation in the setting of a 64-bit linkage.
The linker can't guarantee the referenced symbol will not be dynamically resolved
to an address that's unrepresentable in this addressing mode. It won't have that, so
it says:

relocation R_X86_64_PC32 against symbol 'ff_h264_cabac_tables' can not be used when making a shared object

and its advice:

recompile with -fPIC

is based on the hypothesis that the culprit object file was not compiled with -fPIC.
Which is probably, but not necessarily, the right hypothesis and is not right about
your culprit libavcodec.a(h264_cabac.o)

Compiling with -fPIC will guarantee you PIC-safe relocations provided that the compiler
is allowed to do all the assembly and code-generation. But it wasn't allowed to
with your specimen of h264_cabac.o or either of my specimens. All those specimens
were compiled from FFmpeg/libavcodec/x86/h264_cabac.c in the FFmpeg source tree.
Look at that file and you'll see that it defines functions that refer to the extern
global variable ff_h264_cabac_tables and are implemented in inline, hand-crafted
assembly. GCC can be told to compile these functions -fPIC, but it doesn't get
a chance. The position-independence of these functions is the responsibility of
the author of the assembly code.

We can show that GCC is able to compile a h264_cabac.o that has only PIC-safe
relocations, if it's allowed to. That will collaterally prove that your linkage
failure stems from the hand-assembly of our specimens of the file and will also show
you one fix for the linkage failure. FFmpeg's ./configure script has the option:

--disable-asm            disable all assembly optimizations

which has the effect, among others, of causing h264_cabac.o to be compiled
from the pure C source file FFmpeg/libavcodec/h264_cabac.c instead of the the
inline assembly source FFmpeg/libavcodec/x86/h264_cabac.c. So let's try that:

$ cd FFmpeg
$ make clean
$ ./configure --enable-shared --disable-asm
$ make
$ cd libavcodec
$ readelf --relocs h264_cabac.o | grep ff_h264_cabac_tables
00000000000a  00300000002a R_X86_64_REX_GOTP 0000000000000000 ff_h264_cabac_tables - 4
0000000000ca  00300000002a R_X86_64_REX_GOTP 0000000000000000 ff_h264_cabac_tables - 4
000000001eb5  00300000002a R_X86_64_REX_GOTP 0000000000000000 ff_h264_cabac_tables - 4
0000000021c6  00300000002a R_X86_64_REX_GOTP 0000000000000000 ff_h264_cabac_tables - 4
0000000026fe  00300000002a R_X86_64_REX_GOTP 0000000000000000 ff_h264_cabac_tables - 4
000000002a17  00300000002a R_X86_64_REX_GOTP 0000000000000000 ff_h264_cabac_tables - 4
000000002f13  00300000002a R_X86_64_REX_GOTP 0000000000000000 ff_h264_cabac_tables - 4
00000000324c  00300000002a R_X86_64_REX_GOTP 0000000000000000 ff_h264_cabac_tables - 4
000000003509  00300000002a R_X86_64_REX_GOTP 0000000000000000 ff_h264_cabac_tables - 4
00000000362a  00300000002a R_X86_64_REX_GOTP 0000000000000000 ff_h264_cabac_tables - 4
0000000037d7  00300000002a R_X86_64_REX_GOTP 0000000000000000 ff_h264_cabac_tables - 4
00000000592b  00300000002a R_X86_64_REX_GOTP 0000000000000000 ff_h264_cabac_tables - 4

Now, all the relocations referencing ff_h264_cabac_tables are PIC-safe.
We might as well prove that this h264_cabac.o can be linked in a shared library. We know
that ff_h264_cabac_tables is undefined in h264_cabac.o, so we'll also need to link the
object file in which it is defined. It happens to be ./cabac.o.

$ gcc -shared -o libfoo.so h264_cabac.o cabac.o

Voila:

$ file libfoo.so
libfoo.so: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, BuildID[sha1]=ed63107b715b357853da94d4a031c0b06c30c5f2, not stripped

You might still feel a little aggrieved, however, if you have to link your own
shared library with this unoptimized h264_cabac.o and a little disappointed by
the assembly coding flaw that obliges you to. These feelings would be premature.

Remember, I've already built FFmpeg successfully with plain vanilla ./configure --enable-shared.
I said that linker's objection in your failing linkage is that the R_X86_64_PC32
relocation referencing ff_h264_cabac_tables might not be feasible at runtime, if ff_h264_cabac_tables
is dynamically resolved. It is not an objection to the type R_X86_64_PC32
relocation as such. It is a precautionary objection, based on ignorance of how ff_h264_cabac_tables
will ultimately be resolved.

But we know that ff_h264_cabac_tables is in fact defined in cabac.o and that
we'll include it the same linkage with h264_cabac.o, just as they're both included
in the linkage of libavcodec.so. And we can tell the linker that any global
references in the linkage are to be statically resolved to definitions in the
shared library being linked, if at all, by passing it the argument:

-Bsymbolic

This will remove its precautionary objection to any R_X86_64_PC32 relocation.
It knows it will be able to decide at linktime whether the R_X86_64_PC32
relocation against ff_h264_cabac_tables is feasible. If not, it will give
a different error: relocation truncated to fit:... Otherwise it will succeed
without comment.

Inevitably, that's how libavcodec.so is successfully linked in the stock FFmpeg build.
Once more from the top:

$ cd FFmpeg
$ make clean
$ ./configure --enable-shared
$ make

Then force a relink of libavcodec.so:

$ rm libavcodec/h264_cabac.o
$ $ make libavcodec/libavcodec.so V=1
gcc -I. -I./ -D_ISOC99_SOURCE -D_FILE_OFFSET_BITS=64 -D_LARGEFILE_SOURCE \
-D_POSIX_C_SOURCE=200112 -D_XOPEN_SOURCE=600 -DPIC -DZLIB_CONST -DHAVE_AV_CONFIG_H \
-std=c11 -fomit-frame-pointer -fPIC -pthread  -g -Wdeclaration-after-statement \
-Wall -Wdisabled-optimization -Wpointer-arith -Wredundant-decls -Wwrite-strings \
-Wtype-limits -Wundef -Wmissing-prototypes -Wno-pointer-to-int-cast -Wstrict-prototypes \
-Wempty-body -Wno-parentheses -Wno-switch -Wno-format-zero-length -Wno-pointer-sign \
-O3 -fno-math-errno -fno-signed-zeros -fno-tree-vectorize -Werror=format-security \
-Werror=implicit-function-declaration -Werror=missing-prototypes -Werror=return-type \
-Werror=vla -Wformat -fdiagnostics-color=auto -Wno-maybe-uninitialized  \
-MMD -MF libavcodec/h264_cabac.d -MT libavcodec/h264_cabac.o -c \
-o libavcodec/h264_cabac.o libavcodec/h264_cabac.c
sed 's/MAJOR/57/' libavcodec/libavcodec.v | cat > libavcodec/libavcodec.ver
gcc -shared -Wl,-soname,libavcodec.so.57 -Wl,-Bsymbolic ... etc. etc. ...
                                         ^^^^^^^^^^^^^^   

So there is no assembly coding flaw. To link the hand-optimized h264_cabac.o
in a shared library you just need to add -Wl,-Bsymbolic to the gcc linkage options. It's
a requirement of the optimization.

Let's prove that minimally:

$ cd libavcodec/
$ gcc -shared -o libfoo.so h264_cabac.o cabac.o
/usr/bin/ld: h264_cabac.o: relocation R_X86_64_PC32 against symbol `ff_h264_cabac_tables' can not be used when making a shared object; recompile with -fPIC
/usr/bin/ld: final link failed: Bad value
collect2: error: ld returned 1 exit status

There's your failure again. And:

$ gcc -shared -Wl,-Bsymbolic -o libfoo.so h264_cabac.o cabac.o
$ file libfoo.so 
libfoo.so: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, BuildID[sha1]=7dc86aeae353c4d92cdb5fa35d169bf019b47eb2, not stripped

success.

Linking archives (.a) into shared object (.so)

symbols/object files in .a files that's not used, will be discarded by the linker.

Use -Wl,--whole-archive for the linking to include the entire .a file
Edit, you'll need to add -Wl,--no-whole-archive after you specify your library as well, so the whole thing will be -Wl,--whole-archive archive1.a archive2.a -Wl,--no-whole-archive

Shared library from archive (.a) file in C

If you wish to build a shared library from a static library you have to tell the linker to use all function/symbols included in static library (.a). Otherwise nothing will be included in the shared library (.so).

You have to use --whole-archive/--no-whole-archive pair in linking.

CFLAGS = -Wall -Werror
LDFLAGS = -L/home/betatest/Public/implicit-rule-archive -Wl,-rpath,'$$ORIGIN'

all : run

run : main.o libfoo.so
    $(CC) $(LDFLAGS) -o $@ $^

libfoo.so : CFLAGS += -fPIC # Build objects for .so with -fPIC.
libfoo.so : libfoo.a
    $(CC) -shared -o $@ -Wl,--whole-archive $^  -Wl,--no-whole-archive

libfoo.a : foo.o bar.o
     ar cvq libfoo.a foo.o bar.o

# Compile any .o from .c. Also make dependencies automatically.
%.o : %.c
    $(CC) -c $(CFLAGS) -o $@ $<

#Include dependencies on subsequent builds.

.PHONY : all clean

clean :
    -rm -f *.o *.d run libfoo.*

You can check the exported functions using nm command:

$ nm -D libfoo.so | grep  ' T '
0000000000000702 T bar
0000000000000714 T _fini
00000000000006f0 T foo
0000000000000590 T _init

When no --whole-archive/--no-whole-archive pair is used you get:

$ nm -D libfoo.so | grep  ' T '
0000000000000714 T _fini
0000000000000590 T _init

object file from .a not included in .so

Creating a dummy reference for the required symbols in my main file did not solve the problem. The referenced symbols appeared in the binary dump (obtained using nm) with a U (= undefined) marker. I managed to solve the problem by linking the object file directly when creating the .so file instead of including it in the .a library first. As these functions were marked extern they were included in the .so even though they were not being referenced within the library. Had they not been marked extern, they would not have been included just like sylvainulg said.

Thanks to Dmitry for pointing out the --whole-archive option. I did not know that such an option exists.

Include static lib in dynamic lib

From man ld:

--whole-archive

For each archive mentioned on the command line after the --whole-archive option, include every object file in the archive in the
link, rather than searching the archive for the required object files. This is normally used to turn an archive file into a shared
library, forcing every object to be included in the resulting shared library. This option may be used more than once.

Two notes when using this option from gcc: First, gcc doesn't know about this option, so you have to use -Wl,-whole-archive.
Second, don't forget to use -Wl,-no-whole-archive after your list of archives, because gcc will add its own list of archives to your
link and you may not want this flag to affect those as well.

Example:

g++ -shared -o yourlib a.o. b.o. c.o -Wl,-whole-archive libstatic.a -Wl,-no-whole-archive

Note also that in your example you first put static library, then the object files - in this case the symbols used in the object files and defined in static library will not be found unless you use --whole-archive linker option. If you want to include just the needed symbols from the static library, you need to put it after the object files.

g++ -o your_app a.o b.o c.o -lyour_static_lib

Include static lib in shared object?

This really is a 2 step process:

ar -x mylib.a
gcc -shared *.o -o mylib.so

---

Related Topics