About the Binary Compatibility of Linux

Linux distribution binary compatibility

Enter Linux Standard Base to reduce the differences between individual Linux distributions.
See

http://www.linuxfoundation.org/collaborate/workgroups/lsb
http://en.wikipedia.org/wiki/Linux_Standard_Base

About the binary compatibility of Linux

Normally it can't. It will complain about libc being too old, for one.

If you statically link with libstdc++ and carefully avoid newer glibc features, you may be able to get away with it. The latter is not always possible though. Static linking with libc is not officially supported and may work or not work for you.

Determining binary compatibility under linux

GNU libraries (glibc and libstdc++) support a mechanism called symbol versioning. First of all, these libraries export special symbols used by dynamic linker to resolve the appropriate symbol version (CXXABI_* and GLIBCXX_* in libstdc++, GLIBC_* in glibc). A simple script to the tune of:

nm -D libc.so.6 | grep " A "

will return a list of version symbols which can then be further shell processed to establish the maximum supported libc interface version (same works for libstdc++). From a C code, one has the option to do the same using dlvsym() (first dlopen() the library, then check whether certain minimal version of the symbols you need can be looked up using dlvsym()).

Other options for obtaining a glibc version in runtime include gnu_get_libc_version() and confstr() library calls.

However, the proper use of versioning interface is to write code which explicitly links to a specific glibc/libstdc++ library version. For example, code linking to GLIBC_2.10 interface version is expected to work with any glibc version newer than 2.10 (all versions up to 2.18 and beyond). While it is possible to enable versioning on a per symbol basis (using a ".symver" assembler/linker directive) the more reasonable approach is to set up a chroot environment using older (minimal supported) version of the toolchain and compile the project against it (it will seamlessly run with whatever newer version encountered).

Binary compatibility between Linux distributions

I think the trick is to build on a flavour of linux with the oldest kernel and C library versions of any of the platforms you wish to support. In my job we build on Debian 4, which allows us to officially support Debian 4 and above, RedHat 3,4,5, SuSE 10 plus various other distros (SELinux etc.) in an unofficial fashion.

I suspect by building on a nice new version of linux, it becomes difficult to support people on older machines.

(edit) I should mention that we use the default compiler that comes with Debian 4, which I think is GCC 4.1.2. Installing newer compiler versions tends to make compatibility much worse.

What determines binary compatibility of shared libraries on Linux?

I can't explain why your .so library doesn't link all the used symbols statically (and instead leaves them as undefined), but I can offer practical suggestion on how to fix the issue at hand.

You can stop linking libstdc++ statically into your plugin, and instead use the one available to host system. This doesn't work for you because of ABI incompatibility between build and target platforms. You can downgrade an ABI in use for your plugin by specifying macro _GLIBCXX_USE_CXX11_ABI=0.

Easy way to guarantee binary compatibility for C++ library, C linkage?

For binary compatibility, what also matters is the application binary interface (ABI).

BTW, C++ functions can not only return some results, but also throw some exceptions. You need to document that (even if you decide that no exceptions are thrown, which is actually difficult to achieve, e.g. because new can throw std::bad_alloc and this can happen inside some internal layers of your software).

Notice that (notably on Linux) the ABI of the C++ standard library did change with various versions of the compiler. For examples, implementation of std::string-s or of standard C++ containers did vary (in a subtle incompatible way, at the binary level). You could statically link the C++ standard library (but this might not always be enough, or be brittle, e.g. if the user program also needs it because his code is in C++).

^{Since most C++ compilers and standard libraries are free software on Linux, you could dive into their source code to understand all the details. This should take you years of efforts and you need to budget that.}

So it is harder than what you believe. At the very least, document the version of the C++ compiler used to build your thing, and the version of the C++ standard library involved.

Another approach (which I recommend) could be to make your thing free software or open source and publish the source code on github or elsewhere, and let your user compile your source code (with his C++ compiler and C++ standard library).

Binary compatibility with various C++ compilers and standard C++ libraries is actually difficult to achieve, because the evil is in the details (if you release only some binary thing). You might publish several binaries compiled with various compiler versions (e.g. with g++-5, g++-6, clang++-4.0 etc...).

I don't know exact way to guarantee binary compatibility for C++ library

Such a general goal is unrealistic and over-ambitious. At most you might publish several binaries and document with what exact C++ compiler (and version, and compiler options) and C++ standard library each have been compiled.

The compatibility I'd like to achieve is compiler forward compatibility and standard library forward compatibility.

This is impossible in general. Various C++ compilers did break ABI compatibility in the past (and this has been documented). The evil is in the details (so even if it apparently seems to work most of the time, it could often be buggy).

Am I right?

No, you are wrong and over-ambitious. At most you could release a binary (probably you should release several ones) and tell exactly how it was built (what C++ compiler and version, what compilation flags, what C++ standard library and version and even what C standard library and version; if you use some external C++ or C library - like Qt, Boost, Sqlite, etc... - you also need to document their version). Binary compatibility for C++ is a fiction.

You could and probably should use (on Linux particularly) package management systems, e.g. publish some .deb package for some particular Linux distributions (e.g. a given version of Debian or Ubuntu). You'll list exact dependencies in your binary package.

Be aware that maintaining several binary versions (and binary packages) is a lot of boring work that you should budget. You might ask permission from your manager or client to open the source of your library (and quite often this takes less work). For instance, your library might be published under GPLv3 license: open source programs could freely use it, but proprietary applications would have to buy some other license from your company.

Binary compatibility over what range of machines?

There are several levels/sources of binary incompatibility.

Firstly, addressing library incompatibility

Normally, if you're running a binary on another machine, with a different version of the "same OS" (whatever that means...), then it will run fine. Completely. The issue isn't with code not working, but missing code: the bits of the OS that the binary depends on that aren't there on the target machine. It's a failure to find code to run, not to run code (which would be perfectly OK, until the point where it tried to use the missing bit!)

So, binaries from your Ubuntu gcc will most likely run on any linux system that's no older than the machine it was compiled on. It depends on exactly what functionality the binary depends on from the OS and system libraries.

Very few binaries have no external dependencies. Examine the dependencies using ldd on your output. The one most likely to cause problems is the libgcc dependency. libc and friends change very infrequently, so hardly ever cause difficulties with compatibility. GCC changes often, so will restrict the target machines your binaries will run on.

The general rule is: use as your build machine the oldest distro you want to run on. So, with a RHEL 2 build machine, you'll be able to run the binary on any system no older (with some exceptions). That's a common guideline that keeps things simple. If you need binary compatibility across lots of distros, statically linking to libstdc++ (if you use C++) is an excellent choice. Statically linking to libgcc is dangerous, and not to be attempted unless you really know what you're up to.

Finally, note that library compatibility is much simpler on other UNIX platforms; it's only linux that's such a pain. Anything compiled on AIX 6 (say) or SunOS 5.8 or HP-UX 11.00 or whatever runs with no issues on all later releases. The environment is homogenous enough that they can just ship with piles of legacy cruft in their system libraries, to guarantee that every symbol present on the old release is available on the latest one, with the same semantics.

Secondly, cross-OS binaries: run-time differences between OSes

It's all just machine code, so you might think binaries should work on other OSes. They won't. A big reason is system calls: when you need to invoke something from the kernel (which is necessary for most non-trivial functionality), you have to know how to "talk to the kernel". That is, you make a syscall and tell the OS, "do the 42 thing, you know".

The numbers for the syscalls depend on the kernel. So, the Solaris binary would be OK from linux, apart from that niggle that stops pretty much everything working.

In fact, some OSes do support the syscalls of other kernels. FreeBSD knows the linux syscalls, and when you tell it "do the 42 thing", it checks a flag in the header of your ELF, and does the appropriate action for the linux syscall numbers if the ELF is stamped as a linux one. Neat. (Of course, you need to link in linux libs as well... But, a statically linked linux binary will run fine in FreeBSD with no externals.)

Thirdly, load-time cross-OS compatibility

A binary isn't "just a binary"; it contains a whole load of meta-data that gets interpreted by the kernel before it can run the thing. Most of the binary is a blob of code and data, plus linker information. But, even something that doesn't have any external dependencies has to be parseable by the kernel.

Some kernels understand multiple binary formats, such as PE, ELF, or a.out (obsolete). The linux binaries will never run on Windows, even if they don't make any syscalls (such a binary can't exit cleanly, but for the sake of example...). That's because MS is not about to add ELF support to their kernel: the wrapper around the code that describes how it is to be loaded and started can't be read by Windows, so the code inside it won't be run: Windows doesn't know which bits of the file are even code!

Binary compatibility of struct in separately compiled code

Given a CPU architecture, is the exact binary form of a struct determined exactly?

No, the representation or layout (“binary form”) of a structure is ultimately determined by the C implementation, not by the CPU architecture. Most C implementations intended for normal purposes follow recommendations provided by the manufacturer and/or the operating system. However, there may be circumstances where, for example, a certain alignment for a particular type might give slightly better performance but is not required, and so one C implementation might choose to require that alignment while another does not, and this can result in different structure layout.

In addition, a C implementation might be designed for special purposes, such as providing compatibility with legacy code, in which case it might choose to replicate the alignment of some old compiler for another architecture rather than to use the alignment required by the target processor.

However, let’s consider structures in separate compilations using one C implementation. Then C 2018 6.2.7 1 says:

… Moreover, two structure, union, or enumerated types declared in separate translation units are compatible if their tags and members satisfy the following requirements: If one is declared with a tag, the other shall be declared with the same tag. If both are completed anywhere within their respective translation units, then the following additional requirements apply: there shall be a one-to-one correspondence between their members such that each pair of corresponding members are declared with compatible types; if one member of the pair is declared with an alignment specifier, the other is declared with an equivalent alignment specifier; and if one member of the pair is declared with a name, the other is declared with the same name. For two structures, corresponding members shall be declared in the same order. For two structures or unions, corresponding bit-fields shall have the same widths…

Therefore, if two structures are declared identically in separate translation units, or with the minor variations permitted in that passage, then they are compatible, which effectively means they have the same layout or representation.

Technically, that passage applies only to separate translation units of the same program. The C standard defines behaviors for one program; it does not explicitly define interactions between programs (or fragments of programs, such as kernel extensions) and the operating system, although to some extent you might consider the operating system and everything running in it as one program. However, for practical purposes, it applies to everything compiled with that C implementation.

This means that as long as you use the same C implementation as the kernel is compiled with, identically declared structures will have the same representation.

Another consideration is that we might use different compilers for compiling the kernel and compiling programs. The kernel might be compiled with Clang while a user prefers to use GCC. In this case, it is a matter for the compilers to document their behaviors. The C standard does not guarantee compatibility, but the compilers can, if they choose to, perhaps by both documenting that they adhere to a particular Application Binary Interface (ABI).

Also note that a “C implementation” as discussed above is not just a particular compiler but a particular compiler with particular switches. Various switches may change how a compiler behaves in ways that cause to be effectively a different C implementation, such as switches to conform to one version of the C standard or another, switches affecting whether structures are packed, switches affecting sizes of integer types, and so on.

Standard library ABI compatibility

ABIs in practice are not linked to the standard, for example consider this following code compiled with gcc 4.9.4 and gcc 5.1
using the same flags:

-std=c++11 -O2

#include <string>
int main(){
    return sizeof (std::string);
}

gcc 4.9.4 returns 8 from main, gcc 5.1 returns 32.

As for guarantees: it is complicated:

Nothing is guaranteed by the standard.

Practically MSVC used to break ABI compatability, they stopped (v140,v141,v142 use the same ABI), clang/gcc have a stable ABI for a long time.

For those interested in learning more:
For a broad discussion of ABI/C++ standard that is not directly related to this question you an look at this blog post.