Compile the Python Interpreter Statically

Compile the Python interpreter statically?

I found this (mainly concerning static compilation of Python modules):

http://bytes.com/groups/python/23235-build-static-python-executable-linux

Which describes a file used for configuration located here:

<Python_Source>/Modules/Setup

If this file isn't present, it can be created by copying:

<Python_Source>/Modules/Setup.dist

The Setup file has tons of documentation in it and the README included with the source offers lots of good compilation information as well.

I haven't tried compiling yet, but I think with these resources, I should be successful when I try. I will post my results as a comment here.

Update

To get a pure-static python executable, you must also configure as follows:

./configure LDFLAGS="-static -static-libgcc" CPPFLAGS="-static"

Once you build with these flags enabled, you will likely get lots of warnings about "renaming because library isn't present". This means that you have not configured Modules/Setup correctly and need to:

a) add a single line (near the top) like this:

*static*

(that's asterisk/star the word "static" and asterisk with no spaces)

b) uncomment all modules that you want to be available statically (such as math, array, etc...)

You may also need to add specific linker flags (as mentioned in the link I posted above). My experience so far has been that the libraries are working without modification.

It may also be helpful to run make with as follows:

make 2>&1 | grep 'renaming'

This will show all modules that are failing to compile due to being statically linked.

How to statically link the python interpreter?

I'm answering my own question since I figured out the fix. If anyone else is having the same problem, you have to export all the symbols to the dynamic symbol table at link time. To do this, you have to pass a flag -E to the linker i.e -Wl, -E. That should solve the problem.

This is a qcc specific flag so if you're experiencing this problem in gcc, you can try passing --whole-archive flag to the linker.

Is there a way to compile a python application into static binary?

There are two ways you could go about to solve your problem

Use a static builder, like freeze, or pyinstaller, or py2exe
Compile using cython

This answer explains how you can go about doing it using the second approach, since the first method is not cross platform and version, and has been explained in other answers. Also, using programs like pyinstaller typically results in huge file sizes, while using cython will result in a file that's much smaller

First, install cython.
```
sudo -H pip3 install cython
```
Then, you can use cython to generate a C file out of the Python .py file
(in reference to https://stackoverflow.com/a/22040484/5714445)
```
cython example_file.py --embed
```

Use GCC to compile it after getting your current python version (Note: The below assumes you are trying to compile it to Python3)

PYTHONLIBVER=python$(python3 -c 'import sys; print(".".join(map(str, sys.version_info[:2])))')$(python3-config --abiflags)
gcc -Os $(python3-config --includes) example_file.c -o output_bin_file $(python3-config --ldflags) -l$PYTHONLIBVER

You will now have a binary file output_bin_file, which is what you are looking for

Other things to note:

Change example_file.py to whatever file you are actually trying to compile.
Cython is used to use C-Type Variable definitions for static memory allocation to speed up Python programs. In your case however, you will still be using traditional Python definitions.
If you are using additional libraries (like opencv, for example), you might have to provide the directory to them using -L and then specify the name of the library using -l in the GCC Flags. For more information on this, please refer to GCC flags
The above method might not work for anaconda python, as you will likely have to install a version of gcc that is compatible with your conda-python.

Compile Python code to statically linked executable with Cython

The experienced problems are obviously from the linker (gcc started a linker under the hood, to see it - just start gcc with -v - in verbose mode). So let's start with a short reminder how the linkage process works:

The linker keeps the names of all symbols it needs to resolve. In the beginning it is only the symbol main. What happens, when linker inspects a library?

If it is a static library, the linker looks at every object file in this library, and if this object files defines some looked for symbols, the whole object file is included (which means some symbols becomes resolved, but some further new unresolved symbols can be added). Linker might need to pass multiple times over a static library.
If it is a shared library, it is viewed by the linker as a library consisting out of a single huge object file (after all, we have to load this library at the run time and don't have to pass multiple times over and over to prune unused symbols): If there is at least one needed symbol the whole library is "linked" (not really the linkage happens at the run-time, this is a kind of a dry-run), if not - the whole library is discarded and never looked again at.

For example if you link with:

gcc -L/path -lpython3.x <other libs> foo.o

you will get a problem, no matter whether python3.x is a shared or a static lib: when the linker sees it, it looks only for the symbol main, but this symbol is not defined in the python-lib, so it the python-lib is discarded and never looked again at. Only when the linker sees the object-file foo.o, it realizes, that the whole Python-Symbols are needed, but now it is already too late.

There is a simple rule to handle this problem: put the object files first! That means:

gcc -L/path  foo.o -lpython3.x <other libs>

Now the linker knows what it needs from the python-lib, when it first sees it.

There are other ways to achieve a similar result.

A) Let the linker to reiterate a group of archives as long as at least one new symbol definition was added per sweep:

gcc -L/path --Wl,-start-group -lpython3.x <other libs> foo.o -Wl,-end-group

Linker-options -Wl,-start-group and -Wl,-end-group says to linker iterate more than once over this group of archives, so the linker has a second chance (or more) to include symbols. This option can lead to longer linkage time.

B) Switching on the option --no-as-needed will lead to a shared library (and only shared library) being linked in, no matter whether in this library defined symbols are needed or not.

gcc -L/path -Wl,-no-as-needed -lpython3.x -Wl,-as-needed <other libs> foo.o

Actually, the default ld-behavior is --no-as-needed, but the gcc-frontend calls ld with option --as-needed, so we can restore the behavior by adding -no-as-needed prior to the python-library and then switch it off again.

Now to your problem of statical linking. I don't think it is advisable to use static versions of all standard libraries (all above glibc), what you should probably do is to link only the python-library statically.

The rules of the linkage are simple: per default the linker tries to open a shared version of the library first and than the static version. I.e. for the library libmylib and paths A and B, i.e.

 -L/A -L/B lmylib

it tries to open libraries in the following order:

A/libmylib.so
A/libmylib.a
B/libmylib.so
B/libmylib.a

Thus if the folder A has only a static version, so this static version is used (no matter whether there is a shared version in folder B).

Because it is quite opaque which library is really used - it depends on the setup of your system, usually one would switch on the logging of the linker via -Wl,-verbose to trouble-shoot.

By using the option -Bstatic one can enforce the usage of the static version of a library:

gcc  foo.o -L/path -Wl,-Bstatic -lpython3.x -Wl,-Bdynamic <other libs>  -Wl,-verbose -o foo

Notable thing:

foo.o is linked before the libraries.
switch the static-mode off, directly after the python-library, so other libraries are linked dynamically.

And now:

 gcc <cflags> L/paths foo.c -Wl,-Bstatic -lpython3.X -Wl,-Bdynamic <other libs> -o foo -Wl,-verbose
...
attempt to open path/libpython3.6m.a succeeded
...
ldd foo shows no dependency on python-lib
./foo
It works!

And yes, if you link against static glibc (I don't recommend), you will need to delete -Xlinker -export-dynamic from the command line.

The executable compiled without -Xlinker -export-dynamic will not be able to load some of c-extension which depend on this property of the executable to which they are loaded with ldopen.

Possible issues due to implicit -pie option.

Recent versions of gcc build with pie-option per default. Often/sometimes, older python versions where build with an older gcc-version, thus python-config --cflags would miss the now necessary -no-pie, as it was not needed back then. In this case the linker will produce an error message like:

relocation R_X86_64_32S against symbol `XXXXX' can not be used when
making a PIE object; recompile with -fPIC

In this case, -no-pie option should be added to <cflags>.

Compile the Python Interpreter Statically