Minimal Set of Files Required to Distribute an Embed-Cython-Compiled Code and Make It Work on Any MAChine

Minimal set of files required to distribute an embed-Cython-compiled code and make it work on any machine

After further research (I tried in an empty Win 7 x64 bit VM, without any VCredist previously installed), it seems that these files are enough:

the program itself, test.exe (produced by cython --embed and compilation with cl.exe)
python37.dll
python37.zip coming from packages named "Windows x86-64 embeddable zip file" in https://www.python.org/downloads/windows/
vcruntime140.dll, as mentioned in Can I bundle the Visual Studio 2015 C++ Redistributable DLL's with my application? or ask the user to install vc_redist.x64.exe before
ucrtbase.dll
more than 30 files api-ms-win-*.dll were required too; if not you will have the following error:
... api-ms-win-crt-runtime-l1-1-0.dll is missing ...

Notes:

if you require another library, like pygame, just copy/paste the folder from C:\Python37\Lib\site-packages\pygame seems to work
for me, concrt140.dll, msvcp140.dll, vccorlib140.dll did not seem necessary

Useful to test all this: Prevent a Python-embedded to look in my default path C:\Python38 for modules.

Distribute embed-cython-compiled .exe and run another machine without python

I solved adding .\sip.pyd. Then I checked resulting folder size , it was around 20 MB. With Pyinstaller, resulting .exe is about 43 MB

Prevent a Python-embedded to look in my default path C:\Python38 for modules

Thanks to @ead's answer and his link getpath.c finally redirecting to getpathp.c in the case of Windows, we can learn that the rule for building the path for module etc. is:

current directory first
PYTHONPATH env. variable
registry key HKEY_LOCAL_MACHINE\SOFTWARE\Python or the same in HKCU
PYTHONHOME env. variable
finally:
Iff - we can not locate the Python Home, have not had a PYTHONPATH
specified, and can't locate any Registry entries (ie, we have nothing
we can assume is a good path), a default path with relative entries is
used (eg. .\Lib;.\DLLs, etc)

Conclusion: in order to debug an embedded version of Python, without interfering with the default system install (C:\Python38 in my case), I finally solved it by temporarily renaming the registry key HKEY_LOCAL_MACHINE\SOFTWARE\Python to HKEY_LOCAL_MACHINE\SOFTWARE\PythonOld.

Side note: I'm not sure I will ever revert this registry key back to normal: my normal Python install shouldn't need it anyway to find its path, since when I run python.exe from anywhere (it is in the PATH for everyday use), it will automatically look in .\Lib\ and .\DLL\ which is correct. I don't see a single use case in which my normal install python.exe wouldn't find its subdir .\Lib\ or .\DLL\ and requiring the registry for this. In which use case would the registry be necessary? if python.exe is started then its path has been found, and it can take its .\Lib subfolder, without help from registry. I think 99,99% of the time this registry feature is doing more harm than good, preventing a Python install to be really "portable" (i.e. that we can move from one folder to another).

Notes:

To be 100% sure, I also did this in command line, but I don't think it's necessary:
```
set PATH=
set PYTHONPATH=
set PYTHONHOME=
```
Might be helpful to do debugging of an embedded Python: import ctypes. If you haven't _ctypes.pyd and libffi-7.dll in your embedded install folder, it should fail. If it doesn't, this means it looks somewhere else (probably in your default system-wide Python install).

Does compiled standalone Cython executable still contain all original source code?

No, it does not embed the code. It relies on being able to find the .pyx file - if you move that file then you will get a traceback but without the original code.

If you inspect the generated C source you'll find that the error handling routine goes through __Pyx_AddTraceback, then __Pyx_CreateCodeObjectForTraceback, which creates a PyCodeObject linked to your .pyx source file.

Under some circumstances (I'm not sure what though) it links to your .c file instead. The same rules will apply though - if it can't find the source it won't show that line.

Even without the .pyx file you will still get a traceback with useful method names - these are preserved in the compiled executable.

Statically link python37.dll and vcruntime140.dll when using cython --embed

Remark: There is probably a better/saner/easier option than the one presented in bellow.

The main difference between these two approaches: while in this approach all C-extension must be backed in into the resulting executable, in the alternative approach C-extension are compiled separately or an additional C-extension could also be added later on to the distribution.

While creating a statically linked embeded-Python-executable is relatively easy on Linux (see for example this SO-post) it is much more complicated on Windows. And you probably don't want to do it.

Also the result might not be what one would expect: Due to limitations of dlls compared to Linux' shared objects, the resulting static python-version will not be able to use/load any other c-extensions, as the one backed-in during the compile/link time (Note: this is not entirely true, a workaround is presented in this answer).

I also would not recommend to switch from vcruntime-dll to its static version - it would only make sense, when everything (exe, c-extensions, other dll which depend on vcruntime) is statically linked into one huge executable.

The first stumbling block: While on Linux python distributions often have a static Python-library already shipped, Windows distributions have only the dll, which cannot be statically linked in.

Thus one needs to build a static library on Windows. A good starting point is this link.

After downloading the source for the right Python version (git clone --depth=1 --branch v3.8.0 https://github.com/python/cpython.git) you can go to cpython\PCBuild and build cpython as explained in documentation (which might vary from version to version).

In my case it was

cd cpython/PCbuild
.\build.bat -e -p x64

No we have a functioning Python3.8 installation, which can be found in cpython/PCbuild/amd64. Create folder cpython/PCbuild/static_amd64 and add the following pyx-file:

#hello.pyx
print("I'm standalone")

copy python38.dll to static_amd64 for the time being.

Now let's build our program with embedded python interpreter:

cython --embed -3 hello.pyx
"C:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\vcvarsall.bat" x64
cl /c hello.c /Fohello.obj  /nologo /Ox /W3 /GL /DNDEBUG /MD -I<path_to_code>\cpython\include -I<path_to_code>\cpython\PC
link hello.obj python38.lib  /OUT:hello_prog.exe /nologo "/LIBPATH:<path_to_code>\cpython\PCbuild\amd64"

After the start, hello_prog.exe lies to us, as it is not really standalone. The good news is: it finds the Python-installation which is needed as described for example here.

Now let's create a static python38-library. For that we open pcbuild.sln in cpython/PCbuild-folder and change pythoncore-project's setting to produce static library in PCbuild\amd64_static-folder. Rebuild it.

Now we can build the embedded-python-exe:

cl /c hello.c /Fohello.obj /D "Py_NO_ENABLE_SHARED" /nologo /Ox /W3 /GL /DNDEBUG /MD -I<path_to_code>\cpython\include -I<path_to_code>\cpython\PC
link hello.obj python38.lib "version.lib" "shlwapi.lib" "ws2_32.lib" "advapi32.lib" "shell32.lib" "ole32.lib" "oleaut32.lib" "kernel32.lib" "user32.lib" "gdi32.lib" "winspool.lib" "comdlg32.lib" "uuid.lib" "odbc32.lib" "odbccp32.lib" /OUT:hello_prog.exe /nologo "/LIBPATH:<path_to_code>\cpython\PCbuild\static_amd64"

Compared to the build against dll we had to change the following:

Py_NO_ENABLE_SHARED (i.e. /D "Py_NO_ENABLE_SHARED") is added to preprocessor definitions, otherwise the linker will look for wrong symbols.
the Windows dependencies (i.e. version.lib and so on) which were brought by the python-dll need to be passed to the linker explicitly now (this can be looked up in the linker-command-line of the pythoncore-project).
lib path show to the static folder, i.e. "/LIBPATH:<path_to_code>\cpython\PCbuild\static_amd64" now.
there might be other smaller troubles (different optimization levels, link-time-code generation, disabling whole-program-optimization and so on) depending on your exact tool chain.

We can delete the python38.dll from static_amd64 now and the hello_prog.exe still works.

On Linux, this would be "mission accomplished", on Windows we are just at the beginning...

Make sure that cpython-folder has a DLLs-folder with all right pyd-files, otherwise create and copy all pyd-files from PCbuild/amd64-folder.

Let's make our pyx-file a little bit more complicated:

import _decimal
print("I'm standalone")

_decimal is a fast implementation of the decimal-module which is a C-extension and can be found in the DLL-folder.

After cythonizing and building it, running hello_prog.exe leads to the following error message:

import _decimal
ImportError: DLL load failed while importing _decimal: The specified module could not be found.

The problem is easy to find:

dumpbin /DEPENDENTS ../amd64/_decimal.pyd
...
python38.dll
...

The extensions of our installation still depends on the python-dll. Let's rebuild them against the static library - we need to change library path to from amd64 to static_amd64, to add preprocessor define Py_NO_ENABLE_SHARED and all missing windows-libraries (i.e ""version.lib"& Co.) and adding /EXPORT:PyInit__decimal to link options, otherwise, due to Py_NO_ENABLE_SHARED it becomes invisible. The result has no dependency on the python-dll! We copy it to the DLLs-folder and ...

hello_prog.exe
# crash/stopped worked

What is happening? We violated one definition rule (ODR) and ended up with two Python-interpreters: the one from the hello_prog.exe, which is initialized and the one from _decimal.pyd which is not initialized. _decimal.pyd "speaks" to its interpreter which is not initialized and bad things happens.

The difference to Linux is the difference between shared-objects and dlls: while shared-objects can use symbols from the exe (if the exe is built with right options) dll cannot and thus must either depend on a dll (which we don't want) or need to have its own version.

To avoid the violation of ODR we have only one way out: it must be linked directly into our hello_word-executable. So let's change the project for _decimal to static library and rebuild it in static_amd64-folder. Deleting the pyd from "DLLs"-folder and adding /WHOLEARCHIVE:_decimal.lib to the linker-command-line (whole archive while otherwise the linker would just discard _decimal.lib as none of its symbols is referenced somewhere), leads to an executable, which has the following error:

ModuleNotFoundError: No module named '_decimal'

This is expected - we need to tell to the interpreter, that the module _decimal is backed in and should not be searched on the python path.

The usual solution for this problem is to use PyImport_AppendInittab just before Py_Initialize, that means we need to change the c-file generated by cython (there might be workarounds, but due to multi-phase initialization it is not that easy. So probably a saner way to embed Python is the one presented here or here were main isn't written by Cython). The c-file should look as follows:

//decalare init-functions
extern  PyObject* PyInit__decimal();
...
int main(int argc, char** argv) {
...
    if (argc && argv)
        Py_SetProgramName(argv[0]);
    PyImport_AppendInittab("_decimal", PyInit__decimal); //  HERE WE GO
                                                         //  BEFORE Py_Initialize
    
    Py_Initialize();

Building everything now leads to an exe which prints

I'm standalone

and this time it is not lying to us!

Now we have to repeat the last steps for all other built-in extension we need.

The above means there are some restrictions for statically built python-interpreter: All built-in modules need to be backed into the executable and we cannot extend the interpreter latter on with libraries like numpy/scipy (but can to do it directly at the compile/link time).

Getting rid of vcruntime-dll is easier: all above steps must be done with /MT option instead of MD-option. However, there might be some problems due to usage of other dlls (e.g. _ctypes needs ffi-dll) which where built with the dll-version (and thus we once again have ODR-violated) - so I would not recommend it.

Minimal Set of Files Required to Distribute an Embed-Cython-Compiled Code and Make It Work on Any MAChine