Can Cython Code Be Compiled to a Dll So C++ Application Can Call It

Minimal set of files required to distribute an embed-Cython-compiled code and make it work on any machine

After further research (I tried in an empty Win 7 x64 bit VM, without any VCredist previously installed), it seems that these files are enough:

• the program itself, test.exe (produced by cython --embed and compilation with cl.exe)

• python37.dll

• python37.zip coming from packages named "Windows x86-64 embeddable zip file" in https://www.python.org/downloads/windows/

• vcruntime140.dll, as mentioned in Can I bundle the Visual Studio 2015 C++ Redistributable DLL's with my application? or ask the user to install vc_redist.x64.exe before

• ucrtbase.dll

• more than 30 files api-ms-win-*.dll were required too; if not you will have the following error:

  ... api-ms-win-crt-runtime-l1-1-0.dll is missing ...

Notes:

• if you require another library, like pygame, just copy/paste the folder from C:\Python37\Lib\site-packages\pygame seems to work

• for me, concrt140.dll, msvcp140.dll, vccorlib140.dll did not seem necessary

Useful to test all this: Prevent a Python-embedded to look in my default path C:\Python38 for modules.

Using Cython to expose functionality to another application

I think the solution is to use both. Let me explain.

Cython makes it convenient to make a fast plugin using python but inconvenient (if at all possible) to make the right "kind" of DLL. You would probably have to use the standalone mode so that the necessary python runtime is included and then mess with the generated c code so an appropriate DLL gets compiled.

Conversely, elmer makes it convenient to make the DLL but runs "pure" python code which might not be fast enough. I assume speed is an issue because you are considering cython as opposed to simple embedding.

My suggestion is this: the pure python code that elmer executes should import a standard cython python extension and execute code from it. This way you don't have to hack anything ugly and you have the best of both worlds.

---

One more solution to consider is using shedskin, because that way you can get c++ code from your python code that is independent from the python runtime.

Use extern __declspec(dllimport) in Cython

The code functions properly once I literally hand-copy and paste over the DLLs into the generated .egg folder in my .venv\Lib\site-packages folder:

pyembree-0.1.6-py3.8-win-amd64.egg
├───EGG-INFO
│       dependency_links.txt
│       native_libs.txt
│       not-zip-safe
│       PKG-INFO
│       SOURCES.txt
│       top_level.txt
│       
└───pyembree
    │   embree.dll
    │   freeglut.dll
    │   mesh_construction.cp38-win_amd64.pyd
    │   mesh_construction.cpp
    │   mesh_construction.py
    │   rtcore.cp38-win_amd64.pyd
    │   rtcore.cpp
    │   rtcore.py
    │   rtcore_scene.cp38-win_amd64.pyd
    │   rtcore_scene.cpp
    │   rtcore_scene.py
    │   tbb.dll
    │   tbbmalloc.dll
    │   triangles.cp38-win_amd64.pyd
    │   triangles.cpp
    │   triangles.py
    │   __init__.py
    │
    └───__pycache__
            mesh_construction.cpython-38.pyc
            rtcore.cpython-38.pyc
            rtcore_scene.cpython-38.pyc
            triangles.cpython-38.pyc
            __init__.cpython-38.pyc

However, how can I tell python to copy and paste these DLLs over? Can I put something in my setup.py file?

Edit: Per @ead's comments, the setup.py can be updated to the following to automate copying the DLLs over the the right folder (thanks @ead!):

import os
from setuptools import find_packages, setup

import numpy as np
from Cython.Build import cythonize
from Cython.Distutils import build_ext

include_path = [
    np.get_include(),
]

ext_modules = cythonize("pyembree/*.pyx", language_level=3, include_path=include_path)

for ext in ext_modules:
    ext.include_dirs = include_path
    ext.libraries = [
        "pyembree/embree2/lib/embree",
        "pyembree/embree2/lib/tbb",
        "pyembree/embree2/lib/tbbmalloc",
    ]

setup(
    name="pyembree",
    version="0.1.6",
    cmdclass={"build_ext": build_ext},
    ext_modules=ext_modules,
    zip_safe=False,
    packages=find_packages(),
    include_package_data=True,
    package_data={"pyembree": ["*.cpp", "*.dll"]},
)

Making an executable in Cython

What you want is the --embed flag for the Cython compiler.
There isn't a ton of documentation on it, but this is what I was able to find. It does link to a simple working example.

To compile the Cython source code to a C file that can then be compiled to an executable you use a command like cython myfile.pyx --embed and then compile with whichever C compiler you are using.

When you compile the C source code, you will still need to include the directory with the Python headers and link to the corresponding Python shared library on your system (a file named something like libpython27.so or libpython27.a if you are using Python 2.7).

Edit: Here are some more instructions on how to get the commands for including the proper headers and linking against the proper libraries.

As I said earlier, you need to run the Cython compiler like this:

cython <cython_file> --embed

To compile using gcc, you will need to find where the python headers are on your system (you can get this location by running distutils.sysconfig.get_python_inc() (you'll have to import it first).
It is probably just the /include subdirectory in your Python installation directory.

You will also have to find the python shared library.
For Python 2.7 it would be libpython27.a on Windows or libpython2.7.so on Linux.

Your gcc command will then be

gcc <C_file_from_cython> -I<include_directory> -L<directory_containing_libpython> -l<name_of_libpython_without_lib_on_the_front> -o <output_file_name>

It may be wise to include the -fPIC flag.
On Windows 64 bit machines you will also have to include the flags -D MS_WIN64 that tells mingw to compile for 64 bit windows.

If you are compiling something that depends on NumPy, you will also need to include the directory containing the NumPy headers.
You can find this folder by running numpy.get_include() (again, after importing numpy).
Your gcc command then becomes

gcc <C_file_from_cython> -I<include_directory> -I<numpy_include_directory> -L<directory_containing_libpython> -l<name_of_libpython_without_lib_on_the_front> -o <output_file_name>

This gcc command option guide may be helpful.

Also, I would recommend you use Cython memory views if possible.
That will make it so that you won't have to include the NumPy headers and include the NumPy pxd file in your Cython file.
It also makes slicing operations easier for the C compiler to optimize.

How can I manually compile Cython code that uses C++?

The problem here is that you said that somewhere you will provide the definition of a class called Rectangle -- where the example code states

cdef extern from "Rectangle.h" namespace "shapes":
    cdef cppclass Rectangle:
        ...

However, when you compiled the library you didn't provide the code for Rectangle, or a library that contained it, so rect.so has no idea where to find this Rectangle class.

To run your code you must first create the Rectangle object file.

gcc -c Rectangle.cpp # creates a file called Rectangle.o

Now you can either create a library to dynamically link against, or statically link the object file into rect.so. I'll cover statically linking first as it's simplest.

gcc -shared -fPIC -I /usr/include/python2.7 rect.cpp Rectangle.o -o rect.so

Note that I haven't included the library for python. This is because you expect your library to be loaded by the python interpreter, thus the python libraries will already be loaded by the process when your library is loaded. In addition to providing rect.cpp as a source I also provide Rectangle.o. So lets try running a program using your module.

run.py

import rect
print(rect.PyRectangle(0, 0, 1, 2).getLength())

Unfortunately, this produces another error:

ImportError: /home/user/rectangle/rect.so undefined symbol: _ZTINSt8ios_base7failureE

This is because cython needs the c++ standard library, but python hasn't loaded it. You can fix this by adding the c++ standard library to the required libraries for rect.so

gcc -shared -fPIC -I/usr/include/python2.7 rect.cpp Rectangle.o -lstdc++ \
     -o rect.so

Run run.py again and all should work. However, the code for rect.so is larger than it needs to be, especially if you produce multiple libraries that depend on the same code. You can dynamically link the Rectangle code, by making it a library as well.

gcc -shared -fPIC Rectangle.o -o libRectangle.so
gcc -shared -fPIC -I/usr/include/python2.7 -L. rect.cpp -lRectangle -lstdc++ \
     -o rect.so

We compile the Rectangle code into a shared library in the current directory and provide -L. so gcc knows to look for libraries in the current directory and -lRectangle so gcc knows to look for the Rectangle library. Finally, to be able to run your code you must tell python where the Rectangle library lives. Before running python enter

export LD_LIBRARY_PATH="/home/user/rectangle" # where libRectangle.so lives

You can use a shell script to make sure this is done every time before you run your program, but it makes things messier. Best to just stick with statically linking Rectangle.

---

Related Topics