Collapse Multiple Submodules to One Cython Extension

Collapse multiple submodules to one Cython extension

First off, I should note that it's impossible to compile a single .so file with sub packages using Cython. So if you want sub packages, you're going to have to generate multiple .so files, as each .so can only represent a single module.

Second, it doesn't appear that you can compile multiple Cython/Python files (I'm using the Cython language specifically) and link them into a single module at all.

I've tried to compile multiply Cython files into a single .so every which way, both with distutils and with manual compilation, and it always fails to import at runtime.

It seems that it's fine to link a compiled Cython file with other libraries, or even other C files, but something goes wrong when linking together two compiled Cython files, and the result isn't a proper Python extension.

The only solution I can see is to compile everything as a single Cython file. In my case, I've edited my setup.py to generate a single .pyx file which in turn includes every .pyx file in my source directory:

includesContents = ""
for f in os.listdir("src-dir"):
    if f.endswith(".pyx"):
        includesContents += "include \"" + f + "\"\n"

includesFile = open("src/extension-name.pyx", "w")
includesFile.write(includesContents)
includesFile.close()

Then I just compile extension-name.pyx. Of course this breaks incremental and parallel compilation, and you could end up with extra naming conflicts since everything gets pasted into the same file. On the bright side, you don't have to write any .pyd files.

I certainly wouldn't call this a preferable build method, but if everything absolutely has to be in one extension module, this is the only way I can see to do it.

How to build multiple Cython files into one python extension

Turn out it is pretty easy. I even don't need to change the setup.py
I just moved the children classes' code to their own files and include them from the parent class.

It would became something like this

• Child1.pyx
• Child2.pyx
• Child3.pyx
• MyLibrary.pyx

and in the MyLibrary.pyx

.
.
.
include Child1.pyx
include Child2.pyx
include Child3.pyx

class parent:
.
.
.

Calling a cython library with multiple pyx files through c++

There's a number of similar looking questions about bundling multiple Cython modules together (e.g. 1, 2) which isn't really viable because Python uses file paths to handle modules. However, this question isn't quite the same because you're calling it from C++, which gives you an extra option.

You need to use the C API function PyImport_AppendInittab to Python to treat impl_file1 as a builtin module so it doesn't search the path for a file to import. Start by providing a declaration of the import function (since you won't get that from your header file):

extern "C" {
// PyObject* PyInit_impl_file1(); // Python 3
void initimpl_file1(); // Python 2
}

Then, in main, before Py_Initialize, add:

PyImport_AppendInittab("impl_file1", initimpl_file1); // change the name for Python 3

project structure for wrapping many c++ classes in cython to a single shared object

While waiting for a definitive answer, I kept playing around with organizing my code. The including of pyx files into a single pyx for compilation has been working so far.

My setup.py is simple like:

ext_modules = [
    Extension(
        "openni", 
        ["src/openni.pyx"], 
        language="c++",
        include_dirs=['src/', '/usr/include/ni'],
        libraries=['OpenNI'],
    )
],

The main openni.pyx looks like:

include "constants.pyx"
include "exceptions.pyx"
include "context.pyx"
...

I have a common libopenni.pxd to provide declaration-only externs to the rest of the modules.

I name my cppclass declarations a different pxd name than the pyx class definitions to avoid name collision:

xncontext.pxd

cdef extern from "XnCppWrapper.h" namespace "xn":
    cdef cppclass Context:
           ...

context.pyx:

from libopenni cimport *
from xncontext cimport Context as c_Context 

cdef class Context:
    cdef c_Context *handle   
        ...

---

Related Topics