Extending Setuptools Extension to Use Cmake in Setup.Py

Extending setuptools with CMake. Build extension does not install

Its best to just you skbuild. They have examples for c++, cython, c. I just put together some examples for f2py/fortran: https://github.com/Nicholaswogan/skbuild-f2py-examples

Python 2.7 C++ extensions with CMake and SetupTools?

The answer to "How do I tell my setup.py which files to keep, and where will it put them?" seems to be self.get_ext_fullpath(ext.name) as is found in this solution as pointed out by abarnert and hoefling. This method returns a temporary root directory into which you can put your build output. Pip then appears to scan that directory and copy everything into your real environment after the build is done.

Unfortunately, the Linux dynamic linker can't find my dependency (libmy_project.so) for my extension, as virtualenv doesn't set LD_LIBRARY_PATH or anything, so I ended up just rolling all of the compilation units into my extension. The full solution is below:

setup.py

import os
import subprocess

from setuptools import setup, Extension
from setuptools.command.build_ext import build_ext

class CMakeExtension(Extension):
    def __init__(self, name, sourcedir=''):
        Extension.__init__(self, name, sources=[])
        self.sourcedir = os.path.abspath(sourcedir)

class CMakeBuild(build_ext):
    def run(self):
        for ext in self.extensions:
            self.build_extension(ext)

    def build_extension(self, ext):
        if not os.path.exists(self.build_temp):
            os.makedirs(self.build_temp)

        extdir = self.get_ext_fullpath(ext.name)
        if not os.path.exists(extdir):
            os.makedirs(extdir)

        # This is the temp directory where your build output should go
        install_prefix = os.path.abspath(os.path.dirname(extdir))
        cmake_args = ['-DCMAKE_LIBRARY_OUTPUT_DIRECTORY={}'.format(install_prefix)]

        subprocess.check_call(['cmake', ext.sourcedir, cmake_args], cwd=self.build_temp)
        subprocess.check_call(['cmake', '--build', '.'], cwd=self.build_temp)

setup(
    name='my_project',
    version='0.0.1',
    author='Me',
    author_email='rcv@stackoverflow.com',
    description='A really cool and easy to install library',
    long_description='',
    ext_modules=[CMakeExtension('.')],
    cmdclass=dict(build_ext=CMakeBuild),
    zip_safe=False,
    install_requires=[
        'cmake',
    ]
)

CMakeLists.txt

cmake_minimum_required (VERSION 3.0.0)
project (my_project)

# Find SomePackage
find_package(SomePackage REQUIRED COMPONENTS cool_unit great_unit)
include_directories(${SomePackage_INCLUDE_DIR})

# Find Python
find_package(PythonLibs 2.7 REQUIRED)
include_directories(${PYTHON_INCLUDE_DIR})

# Build py_my_project
add_library(py_my_project SHARED src/python/pyMyProject.cpp src/MyProject.cpp)
set_target_properties(py_my_project PROPERTIES PREFIX "")
target_link_libraries(py_my_project ${SomePackage_LIBRARIES} 
                      ${PYTHON_LIBRARIES})

file(WRITE ${CMAKE_CURRENT_BINARY_DIR}/__init__.py "__all__ = ['py_my_project']")

Mix cython and cmake extensions in python setuptools

The workaround I found:

from setuptools import setup, Extension
from setuptools.command.build_ext import build_ext

class CMakeExtension(Extension):
    # ...

class CMakeBuild(build_ext):
    # ...

if sys.argv[1] == "build_ext":
    c_ext = [Extension(...)]
elif sys.argv[1] == "build_cmk":
    c_ext = [CMakeExtension(...)]
else:
    raise NotImplementedError

setup(
    # ...
    ext_modules=cythonize(c_ext),
    cmdclass={"build_ext": build_ext, "build_cmk": CMakeBuild},
    # ...
)

And then run:

python setup.py build_ext --inplace
python setup.py build_cmk --inplace

Package a pre-built python extension

Obviously, this is not the most robust way to distribute python-packages as it will not work for different OSes or may lead to strange results if there is Python-version mismatch - but nevertheless it is possible.

Let's consider following folder structure:

/
|--- setup.py
|--- my_package
        |-------  __init__.py
        |-------  impl.pyx          [needed only for creation of impl.so]
        |-------  impl-XXX.so       [created via "cythonize -i impl.pyx"]

With the following content:

__init__.py:

from .impl import foo

impl.pyx:

def foo():
    print("I'm foo from impl")

setup.py:

from setuptools import setup, find_packages

kwargs = {
      'name':'my_package',
      'version':'0.1.0',
      'packages':find_packages(),

       #ensure so-files are copied to the installation:
      'package_data' : { 'my_package': ['*.so']},
      'include_package_data' : True,
      'zip_safe' : False  
}

setup(**kwargs)

Now after calling python setup.py install, the package is installed and can be used:

>>> python -c "import my_package; my_package.foo()"
I'm foo from impl

NB: Don't call the test from the folder with the setup file, because then not the installed but local version of my_package can be used.

---

You might want to have different so-binaries for different Python versions. It is possible to have the same extension compiled for different Python versions - you have to add the right suffix to the resulting shared library, for example:

• impl.cpython-36m-x86_64-linux-gnu.so for Python3.6 on my linux machine
• impl.cpython-37m-x86_64-linux-gnu.so for Python3.7
• impl.cp36-win_amd64.pyd on windows

One can get the suffix for extensions on the current machine using

>>> import importlib
>>> importlib.machinery.EXTENSION_SUFFIXES
['.cp36-win_amd64.pyd', '.pyd']

How do I build a Python extension module with CMake?

The directory where setuptools looks for the compiled module can be obtained by build_ext.get_ext_fullpath(ext.name). In the above code the resulting path is passed to CMake by setting the variable CMAKE_LIBRARY_OUTPUT_DIRECTORY_RELEASE.

Since f2py is invoked through a custom command, the extension module is not automatically copied to the output directory. This can be achieved by another call to add_custom_command:

  add_custom_command(TARGET "${F2PY_TARGET_MODULE_NAME}" POST_BUILD
    COMMAND "${CMAKE_COMMAND}" -E copy_if_different "${CMAKE_CURRENT_BINARY_DIR}/${generated_module_file}" "${CMAKE_LIBRARY_OUTPUT_DIRECTORY_RELEASE}/${generated_module_file}")

Building Python submodules from C++ extensions via cmake

I think I've resolved the issue with the following lines:

Change in the setup.py file:

ext_modules = [
    Extension(
        "frontend.backend", sources=["frontend/backend/backend.cpp"]
    )
]

Change in the CMakeLists.txt file:

pybind11_add_module(backend "frontend/backend/backend.cpp")
set_target_properties( backend
    PROPERTIES
    ARCHIVE_OUTPUT_DIRECTORY "${CMAKE_BINARY_DIR}/frontend"
    LIBRARY_OUTPUT_DIRECTORY "${CMAKE_BINARY_DIR}/frontend"
    RUNTIME_OUTPUT_DIRECTORY "${CMAKE_BINARY_DIR}/frontend"
)

The shared library object backend.platform.so must be located in the frontend directory. Neither the pybind module name nor the sourcefile .cpp should contain any "." in the names, because the get_ext_fullpath() method from build_ext will split by dots. Only the frontend directory containts an init.py file.

---

Related Topics