What Is Inside So File of Python Library Distribution

Package only binary compiled .so files of a python library compiled with Cython

Unfortunately, the answer suggesting setting packages=[] is wrong and may break a lot of stuff, as can e.g. be seen in this question. Don't use it. Instead of excluding all packages from the dist, you should exclude only the python files that will be cythonized and compiled to shared objects.

Below is a working example; it uses my recipe from the question Exclude single source file from python bdist_egg or bdist_wheel. The example project contains package spam with two modules, spam.eggs and spam.bacon, and a subpackage spam.fizz with one module spam.fizz.buzz:

root
├── setup.py
└── spam
    ├── __init__.py
    ├── bacon.py
    ├── eggs.py
    └── fizz
        ├── __init__.py
        └── buzz.py

The module lookup is being done in the build_py command, so it is the one you need to subclass with custom behaviour.

Simple case: compile all source code, make no exceptions

If you are about to compile every .py file (including __init__.pys), it is already sufficient to override build_py.build_packages method, making it a noop. Because build_packages doesn't do anything, no .py file will be collected at all and the dist will include only cythonized extensions:

import fnmatch
from setuptools import find_packages, setup, Extension
from setuptools.command.build_py import build_py as build_py_orig
from Cython.Build import cythonize

extensions = [
    # example of extensions with regex
    Extension('spam.*', ['spam/*.py']),
    # example of extension with single source file
    Extension('spam.fizz.buzz', ['spam/fizz/buzz.py']),
]

class build_py(build_py_orig):
    def build_packages(self):
        pass

setup(
    name='...',
    version='...',
    packages=find_packages(),
    ext_modules=cythonize(extensions),
    cmdclass={'build_py': build_py},
)

Complex case: mix cythonized extensions with source modules

If you want to compile only selected modules and leave the rest untouched, you will need a bit more complex logic; in this case, you need to override module lookup. In the below example, I still compile spam.bacon, spam.eggs and spam.fizz.buzz to shared objects, but leave __init__.py files untouched, so they will be included as source modules:

import fnmatch
from setuptools import find_packages, setup, Extension
from setuptools.command.build_py import build_py as build_py_orig
from Cython.Build import cythonize

extensions = [
    Extension('spam.*', ['spam/*.py']),
    Extension('spam.fizz.buzz', ['spam/fizz/buzz.py']),
]
cython_excludes = ['**/__init__.py']

def not_cythonized(tup):
    (package, module, filepath) = tup
    return any(
        fnmatch.fnmatchcase(filepath, pat=pattern) for pattern in cython_excludes
    ) or not any(
        fnmatch.fnmatchcase(filepath, pat=pattern)
        for ext in extensions
        for pattern in ext.sources
    )

class build_py(build_py_orig):
    def find_modules(self):
        modules = super().find_modules()
        return list(filter(not_cythonized, modules))

    def find_package_modules(self, package, package_dir):
        modules = super().find_package_modules(package, package_dir)
        return list(filter(not_cythonized, modules))

setup(
    name='...',
    version='...',
    packages=find_packages(),
    ext_modules=cythonize(extensions, exclude=cython_excludes),
    cmdclass={'build_py': build_py},
)

How do I view python modules? They are .so files and PyCharm does not recognize how to open them

.so files are native shared libraries. They are written in C rather than Python. If you're curious about them, you can look at Python's source code.

How to edit and use a python package containing an .so file

Go to the site whence pip fetches things, find your package, and follow the link to its source distribution. Building that yourself often requires more tools and expertise than usingpip, which is the cost of customization. (The GPL, despite being more restrictive (in this peculiar sense) than most Free licenses, certainly allows merely providing Internet access to the sources for binaries so distributed.)

What is the right way to ship dynamic library *.so with python package?

Both ways in question could not work. I finally choose change RUNPATH of my dynamic library.

Use patchelf ( or chrpath if your *.so already have RUNPATH), put all dynamic libraries in one folder and run:

find . -name "*.so" | xargs -l1 -t patchelf --set-rpath "\$ORIGIN"

Dynamic library will search depends in RUNPATH while "\$ORIGIN" means current binary path, so if we put all *.so in one directory, depended *.so will be found.

then package them in wheel use

from setuptools import setup, find_packages
setup(
    ...
    package_data={
        # If any package contains *.so, include them:
        "": ["*.so",],
    }
)

now, everything works.

---

Related Topics