Wrapping a C Library in Python: C, Cython or Ctypes

Wrapping a C library in Python: C, Cython or ctypes?

ctypes is your best bet for getting it done quickly, and it's a pleasure to work with as you're still writing Python!

I recently wrapped an FTDI driver for communicating with a USB chip using ctypes and it was great. I had it all done and working in less than one work day. (I only implemented the functions we needed, about 15 functions).

We were previously using a third-party module, PyUSB, for the same purpose. PyUSB is an actual C/Python extension module. But PyUSB wasn't releasing the GIL when doing blocking reads/writes, which was causing problems for us. So I wrote our own module using ctypes, which does release the GIL when calling the native functions.

One thing to note is that ctypes won't know about #define constants and stuff in the library you're using, only the functions, so you'll have to redefine those constants in your own code.

Here's an example of how the code ended up looking (lots snipped out, just trying to show you the gist of it):

from ctypes import *

d2xx = WinDLL('ftd2xx')

OK = 0
INVALID_HANDLE = 1
DEVICE_NOT_FOUND = 2
DEVICE_NOT_OPENED = 3

...

def openEx(serial):
    serial = create_string_buffer(serial)
    handle = c_int()
    if d2xx.FT_OpenEx(serial, OPEN_BY_SERIAL_NUMBER, byref(handle)) == OK:
        return Handle(handle.value)
    raise D2XXException

class Handle(object):
    def __init__(self, handle):
        self.handle = handle
    ...
    def read(self, bytes):
        buffer = create_string_buffer(bytes)
        count = c_int()
        if d2xx.FT_Read(self.handle, buffer, bytes, byref(count)) == OK:
            return buffer.raw[:count.value]
        raise D2XXException
    def write(self, data):
        buffer = create_string_buffer(data)
        count = c_int()
        bytes = len(data)
        if d2xx.FT_Write(self.handle, buffer, bytes, byref(count)) == OK:
            return count.value
        raise D2XXException

Someone did some benchmarks on the various options.

I might be more hesitant if I had to wrap a C++ library with lots of classes/templates/etc. But ctypes works well with structs and can even callback into Python.

wrap a c library in cython which includes a python internal c symbol

As you've found, this is caused by macros defined by Python 2 causing the renaming. They aren't defined in Python 3, so one option is to upgrade.

Assuming you don't want to upgrade (which is probably reasonable...) then it's trickier. The easiest thing to try is just to add

#undef c_pow
#undef c_abs

to the top of jdmath.h. (Alternatively, if you can't modify jdmath.h then create a new header, that undefines the macros and includes jdmath.h then use that new header from Cython).

Hopefully this will work fine. It's possible however that this will break something in Cython (you'll get an obvious compile error) if it attempts to use the Python defined c_pow and c_abs itself. In this case you want to create a header that creates functions with new names:

// undefine so we can access c_pow and c_abs
#undef c_pow
#undef c_abs

#include "dev/jdmath.h"

inline void not_c_pow(/* fill in args yourself*/) { c_pow(/*args*/); }
inline void not_c_abs(/* fill in args yourself*/) { c_abs(/*args*/); }

// redefine to original state
#define c_pow _Py_c_pow
#define c_abs _Py_c_abs

In Cython you'd use not_c_pow and not_c_abs instead and it should work.

Is wrapping C++ library with ctypes a bad idea?

For C++ a library to be accessible from Python it must use C export names, which basically means that a function named foo will be accessible from ctypes as foo.

This can be achieved only by enclosing the public interface with export C {}, which in turn disallows function overloading and templates therein (only the public interface of the library to be wrapped is relevant, the inner workings are not and may use any C++ features they like).

Reason for this is that C++ compilers use a mechanism called name mangling to generate unique names for overloaded or templated symbols. While ctypes would still find a function provided you knew its mangled name, the mangling scheme depends on the compiler/linker being used and is nothing you can rely on. In short: do not use ctypes to wrap libraries that use C++ features in their public interface.

Cython takes a different approach. It aids you at building a C extension module that does the interfacing with the original library. Therefore, linking to the C++ library is done by the regular C++ linkage mechanism, thus avoiding the aforementioned problem. The trouble with Cython is that C extension libraries need to to be recompiled for every platform, but anyway, this applies to the C++ library to be wrapped as well.

Personally, I'd say that in most cases the time to fire up Cython is a time that is well-spent and will eventually pay off in comparison to ctypes (with an exception for really simple Cish interfaces).

I don't have any experience with boost.python, so I can't comment on it (however, I don't have the impression that it is very popular either).

wrapping a C library (GSL) in a cython code by using callback

After compiling it with the below commands:

cython test_gsl.pyx
gcc -m64 -pthread -fno-strict-aliasing -Wstrict-prototypes -DNDEBUG -g -fwrapv -O3 -Wall -fPIC -I/usr/include -I/vol/dalek/anaconda/include/python2.7 -c test_gsl.c -o build/temp.linux-x86_64-2.7/test_gsl.o
gcc -pthread -shared -L/usr/lib/ -L/vol/dalek/anaconda/lib -o test_gsl.so  build/temp.linux-x86_64-2.7/test_gsl.o -lpython2.7  -lgsl -lgslcblas -lm

If I import test_gsl in python without pyximport, it works perfectly. Since pyximport doesn't support linking against external libraries.

Creating a Python type in C using an external library: ctypes or setuptools?

The second approach also yields problems with distribution. And if you create a Python object in C you should do it in the context of a module. For scenarios where distribution is problematic, you could link this module statically instead.

For your issue with linking you'll find more information about Library options in the documentation. Since your library resides in a directory which should be in the standard library search path, you'd only need to define your library with the libraries option of the Extension class:

mymodule_ext = Extension('mymodule', ['mymodule.c'], libraries=['otherproject'])

If you're not using the standard lib* prefix you'd need to use libraries=[':otherproject.so'] instead.

---

Related Topics