How to Implement a C++ Class in Python, to Be Called by C++

Calling a Python class method from C++, if given an initialised class as PyObject

Simpler than you're making it. You don't need to retrieve anything from the base class directly. Just do:

PyObject* result = PyObject_CallMethod(expression, "get_source", NULL);
if (result == NULL) {
    // Exception occurred, return your own failure status here
}
// result is a PyObject* (in this case, it should be a PyUnicode_Object)

PyObject_CallMethod takes an object to call a method of, a C-style string for the method name, and a format string + varargs for the arguments. When no arguments are needed, the format string can be NULL.

The resulting PyObject* isn't super useful to C++ code (it has runtime determined 1, 2 or 4 byte characters, depending on the ordinals involved, so straight memory copying from it into std::string or std::wstring won't work), but PyUnicode_AsUTF8AndSize can be used to get a UTF-8 encoded version and length, which can be used to efficiently construct a std::string with equivalent data.

If performance counts, you may want to explicitly make a PyObject* representing "get_source" during module load, e.g. with a global like:

PyObject *get_source_name;

which is initialized in the module's PyMODINIT_FUNC with:

get_source_name = PyUnicode_InternFromString("get_source");

Once you have that, you can use the more efficient PyObject_CallMethodObjArgs with:

PyObject* result = PyObject_CallMethodObjArgs(expression, get_source_name, NULL);

The savings there are largely in avoiding constructing a Python level str from a C char* over and over, and by using PyUnicode_InternFromString to construct the string, you're using the interned string, making the lookup more efficient (since the name of get_source is itself automatically interned when def-ed in the interpreter, no actual memory comparison of the contents takes place; it realizes the two strings are both interned, and just checks if they point to the same memory or not).

How can I use C++ class in Python?

Look into Boost.Python. It's a library to write python modules with C++.

Also look into SWIG which can also handle modules for other scripting languages. I've used it in the past to write modules for my class and use them within python. Works great.

You can do it manually by using the Python/C API, writing the interface yourself. It's pretty lowlevel, but you will gain a lot of additional knowledge of how Python works behind the scene (And you will need it when you use SWIG anyway).

How to call a C++ member function in python (SWIG)?

Default accessibility is private in C++, you then need to move it in a public: section :

class lib{
  public:
    lib();
    int test(int i);
};

Also note that test is a instance method, you need to instantiate the class.

Calling C functions in Python

If I understand well, you have no preference for dialoging as c => python or like python => c.
In that case I would recommend Cython. It is quite open to many kinds of manipulation, specially, in your case, calling a function that has been written in Python from C.

Here is how it works (public api) :

The following example assumes that you have a Python Class (self is an instance of it), and that this class has a method (name method) you want to call on this class and deal with the result (here, a double) from C. This function, written in a Cython extension would help you to do this call.

cdef public api double cy_call_func_double(object self, char* method, bint *error):
    if (hasattr(self, method)):
        error[0] = 0
        return getattr(self, method)();
    else:
        error[0] = 1

On the C side, you'll then be able to perform the call like so :

PyObject *py_obj = ....
...
if (py_obj) {
    int error;
    double result;
    result = cy_call_func_double(py_obj, (char*)"initSimulation", &error);
    cout << "Do something with the result : " << result << endl;
}

Where PyObject is a struct provided by Python/C API
After having caught the py_obj (by casting a regular python object, in your cython extension like this : <PyObject *>my_python_object), you would finally be able to call the initSimulation method on it and do something with the result.
(Here a double, but Cython can deal easily with vectors, sets, ...)

Well, I am aware that what I just wrote can be confusing if you never wrote anything using Cython, but it aims to be a short demonstration of the numerous things it can do for you in term of merging.

By another hand, this approach can take more time than recoding your Python code into C, depending on the complexity of your algorithms.
In my opinion, investing time into learning Cython is pertinent only if you plan to have this kind of needs quite often...

Hope this was at least informative...

python - how to implement a C-function as awaitable (coroutine)

Note: this answer covers CPython and the asyncio framework. The concepts, however, should apply to other Python implementations as well as other async frameworks.

How do I write a C-function so I can await on it?

The simplest way to write a C function whose result can be awaited is by having it return an already made awaitable object, such as an asyncio.Future. Before returning the Future, the code must arrange for the future's result to be set by some asynchronous mechanism. All of these coroutine-based approaches assume that your program is running under some event loop that knows how to schedule the coroutines.

But returning a future isn't always enough - maybe we'd like to define an object with an arbitrary number of suspension points. Returning a future suspends only once (if the returned future is not complete), resumes once the future is completed, and that's it. An awaitable object equivalent to an async def that contains more than one await cannot be implemented by returning a future, it has to implement a protocol that coroutines normally implement. This is somewhat like an iterator implementing a custom __next__ and be used instead of a generator.

Defining a custom awaitable

To define our own awaitable type, we can turn to PEP 492, which specifies exactly which objects can be passed to await. Other than Python functions defined with async def, user-defined types can make objects awaitable by defining the __await__ special method, which Python/C maps to the tp_as_async.am_await part of the PyTypeObject struct.

What this means is that in Python/C, you must do the following:

• specify a non-NULL value for the tp_as_async field of your extension type.
• have its am_await member point to a C function that accepts an instance of your type and returns an instance of another extension type that implements the iterator protocol, i.e. defines tp_iter (trivially defined as PyIter_Self) and tp_iternext.
• the iterator's tp_iternext must advance the coroutine's state machine. Each non-exceptional return from tp_iternext corresponds to a suspension, and the final StopIteration exception signifies the final return from the coroutine. The return value is stored in the value property of StopIteration.

For the coroutine to be useful, it must also be able to communicate with the event loop that drives it, so that it can specify when it is to be resumed after it has suspended. Most of coroutines defined by asyncio expect to be running under the asyncio event loop, and internally use asyncio.get_event_loop() (and/or accept an explicit loop argument) to obtain its services.

Example coroutine

To illustrate what the Python/C code needs to implement, let's consider simple coroutine expressed as a Python async def, such as this equivalent of asyncio.sleep():

async def my_sleep(n):
    loop = asyncio.get_event_loop()
    future = loop.create_future()
    loop.call_later(n, future.set_result, None)
    await future
    # we get back here after the timeout has elapsed, and
    # immediately return

my_sleep creates a Future, arranges for it to complete (its result to become set) in n seconds, and suspends itself until the future completes. The last part uses await, where await x means "allow x to decide whether we will now suspend or keep executing". An incomplete future always decides to suspend, and the asyncio Task coroutine driver special-cases yielded futures to suspend them indefinitely and connects their completion to resuming the task. Suspension mechanisms of other event loops (curio etc) can differ in details, but the underlying idea is the same: await is an optional suspension of execution.

__await__() that returns a generator

To translate this to C, we have to get rid of the magic async def function definition, as well as of the await suspension point. Removing the async def is fairly simple: the equivalent ordinary function simply needs to return an object that implements __await__:

def my_sleep(n):
    return _MySleep(n)

class _MySleep:
    def __init__(self, n):
        self.n = n

    def __await__(self):
        return _MySleepIter(self.n)

The __await__ method of the _MySleep object returned by my_sleep() will be automatically called by the await operator to convert an awaitable object (anything passed to await) to an iterator. This iterator will be used to ask the awaited object whether it chooses to suspend or to provide a value. This is much like how the for o in x statement calls x.__iter__() to convert the iterable x to a concrete iterator.

When the returned iterator chooses to suspend, it simply needs to produce a value. The meaning of the value, if any, will be interpreted by the coroutine driver, typically part of an event loop. When the iterator chooses to stop executing and return from await, it needs to stop iterating. Using a generator as a convenience iterator implementation, _MySleepIter would look like this:

def _MySleepIter(n):
    loop = asyncio.get_event_loop()
    future = loop.create_future()
    loop.call_later(n, future.set_result, None)
    # yield from future.__await__()
    for x in future.__await__():
        yield x

As await x maps to yield from x.__await__(), our generator must exhaust the iterator returned by future.__await__(). The iterator returned by Future.__await__ will yield if the future is incomplete, and return the future's result (which we here ignore, but yield from actually provides) otherwise.

__await__() that returns a custom iterator

The final obstacle for a C implementation of my_sleep in C is the use of generator for _MySleepIter. Fortunately, any generator can be translated to a stateful iterator whose __next__ executes the piece of code up to the next await or return. __next__ implements a state machine version of the generator code, where yield is expressed by returning a value, and return by raising StopIteration. For example:

class _MySleepIter:
    def __init__(self, n):
        self.n = n
        self.state = 0

    def __iter__(self):  # an iterator has to define __iter__
        return self

    def __next__(self):
        if self.state == 0:
            loop = asyncio.get_event_loop()
            self.future = loop.create_future()
            loop.call_later(self.n, self.future.set_result, None)
            self.state = 1
        if self.state == 1:
            if not self.future.done():
                return next(iter(self.future))
            self.state = 2
        if self.state == 2:
            raise StopIteration
        raise AssertionError("invalid state")

Translation to C

The above is quite some typing, but it works, and only uses constructs that can be defined with native Python/C functions.

Actually translating the two classes to C quite straightforward, but beyond the scope of this answer.

---

Related Topics