Integrate Python and C++

Integrate Python And C++

Interfacing Python with C/C++ is not an easy task.

Here I copy/paste a previous answer on a previous question for the different methods to write a python extension. Featuring Boost.Python, SWIG, Pybindgen...

• You can write an extension yourself in C or C++ with the Python C-API.

  In a word: don't do that except for learning how to do it. It's very difficult to do it correctly. You will have to increment and decrement references by hand and write a lot of code just to expose one function, with very few benefits.

• Swig:

  pro: you can generate bindings for many scripting languages.

  cons: I don't like the way the parser works. I don't know if they've made some progress but two years ago the C++ parser was quite limited. Most of the time I had to copy/paste my .h headers to add some % characters and to give extra hints to the swig parser.

  I also needed to deal with the Python C-API from time to time for (not so) complicated type conversions.

  I'm not using it anymore.

• Boost.Python:

  pro:
  It's a very complete library. It allows you to do almost everything that is possible with the C-API, but in C++. I never had to write a C-API code with this library. I also never encountered a bug due to the library. Code for bindings either works like a charm or refuses to compile.

  It's probably one of the best solutions currently available if you already have some C++ library to bind. But if you only have a small C function to rewrite, I would probably try with Cython.

  cons: if you don't have a precompiled Boost.Python library you're going to use Bjam (sort of a replacement of make). I really hate Bjam and its syntax.

  Python libraries created with B.P tend to become obese. It also takes a lot of time to compile them.

• Py++: it's Boost.Python made easy. Py++ uses a C++ parser to read your code and then generates Boost.Python code automatically. You also have a great support from its author (no it's not me ;-) ).

  cons: only the problems due to Boost.Python itself.

  Edit this project looks discontinued. While probably still working it may be better to consider switching.

• Pybindgen:

  It generates the code dealing with the C-API. You can either describe functions and classes in a Python file, or let Pybindgen read your headers and generate bindings automatically (for this it uses pygccxml, a python library wrote by the author of Py++).

  cons: it's a young project, with a smaller team than Boost.Python. There are still some limitations: you cannot expose your own C++ exceptions, you cannot use multiple inheritance for your C++ classes.

  Anyway it's worth trying!

• Pyrex and Cython:

  Here you don't write real C/C++ but a mix between Python and C. This intermediate code will generate a regular Python module.

Edit Jul 22 2013: Now Py++ looks discontinued, I'm now looking for a good alternative. I'm currently experimenting with Cython for my C++ library. This language is a mix between Python and C. Within a Cython function you can use either Python or C/C++ entities (functions, variables, objects, ...).

Cython is quite easy to learn, has very good performance, and you can even avoid C/C++ completely if you don't have to interface legacy C++ libraries.

However for C++ it comes with some problems. It is less "automagic" than Py++ was, so it's probably better for stable C++ API (which is now the case of my library). The biggest problem I see with Cython is with C++ polymorphism. With Py++/boost:python I was able to define a virtual method in C++, override it in Python, and have the Python version called within C++. With Cython it's still doable but you need to explicitly use the C-Python API.

Edit 2017-10-06:

There is a new one, pybind11, similar to Boost.Python but with some potential advantages. For example it uses C++11 language features to make it simpler to create new bindings. Also it is a header-only library, so there is nothing to compile before using it, and no library to link.

I played with it a little bit and it was indeed quite simple and pleasant to use. My only fear is that like Boot.Python it could lead to long compilation time and large libraries. I haven't done any benchmark yet.

integrate python with C++

I decided to use pybind11 to integrate python and c++. A simple and good reference for learning working with the library, you can use this tutorial

Is it reasonable to integrate python with c for performance?

In my experience it is rarely necessary to optimize using C. I prefer to identify bottlenecks and improve algorithms in those areas completely in Python. Using hash tables, caching, and generally re-organizing your data structures to suit future needs has amazing potential for speeding up your program. As your program develops you'll get a better sense of what kind of material can be precalculated, so don't be afraid to go back and redo your storage and algorithms. Additionally, look for chances to kill "two birds with one stone", such as sorting objects as you render them instead of doing huge sorts.

When everything is worked to the best of your knowledge, I'd consider using an optimizer like Psyco. I've experienced literally 10x performance improvements just by using Psyco and adding one line to my program.

If all else fails, use C in the proper places and you'll get what you want.

What is the most conventional way to integrate C code into a Python library using distutils?

The shared libraries are the right way to go in this case. The distutils have ability to build the static libraries as follows:

from distutils.ccompiler import new_compiler
from distutils import sysconfig

c = new_compiler()
workdir = "."
c.add_include_dir( sysconfig.get_python_inc() )
c.add_include_dir("./include")
objects = c.compile(["file1.c", "file2.c"])
c.link_shared_lib(objects, "mylibrary", output_dir=workdir)

This will generate the .so library in the working directory.

For example how it's used in real setup see the following example

Use Python code in C/C++

Embedding the CPython interpreter into a C or C++ program is actually pretty straightforward.

The official documentation has some complete examples.

Also, check out SWIG and Boost.Python.

Calling C/C++ from Python?

You should have a look at Boost.Python. Here is the short introduction taken from their website:

The Boost Python Library is a framework for interfacing Python and
C++. It allows you to quickly and seamlessly expose C++ classes
functions and objects to Python, and vice-versa, using no special
tools -- just your C++ compiler. It is designed to wrap C++ interfaces
non-intrusively, so that you should not have to change the C++ code at
all in order to wrap it, making Boost.Python ideal for exposing
3rd-party libraries to Python. The library's use of advanced
metaprogramming techniques simplifies its syntax for users, so that
wrapping code takes on the look of a kind of declarative interface
definition language (IDL).

Can we use C code in Python?

I want to invoke those C function or executables in python. Is that possible.

Yes, you can write C code that can be imported into Python as a module. Python calls these extension modules. You can invoke it from Python directly, an example from the documentation:

Python Code

import example
result = example.do_something()

C Code

static PyObject * example(PyObject *self)
{
    // do something
    return Py_BuildValue("i", result);
}

If I want the C code to be a library, which means I use it with #include and linkage of the *.o likely in python, how to do it or is that possible.

You build it as a shared library *.dll or *.so
You can also investigate using distutils to distribute your module.

If I write the C code into executable, which means it becomes a command, can I invoke it in python directly?

If you write a *.exe then you are doing the opposite (invoking Python from C). The method you choose (exe vs shared library) depends on if you want a "C program with some Python" or a "Python program with some C".

Also, I heard that python code can be compiled, does that mean we can execute the code without the source file? Are the output files binary files? Does it improve performance?

Python reads *.py files and compiles to *.pyc bytecode files when you run it. The bytecode is then run in the Python virtual machine. This means "executing the same file is faster the second time as recompilation from source to bytecode can be avoided." (from the Python glossary) So if you haven't edited your *.py files, it will run the *.pyc. You can distribute *.pyc files without *.py files, however they are not encrypted and can be reverse-engineered.

---

Related Topics