Extending Python with C/C++

Python C Extension with Multiple Functions

Make export "C" include both functions:

#include "func1.h"

extern "C" {

double func1(double x, double y, double z) {
    return x*x + y*y + z*z*z + func2(x,y,z);
}

double func2(double x, double y, double z) {
    return x*y*z;
}

}

extern "C" {

double func1(double x, double y, double z);
double func2(double x, double y, double z);

}

Without the braces, extern "C" is only applied to the next declaration, which is double func1(double x, double y, double z);.

Differences between Cython, extending C/C++ with Python.h, etc

It helps to know what you need to do here.

If you're not using ctypes for function calls, it's unlikely that it will save you anything to just have ctypes types involved. If you already have some DLL lying around with a "solve it for me" function, then sure, ctypes it is.

Cython creates extension modules, so anything you can do with Cython could also be done with an extension module, it just depends on how comfortable you are writing extensions by hand. Cython is more limited than writing extension by hand, and harder to "see" performance in (the rules for optimizing Cython are basically the opposite of optimizing CPython code, and if you forget to cdef the right things, you gain nothing), but Cython is generally simpler too.

Writing a separate non-extension DLL is only worthwhile if you have non-Python uses for it; otherwise, a Python extension is basically just the DLL case, but better integrated.

Basically, by definition, with infinite time and skill, a CPython extension will beat any other option on performance since it can do everything the others do, and more. It's just more work, and easy to make mistakes (because you're writing C, which is error prone).

How do you extend python with C++?

First of all, even though you don't want to introduce an additional dependency, I suggest you to have a look at PyCXX. Quoting its webpage:

CXX/Objects is a set of C++ facilities to make it easier to write Python extensions. The chief way in which PyCXX makes it easier to write Python extensions is that it greatly increases the probability that your program will not make a reference-counting error and will not have to continually check error returns from the Python C API. CXX/Objects integrates Python with C++ in these ways:

• C++ exception handling is relied on to detect errors and clean up. In a complicated function this is often a tremendous problem when writing in C. With PyCXX, we let the compiler keep track of what objects need to be dereferenced when an error occurs.
• The Standard Template Library (STL) and its many algorithms plug and play with Python containers such as lists and tuples.
• The optional CXX/Extensions facility allows you to replace the clumsy C tables with objects and method calls that define your modules and extension objects.

I think PyCXX is licensed under the BSD license, which means that you can just as well include the whole source code of PyCXX in the distributed tarball of your extension if your extension will be released under a similar license.

If you really and absolutely don't want to depend on PyCXX or any other third-party library, I think you only have to wrap functions that will be called by the Python interpreter in extern "C" { and } to avoid name mangling.

Here's the corrected code:

#include <Python.h>

#include "Flp.h"

static PyObject * ErrorObject;

typedef struct {
    PyObject_HEAD
    PyObject * x_attr; // attributes dictionary
} FlpObject;

extern "C" {
    static void Flp_dealloc(FlpObject * self);
    static PyObject * Flp_getattr(FlpObject * self, char * name);
    static int Flp_setattr(FlpObject * self, char * name, PyObject * v);
    DL_EXPORT(void) initflp();
}

static PyTypeObject Flp_Type = {
    /* The ob_type field must be initialized in the module init function
     * to be portable to Windows without using C++. */
    PyObject_HEAD_INIT(NULL)
    0,          /*ob_size*/
    "Flp",          /*tp_name*/
    sizeof(FlpObject),  /*tp_basicsize*/
    0,          /*tp_itemsize*/
    /* methods */
    (destructor)Flp_dealloc, /*tp_dealloc*/
    0,          /*tp_print*/
    (getattrfunc)Flp_getattr, /*tp_getattr*/
    (setattrfunc)Flp_setattr, /*tp_setattr*/
    0,          /*tp_compare*/
    0,          /*tp_repr*/
    0,          /*tp_as_number*/
    0,          /*tp_as_sequence*/
    0,          /*tp_as_mapping*/
    0,          /*tp_hash*/
};

#define FlpObject_Check(v) ((v)->ob_type == &Flp_Type)

static FlpObject * newFlpObject(PyObject * arg)
{
    FlpObject * self;
    self = PyObject_NEW(FlpObject, &Flp_Type);
    if (self == NULL)
        return NULL;
    self->x_attr = NULL;
    return self;
}

// Flp methods

static void Flp_dealloc(FlpObject * self)
{
    Py_XDECREF(self->x_attr);
    PyMem_DEL(self);
}

static PyObject * Flp_demo(FlpObject * self, PyObject * args)
{
    if (! PyArg_ParseTuple(args, ""))
        return NULL;
    Py_INCREF(Py_None);
    return Py_None;
}

static PyMethodDef Flp_methods[] = {
    {"demo",    (PyCFunction)Flp_demo,  1},
    {NULL,      NULL} // sentinel
};

static PyObject * Flp_getattr(FlpObject * self, char * name)
{
    if (self->x_attr != NULL) {
        PyObject * v = PyDict_GetItemString(self->x_attr, name);
        if (v != NULL) {
            Py_INCREF(v);
            return v;
        }
    }
    return Py_FindMethod(Flp_methods, (PyObject *)self, name);
}

static int Flp_setattr(FlpObject * self, char * name, PyObject * v)
{
    if (self->x_attr == NULL) {
        self->x_attr = PyDict_New();
        if (self->x_attr == NULL)
            return -1;
    }
    if (v == NULL) {
        int rv = PyDict_DelItemString(self->x_attr, name);
        if (rv < 0)
            PyErr_SetString(PyExc_AttributeError,
                    "delete non-existing Flp attribute");
        return rv;
    }
    else
        return PyDict_SetItemString(self->x_attr, name, v);
}
/* --------------------------------------------------------------------- */

/* Function of two integers returning integer */

static PyObject * flp_foo(PyObject * self, PyObject * args)
{
    long i, j;
    long res;
    if (!PyArg_ParseTuple(args, "ll", &i, &j))
        return NULL;
    res = i+j; /* flpX Do something here */
    return PyInt_FromLong(res);
}

/* Function of no arguments returning new Flp object */

static PyObject * flp_new(PyObject * self, PyObject * args)
{
    FlpObject *rv;
    
    if (!PyArg_ParseTuple(args, ""))
        return NULL;
    rv = newFlpObject(args);
    if ( rv == NULL )
        return NULL;
    return (PyObject *)rv;
}

/* Example with subtle bug from extensions manual ("Thin Ice"). */

static PyObject * flp_bug(PyObject * self, PyObject * args)
{
    PyObject *list, *item;
    
    if (!PyArg_ParseTuple(args, "O", &list))
        return NULL;
    
    item = PyList_GetItem(list, 0);
    /* Py_INCREF(item); */
    PyList_SetItem(list, 1, PyInt_FromLong(0L));
    PyObject_Print(item, stdout, 0);
    printf("\n");
    /* Py_DECREF(item); */
    
    Py_INCREF(Py_None);
    return Py_None;
}

/* Test bad format character */

static PyObject * flp_roj(PyObject * self, PyObject * args)
{
    PyObject *a;
    long b;
    if (!PyArg_ParseTuple(args, "O#", &a, &b))
        return NULL;
    Py_INCREF(Py_None);
    return Py_None;
}

/* List of functions defined in the module */

static PyMethodDef flp_methods[] = {
    {"roj",     flp_roj,     1},
    {"foo",     flp_foo,     1},
    {"new",     flp_new,     1},
    {"bug",     flp_bug,     1},
    {NULL,      NULL}       /* sentinel */
};

/* Initialization function for the module (*must* be called initflp) */

DL_EXPORT(void) initflp()
{
    PyObject *m, *d;

    /* Initialize the type of the new type object here; doing it here
     * is required for portability to Windows without requiring C++. */
    Flp_Type.ob_type = &PyType_Type;

    /* Create the module and add the functions */
    m = Py_InitModule("flp", flp_methods);

    /* Add some symbolic constants to the module */
    d = PyModule_GetDict(m);
    ErrorObject = PyErr_NewException("flp.error", NULL, NULL);
    PyDict_SetItemString(d, "error", ErrorObject);
}

How to extend Python and make a C-package?

I don't know if what you're trying to pull here (nested extension modules) is OK, anyway the recommended way for structuring code is via [Python 3.Docs]: Modules - Packages.
However, I did this (reproducing the problem, fixing it) as a personal exercise.

1. Intro

Listing the 2 relevant pages:

• [Python 3.Docs]: Module Objects
• [Python 2.Docs]: Module Objects

The environment:

[cfati@CFATI-5510-0:e:\Work\Dev\StackOverflow\q061692747]> tree /a /f
Folder PATH listing for volume SSD0-WORK
Volume serial number is AE9E-72AC
E:.
|   test00.py
|
+---py2
|       mod.c
|
\---py3
        helper.c
        mod.c

2. Python 2

Dummy module attempting to reproduce the behavior mentioned in the question.

mod.c:

#include <stdio.h>
#include <Python.h>

#define MOD_NAME "mod"
#define SUBMOD_NAME "submod"

static PyObject *pMod = NULL;
static PyObject *pSubMod = NULL;

static PyMethodDef modMethods[] = {
    {NULL}
};

PyMODINIT_FUNC initmod() {
    if (!pMod) {
        pMod = Py_InitModule(MOD_NAME, modMethods);
        if (pMod) {
            PyModule_AddIntConstant(pMod, "i", -69);
            pSubMod = Py_InitModule(MOD_NAME "." SUBMOD_NAME, modMethods);
            if (pSubMod) {
                PyModule_AddStringConstant(pSubMod, "s", "dummy");
                if (PyModule_AddObject(pMod, SUBMOD_NAME, pSubMod) < 0) {
                    Py_XDECREF(pMod);
                    Py_XDECREF(pSubMod);
                    return;
                }
            }
        }
    }
}

Output:

[cfati@CFATI-5510-0:e:\Work\Dev\StackOverflow\q061692747\py2]> sopr.bat
*** Set shorter prompt to better fit when pasted in StackOverflow (or other) pages ***

[prompt]> "f:\Install\pc032\Microsoft\VisualCForPython2\2008\Microsoft\Visual C++ for Python\9.0\vcvarsall.bat" x64
Setting environment for using Microsoft Visual Studio 2008 x64 tools.

[prompt]> dir /b
mod.c

[prompt]> cl /nologo /MD /DDLL /I"c:\Install\pc064\Python\Python\02.07.17\include" mod.c  /link /NOLOGO /DLL /OUT:mod.pyd /LIBPATH:"c:\Install\pc064\Python\Python\02.07.17\libs"
mod.c
   Creating library mod.lib and object mod.exp

[prompt]> dir /b
mod.c
mod.exp
mod.lib
mod.obj
mod.pyd
mod.pyd.manifest

[prompt]> "e:\Work\Dev\VEnvs\py_pc064_02.07.17_test0\Scripts\python.exe"
Python 2.7.17 (v2.7.17:c2f86d86e6, Oct 19 2019, 21:01:17) [MSC v.1500 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>>
>>> [item for item in sys.modules if "mod" in item]
[]
>>> import mod
>>>
>>> [item for item in sys.modules if "mod" in item]  # !!! NOTICE the contents !!!
['mod.submod', 'mod']
>>>
>>> mod
<module 'mod' from 'mod.pyd'>
>>> mod.i
-69
>>> mod.submod
<module 'mod.submod' (built-in)>
>>> mod.submod.s
'dummy'
>>>
>>> from mod.submod import s
>>> s
'dummy'
>>>

As seen, importing the module with submodules, adds the submodules in sys.path (didn't look, but I am 99.99% sure this is performed by Py_InitModule)

3. Python 3

Conversion to Python 3. Since this is the 1^st step, treat the 2 commented lines as they were not there.

mod.c:

#include <stdio.h>
#include <Python.h>
//#include "helper.c"

#define MOD_NAME "mod"
#define SUBMOD_NAME "submod"

static PyObject *pMod = NULL;
static PyObject *pSubMod = NULL;

static PyMethodDef modMethods[] = {
    {NULL}
};

static struct PyModuleDef modDef = {
    PyModuleDef_HEAD_INIT, MOD_NAME, NULL, -1, modMethods,
};

static struct PyModuleDef subModDef = {
    PyModuleDef_HEAD_INIT, MOD_NAME "." SUBMOD_NAME, NULL, -1, modMethods,
};

PyMODINIT_FUNC PyInit_mod() {
    if (!pMod) {
        pMod = PyModule_Create(&modDef);
        if (pMod) {
            PyModule_AddIntConstant(pMod, "i", -69);
            pSubMod = PyModule_Create(&subModDef);
            if (pSubMod) {
                PyModule_AddStringConstant(pSubMod, "s", "dummy");
                if (PyModule_AddObject(pMod, SUBMOD_NAME, pSubMod) < 0) {
                    Py_XDECREF(pMod);
                    Py_XDECREF(pSubMod);
                    return NULL;
                }
                //addToSysModules(MOD_NAME "." SUBMOD_NAME, pSubMod);
            }
        }
    }
    return pMod;
}

Output:

[cfati@CFATI-5510-0:e:\Work\Dev\StackOverflow\q061692747\py3]> sopr.bat
*** Set shorter prompt to better fit when pasted in StackOverflow (or other) pages ***

[prompt]> "c:\Install\pc032\Microsoft\VisualStudioCommunity\2017\VC\Auxiliary\Build\vcvarsall.bat" x64
**********************************************************************
** Visual Studio 2017 Developer Command Prompt v15.9.23
** Copyright (c) 2017 Microsoft Corporation
**********************************************************************
[vcvarsall.bat] Environment initialized for: 'x64'

[prompt]> dir /b
helper.c
mod.c

[prompt]> cl /nologo /MD /DDLL /I"c:\Install\pc064\Python\Python\03.07.06\include" mod.c  /link /NOLOGO /DLL /OUT:mod.pyd /LIBPATH:"c:\Install\pc064\Python\Python\03.07.06\libs"
mod.c
   Creating library mod.lib and object mod.exp

[prompt]> dir /b
helper.c
mod.c
mod.exp
mod.lib
mod.obj
mod.pyd

[prompt]> "e:\Work\Dev\VEnvs\py_pc064_03.07.06_test0\Scripts\python.exe"
Python 3.7.6 (tags/v3.7.6:43364a7ae0, Dec 19 2019, 00:42:30) [MSC v.1916 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>>
>>> [item for item in sys.modules if "mod" in item]
[]
>>> import mod
>>>
>>> [item for item in sys.modules if "mod" in item]  # !!! NOTICE the contents !!!
['mod']
>>>
>>> mod
<module 'mod' from 'e:\\Work\\Dev\\StackOverflow\\q061692747\\py3\\mod.pyd'>
>>> mod.i
-69
>>> mod.submod
<module 'mod.submod'>
>>> mod.submod.s
'dummy'
>>>
>>> from mod.submod import s
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ModuleNotFoundError: No module named 'mod.submod'; 'mod' is not a package
>>> ^Z

[prompt]>

As seen, nested import is not possible. That is because mod.submod is not present in sys.modules. As a generalization, "nested" extension submodules are no longer made importable through the module that contains them initialization function. the only option is to import them manually.
As a note: I think this Python 3 restriction is there for a reason, so what comes below is like playing with fire.

Decomment the 2 lines from mod.c.

helper.c:

int addToSysModules(const char *pName, PyObject *pMod) {
    PyObject *pSysModules = PySys_GetObject("modules");
    if (!PyDict_Check(pSysModules)) {
        return -1;
    }
    PyObject *pKey = PyUnicode_FromString(pName);
    if (!pKey) {
        return -2;
    }
    if (PyDict_Contains(pSysModules, pKey)) {
        Py_XDECREF(pKey);
        return -3;
    }
    Py_XDECREF(pKey);
    if (PyDict_SetItemString(pSysModules, pName, pMod) == -1)
    {
        return -4;
    }
    return 0;
}

Output:

[prompt]> cl /nologo /MD /DDLL /I"c:\Install\pc064\Python\Python\03.07.06\include" mod.c  /link /NOLOGO /DLL /OUT:mod.pyd /LIBPATH:"c:\Install\pc064\Python\Python\03.07.06\libs"
mod.c
   Creating library mod.lib and object mod.exp

[prompt]> "e:\Work\Dev\VEnvs\py_pc064_03.07.06_test0\Scripts\python.exe"
Python 3.7.6 (tags/v3.7.6:43364a7ae0, Dec 19 2019, 00:42:30) [MSC v.1916 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>>
>>> import sys
>>>
>>> [item for item in sys.modules if "mod" in item]
[]
>>> import mod
>>>
>>> [item for item in sys.modules if "mod" in item]  # !!! NOTICE the contents :) !!!
['mod.submod', 'mod']
>>>
>>> from mod.submod import s
>>> s
'dummy'
>>>

4. Closing notes

As I stated above, this seems more like a workaraound. A cleaner solution would be to better organize the modules via packages.

Since this is for demo purposes, and to keep the code as simple as possible, I didn't always check Python C API functions return codes. This can lead to hard to find errors (even crashes) and should never be done (especially in production code).

I am not very sure what PyImport_ExtendInittab effect really is as I didn't play with it, but [Python 3.Docs]: Importing Modules - int PyImport_ExtendInittab(struct _inittab *newtab) states (emphasis is mine):

This should be called before Py_Initialize().

So, calling it in our context, is out of the question.

Also mentioning this (old) discussion (not sure whether it contains relevant information, but still) [Python.Mail]: [Python-Dev] nested extension modules?.

When extending Python with C, How do one dynamically build a complex structure in C?

You're not supposed to use those functions to build dynamically-sized data structures. The FAQ says to use PyTuple_Pack instead of Py_BuildValue for an arbitrary-sized tuple, but that's wrong too; I don't know why it says that. PyTuple_Pack has the same varargs issues as Py_BuildValue.

To build a variable-length tuple from C, use PyTuple_New to construct an uninitialized tuple of the desired length, then loop over the indices and set the elements with PyTuple_SET_ITEM. Yes, this mutates the tuple. PyTuple_SET_ITEM is only safe to use to initialize fresh tuples that have not yet been exposed to other code. Also, be aware that PyTuple_SET_ITEM steals a reference to the new element.

For example, to build a tuple of integers from 0 to n-1:

PyObject *tup = PyTuple_New(n);
for (int i = 0; i < n; i++) {
    // Note that PyTuple_SET_ITEM steals the reference we get from PyLong_FromLong.
    PyTuple_SET_ITEM(tup, i, PyLong_FromLong(i));
}

To build a variable-length list from C, you can do the same thing with PyList_New and PyList_SET_ITEM, or you can construct an empty list with PyList_New(0) and append items with PyList_Append, much like you would use [] and append in Python if you didn't have list comprehensions or sequence multiplication.

How to Access Current File Path in Python C Extension

It depends on when you want to access the __file__ attribute and exactly how you create the module.

In single-phase initialization (when you call PyModule_Create within PyInit_yourmodulename) __file__ is set after the PyInit_* function is called. Therefore you can only access it after that point, when the module is fully imported.

In multi-phase initialization, where PyInit_yourmodulename returns a PyModuleDef object and then the Py_mod_create slot of that PyModuleDef is called, you can set the filename from the spec passed to Py_mod_create. The ModuleSpec object has an attribute origin. This corresponds to the file that the module was loaded from. Thus you can set the __file__ attribute of your module from spec.origin.

If you're creating your modules outside of the PyInit_yourmodulename system (for example, creating multiple modules from a single file) then you're out of luck and will have to work out what __file__ is yourself.

---

Related Topics