Pyeval_Initthreads in Python 3: How/When to Call It? (The Saga Continues Ad Nauseam)

PyEval_InitThreads in Python 3: How/when to call it? (the saga continues ad nauseam)

Your understanding is correct: invoking PyEval_InitThreads does, among other things, acquire the GIL. In a correctly written Python/C application, this is not an issue because the GIL will be unlocked in time, either automatically or manually.

If the main thread goes on to run Python code, there is nothing special to do, because Python interpreter will automatically relinquish the GIL after a number of instructions have been executed (allowing another thread to acquire it, which will relinquish it again, and so on). Additionally, whenever Python is about to invoke a blocking system call, e.g. to read from the network or write to a file, it will release the GIL around the call.

The original version of this answer pretty much ended here. But there is one more thing to take into account: the embedding scenario.

When embedding Python, the main thread often initializes Python and goes on to execute other, non-Python-related tasks. In that scenario there is nothing that will automatically release the GIL, so this must be done by the thread itself. That is in no way specific to the call that calls PyEval_InitThreads, it is expected of all Python/C code invoked with the GIL acquired.

For example, the main() might contain code like this:

Py_Initialize();
PyEval_InitThreads();

Py_BEGIN_ALLOW_THREADS
... call the non-Python part of the application here ...
Py_END_ALLOW_THREADS

Py_Finalize();

If your code creates threads manually, they need to acquire the GIL before doing anything Python-related, even as simple as Py_INCREF. To do so, use the following:

// Acquire the GIL
PyGILState_STATE gstate;
gstate = PyGILState_Ensure();

... call Python code here ...

// Release the GIL. No Python API allowed beyond this point.
PyGILState_Release(gstate);

Embedding python in multithreaded C application

Eventually I figured it out.

After

PyEval_InitThreads();

You need to call

PyEval_SaveThread();

While properly release the GIL for the main thread.

How to call Python from a boost thread?

The most likely culprit is that the Global Interpreter Lock (GIL) is not being held by a thread when it is invoking Python code, resulting in undefined behavior. Verify all paths that make direct or indirect Python calls, acquire the GIL before invoking Python code.

---

The GIL is a mutex around the CPython interpreter. This mutex prevents parallel operations to be performed on Python objects. Thus, at any point in time, a max of one thread, the one that has acquired the GIL, is allowed to perform operations on Python objects. When multiple threads are present, invoking Python code whilst not holding the GIL results in undefined behavior.

C or C++ threads are sometimes referred to as alien threads in the Python documentation. The Python interpreter has no ability to control the alien thread. Therefore, alien threads are responsible for managing the GIL to permit concurrent or parallel execution with Python threads. One must meticulously consider:

• The stack unwinding, as Boost.Python may throw an exception.
• Indirect calls to Python, such as copy-constructors or destructors

One solution is to wrap Python callbacks with a custom type that is aware of GIL management.

---

Using a RAII-style class to manage the GIL provides an elegant exception-safe solution. For example, with the following with_gil class, when a with_gil object is created, the calling thread acquires the GIL. When the with_gil object is destructed, it restores the GIL state.

/// @brief Guard that will acquire the GIL upon construction, and
///        restore its state upon destruction.
class with_gil
{
public:
  with_gil()  { state_ = PyGILState_Ensure(); }
  ~with_gil() { PyGILState_Release(state_);   }

  with_gil(const with_gil&)            = delete;
  with_gil& operator=(const with_gil&) = delete;
private:
  PyGILState_STATE state_;
};

And its usage:

{
  with_gil gil;                      // Acquire GIL.
  // perform Python calls, may throw
}                                    // Restore GIL.

With being able to manage the GIL via with_gil, the next step is to create a functor that properly manages the GIL. The following py_callable class will wrap a boost::python::object and acquire the GIL for all paths in which Python code is invoked:

/// @brief Helper type that will manage the GIL for a python callback.
///
/// @detail GIL management:
///           * Acquire the GIL when copying the `boost::python` object
///           * The newly constructed `python::object` will be managed
///             by a `shared_ptr`.  Thus, it may be copied without owning
///             the GIL.  However, a custom deleter will acquire the
///             GIL during deletion
///           * When `py_callable` is invoked (operator()), it will acquire
///             the GIL then delegate to the managed `python::object`
class py_callable
{
public:

  /// @brief Constructor that assumes the caller has the GIL locked.
  py_callable(const boost::python::object& object)
  {
    with_gil gil;
    object_.reset(
      // GIL locked, so it is safe to copy.
      new boost::python::object{object},
      // Use a custom deleter to hold GIL when the object is deleted.
      [](boost::python::object* object)
      {
        with_gil gil;
        delete object;
      });
  }

  // Use default copy-constructor and assignment-operator.
  py_callable(const py_callable&) = default;
  py_callable& operator=(const py_callable&) = default;

  template <typename ...Args>
  void operator()(Args... args)
  {
    // Lock the GIL as the python object is going to be invoked.
    with_gil gil;
    (*object_)(std::forward<Args>(args)...);
  }

private:
  std::shared_ptr<boost::python::object> object_;
};

By managing the boost::python::object on the free-space, one can freely copy the shared_ptr without having to hold the GIL. This allows for us to safely use the default generated copy-constructor, assignment operator, destructor, etc.

One would use the py_callable as follows:

// thread 1
boost::python::object object = ...; // GIL must be held.
py_callable callback(object);       // GIL no longer required.
work_queue.post(callback);

// thread 2
auto callback = work_queue.pop();   // GIL not required.
// Invoke the callback.  If callback is `py_callable`, then it will
// acquire the GIL, invoke the wrapped `object`, then release the GIL.
callback(...);   

---

Here is a complete example demonstrating having a Python extension invoke a Python object as a callback from a C++ thread:

#include <memory>  // std::shared_ptr
#include <thread>  // std::this_thread, std::thread
#include <utility> // std::forward
#include <boost/python.hpp>

/// @brief Guard that will acquire the GIL upon construction, and
///        restore its state upon destruction.
class with_gil
{
public:
  with_gil()  { state_ = PyGILState_Ensure(); }
  ~with_gil() { PyGILState_Release(state_);   }

  with_gil(const with_gil&)            = delete;
  with_gil& operator=(const with_gil&) = delete;
private:
  PyGILState_STATE state_;
};

/// @brief Helper type that will manage the GIL for a python callback.
///
/// @detail GIL management:
///           * Acquire the GIL when copying the `boost::python` object
///           * The newly constructed `python::object` will be managed
///             by a `shared_ptr`.  Thus, it may be copied without owning
///             the GIL.  However, a custom deleter will acquire the
///             GIL during deletion
///           * When `py_callable` is invoked (operator()), it will acquire
///             the GIL then delegate to the managed `python::object`
class py_callable
{
public:

  /// @brief Constructor that assumes the caller has the GIL locked.
  py_callable(const boost::python::object& object)
  {
    with_gil gil;
    object_.reset(
      // GIL locked, so it is safe to copy.
      new boost::python::object{object},
      // Use a custom deleter to hold GIL when the object is deleted.
      [](boost::python::object* object)
      {
        with_gil gil;
        delete object;
      });
  }

  // Use default copy-constructor and assignment-operator.
  py_callable(const py_callable&) = default;
  py_callable& operator=(const py_callable&) = default;

  template <typename ...Args>
  void operator()(Args... args)
  {
    // Lock the GIL as the python object is going to be invoked.
    with_gil gil;
    (*object_)(std::forward<Args>(args)...);
  }

private:
  std::shared_ptr<boost::python::object> object_;
};

BOOST_PYTHON_MODULE(example)
{
  // Force the GIL to be created and initialized.  The current caller will
  // own the GIL.
  PyEval_InitThreads();

  namespace python = boost::python;
  python::def("call_later",
    +[](int delay, python::object object) {
      // Create a thread that will invoke the callback.
      std::thread thread(+[](int delay, py_callable callback) {
        std::this_thread::sleep_for(std::chrono::seconds(delay));
        callback("spam");
      }, delay, py_callable{object});
      // Detach from the thread, allowing caller to return.
      thread.detach();
  });
}

Interactive usage:

>>> import time
>>> import example
>>> def shout(message):
...     print message.upper()
...
>>> example.call_later(1, shout)
>>> print "sleeping"; time.sleep(3); print "done sleeping"
sleeping
SPAM
done sleeping

---

Related Topics