C++ Back End Call the Python Level Defined Callbacks with Swig Wrapper

Passing python functions to SWIG wrapped C++ code

You can implement the callback logic in Python by using "directors".

Basically, instead of passing callback functions, you pass callback objects instead. The base object can be defined in C++ and provide a virtual callback member function. This object can then be inherited from and the callback function overwritten in Python. The inherited object can then be passed to a C++ function instead of a callback function. For this to work, you need to enable the director feature for such a callback class.

This does require changing the underlying C++ library, though.

What Is The Cleanest Way to Call A Python Function From C++ with a SWIG Wrapped Object

Below is my working solution for solving this problem. It uses the suggestions from both @omnifarious and @flexo above.

In particular we create a Callback class with a SWIG director and then derive from it in Python to get the required callback functionality without introducing a circular dependency.

In addition we provide an interface which allows any Python object that is callable to act as a callback. We achieve this by using the "pythonprend" directive in SWIG to prepend some code to the "setCallback" function. This code simply checks for a callable object and if it finds one, wraps it in an instance of a Callback.

Finally we deal with the memory issues related to having a C++ class (ObjWithPyCallback) reference a director object (i.e. a subclass of Callback).

File example.py:

import cb

class CB(cb.Callback):
    def __init__(self):
        super(CB, self).__init__()
    def call(self, x):
        print("Hello from CB!")
        print(x)

def foo(x):
    print("Hello from foo!")
    print(x)

class Bar:
    def __call__(self, x):
        print("Hello from Bar!")
        print(x)

o = cb.ObjWithPyCallback()
mycb=CB()
o.setCallback(mycb)
o.call()
o.setCallback(foo)
o.call()
o.setCallback(Bar())
o.call()

File ObjWithPyCallback.i:

%module(directors="1") cb
%{
   #include "Callback.h"
   #include "ObjWithPyCallback.h"
%}
%feature("director") Callback;
%feature("nodirector") ObjWithPyCallback;

%feature("pythonprepend") ObjWithPyCallback::setCallback(Callback&) %{
   if len(args) == 1 and (not isinstance(args[0], Callback) and callable(args[0])):
      class CallableWrapper(Callback):
         def __init__(self, f):
            super(CallableWrapper, self).__init__()
            self.f_ = f
         def call(self, obj):
            self.f_(obj)

      args = tuple([CallableWrapper(args[0])])
      args[0].__disown__()
   elif len(args) == 1 and isinstance(args[0], Callback):
      args[0].__disown__()

%}

%include "Callback.h"
%include "ObjWithPyCallback.h"

File Callback.h:

#ifndef CALLBACK_H
#define CALLBACK_H

class ObjWithPyCallback;

class Callback
{
   public:
      Callback(){}

      virtual ~Callback(){}
      virtual void call(ObjWithPyCallback& object){} 
};

#endif

File ObjWithPyCallback.h:

#ifndef OBJWITHPYCALLBACK_H
#define OBJWITHPYCALLBACK_H

class Callback;

class ObjWithPyCallback 
{
   public:

      ObjWithPyCallback();
      ~ObjWithPyCallback();
      void setCallback(Callback &callback);
      void call();

   private:

      Callback* callback_;
};

#endif

File ObjWithPyCallback.cpp:

#include "ObjWithPyCallback.h"
#include "Callback.h"

#include <iostream>

ObjWithPyCallback::ObjWithPyCallback() : callback_(NULL) {}

ObjWithPyCallback::~ObjWithPyCallback()
{
}

void ObjWithPyCallback::setCallback(Callback &callback)
{
   callback_ = &callback;
}

void ObjWithPyCallback::call()
{
   if ( ! callback_ )
   {
      std::cerr << "No callback is set.\n";
   }
   else
   {
      callback_->call(*this);
   }
}

Python Callback from SWIG PyObject_Call Segfault

Since the C++ calling the python callback is within a wxWidget, and the swig wrapper is generated by the special wxPython swig (wxswig?) There is some thread protection required around the function call...

The fixed operator should look like this

void PyMenuCallback::operator() (int id)
{
    cout << "Calling Callback" << endl;
    if (Func == 0 || Func == Py_None || !PyCallable_Check(Func))
        return;
    cout << "Building Args" << endl;   
    PyObject *arglist = Py_BuildValue ("(i)",id);
    cout << "Built: " << arglist << endl;
    cout << "Func: " << Func->ob_type->tp_name << " " << Func->ob_refcnt << endl;

    wxPyBlock_t blocked = wxPyBeginBlockThreads(); //Anti-WxSwig 

    PyObject *result = PyObject_Call(Func,arglist,0);

    wxPyEndBlockThreads(blocked);

    cout << "Executed" << endl;
    Py_XDECREF(arglist);
    Py_XDECREF(result);
}

Make sure to include

#include "wx/wxPython/wxPython.h"
#include "wx/wxPython/wxPython_int.h"

Using SWIG with pointer to function in C struct

I found an answer. If I declare the function pointer as a SWIG "member function", it seems to work as expected:

%module test
%{

typedef struct {
  int (*my_func)(int);
} test_struct;

int add1(int n) { return n+1; }

test_struct *init_test()
{
    test_struct *t = (test_struct*) malloc(sizeof(test_struct));
    t->my_func = add1;
    return t;
}

%}

typedef struct {
    int my_func(int);
} test_struct;

extern test_struct *init_test();

Session:

$ python
Python 2.6.2 (release26-maint, Apr 19 2009, 01:56:41) 
[GCC 4.3.3] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import test
>>> t = test.init_test()
>>> t.my_func(1)
2

I was hoping for something that wouldn't require writing any custom SWIG-specific code (I'd prefer to just "%include" my headers without modification), but this will do I guess.

How to use SWIG to wrap std::function objects?

Although SWIG doesn't provide a std_function.i natively we can build one ourselves this with a bit of work. My answer here is a more generalised version of my a previous of mine answer, looking at this problem for a specific instance and targeting Python. I'll go through several iterations of it, which define a %std_function macro for generic std::function wrapping.

I'm assuming there are four things you want to achieve from a wrapping of std::function, which become our main requirements:

We want to be able to call std::function objects from within Java code.
The wrapped std::function objects need to get passed around like any other object, including crossing the language boundaries in either direction.
It should be possible to write std::function objects inside of Java, which can be passed back to C++ without having to modify existing C++ code that works on std::function objects (i.e. maintaining type erasure of std::function cross language)
We should be able to construct std::function objects in Java using C++ pointer to function types.

I'm going to work through these and show how we can achieve this. Where possible I'll keep the solution language agnostic too.

For the purposes of discussion I'm glossing over the shared_ptr part of your question, it doesn't actually change things because as you've got shared_ptr working that's actually sufficient to use it in this scenario too, it would just make my examples more verbose needlessly.

The solution I'm working towards is actually modelled after the existing shared_ptr support in SWIG. I've put together a test interface to illustrate how it will be used:

%module test
%include "std_function.i"

%std_function(Functor, void, int, double);

%{
#include <iostream>
%}

%inline %{
  std::function<void(int,double)> make_functor() {
    return [](int x, double y){
      std::cout << x << ", " << y << "\n";
    };
  }
%}

Basically to use this all you need do is include the file "std_function.i" and then use the macro %std_function which takes arguments as:

%std_function(Name, Ret, ...)

You call this once per instantiation of the std::function template you want to wrap, where Name is what you want to call the type in Java, Ret is the return type and then the remaining (variadic) arguments are the inputs to your function. So in my test interface above I'm basically looking to wrap std::function<void(int,double)>.

Writing a first version of "std_function.i" isn't actually too tricky. All you need to get basic working requirements #1 and #2 is:

%{
  #include <functional>
%}

%define %std_function(Name, Ret, ...)
%rename(Name) std::function<Ret(__VA_ARGS__)>;
%rename(call) std::function<Ret(__VA_ARGS__)>::operator();
namespace std {
  struct function<Ret(__VA_ARGS__)> {
    // Copy constructor
    function<Ret(__VA_ARGS__)>(const std::function<Ret(__VA_ARGS__)>&);

    // Call operator
    Ret operator()(__VA_ARGS__) const;
  };
}

%enddef

This includes the C++ header file in the generated wrapper code once and then sets up the macro for usage in interfaces. SWIG's support for C++11 variadic templates isn't actually very helpful for us in this usage scenario, so the macro I wrote basically re-implements the default template expansion functionality using C99 variadic macro arguments (which are much better supported). Coincidentally this means the SWIG code we're writing will work with 2.x or even some 1.3.x releases. (I tested with 2.x). Even if/when your version of SWIG does have %template support that works with std::function retaining this macro is still useful for the rest of the glue that makes it actually callable.

The manual expansion of the std:function template is limited to just the bits we care about for our usage: the actual operator() and a copy constructor that might come in handy.

The only other thing to be done is renaming operator() to something that matches the target language, e.g. for Java renaming it to be just a regular function called "call", or if you were targeting Python to __call__ or using tp_slots if required.

This is now sufficient to let our interface work, to demonstrate it I wrote a little bit of Java:

public class run {
    public static void main(String[] argv) {
        System.loadLibrary("test");
        test.make_functor().call(1,2.5);
    }
}

Which I compiled with:

swig2.0 -Wall -c++ -java  test.i
g++ -Wall -Wextra -std=c++11 test_wrap.cxx -o libtest.so -I/usr/lib/jvm/default-java/include/ -I/usr/lib/jvm/default-java/include/linux -shared -fPIC
javac run.java
LD_LIBRARY_PATH=. java run

and it worked.

Requirement #4 is pretty easy to cross off the list now at this point. All we need to do is tell SWIG there's another constructor in std::function which accepts compatible function pointers:

// Conversion constructor from function pointer
function<Ret(__VA_ARGS__)>(Ret(*const)(__VA_ARGS__));

And then we can use this with the %callback mechanism in SWIG, our test interface file becomes:

%module test
%include "std_function.i"

%std_function(Functor, void, int, double);

%{
#include <iostream>
void add_and_print(int a, double b) {
  std::cout << a+b << "\n";
}
%}

%callback("%s_cb");
void add_and_print(int a, double b);
%nocallback;

%inline %{
  std::function<void(int,double)> make_functor() {
    return [](int x, double y){
      std::cout << x << ", " << y << "\n";
    };
  }
%}

and the Java we use to call this is then:

public class run {
    public static void main(String[] argv) {
    System.loadLibrary("test");
    test.make_functor().call(1,2.5);
    new Functor(test.add_and_print_cb).call(3,4.5);
    }
}

Which we compile and run identically successfully at this point.

(Note that it's normal and desirable to see some Java classes created at this point that start with the name "SWIGTYPE_p_f_..." - these wrap the "pointer to function" types that are used by the pointer to function constructor and callback constants)

Requirement #3 is where things start to get trickier. Essentially we've hit the same problem as I answered on making SWIG generate an interface in Java previously, except now we're looking to do it within a macro more generically.

It turns out that in this instance because the interface we want to generate is fairly simple we can use some tricks inside our macro to make SWIG generate it for us.

The main thing that we need to do in order to make this work is to setup SWIG directors to provide cross-language polymorphism and allow something written in Java to implement a C++ interface. This is the class generated with the suffix "Impl" in my code.

To make things "feel right" to Java developers we want to still use the same type for both C++ and Java implemented std::function objects. Even if std::function::operator() were virtual we still don't want to have SWIG directors use that type directly as it's pretty common to pass std::function by value which would lead to type slicing problems. So when a Java developer extends our std::function objects and overrides call we need to do some extra work to make it so that C++ which uses that object actually calls the Java implementation given that we can't just use directors to handle this automatically.

So what we do looks a little bit weird. If you construct a Java object that is meant to implement std::function then there's a special, protected constructor for that. This constructor leaves the swigCPtr member variable, which normally points to a real C++ object as 0 and instead creates an anonymous wrapper object that implements the "Impl" interface and simply proxies everything back to the call member of the Java object.

We have another typemap too that gets applied, in Java, everywhere we pass a std::function object to C++. Its role is to detect which case we have - a C++ implemented std::function object, or a Java one. In the C++ case it does nothing special and everything proceeds as normal. In the Java case it takes the proxy object and asks C++ to convert it back to another, separate std::function instance which gets substituted instead.

This is enough to get us the behaviour we want in both languages without anything that feels weird on either side (other than a lot of mechanical lifting that happens transparently).

The catch here is that it's non-trivial to automatically construct the proxy object. Java has dynamic proxy classes as part of the reflection API, but these only implement interfaces, not extend abstract classes. One possibility I did try to use was void call(Object ...args) on the Java side, which is a variadic function argument. Whilst legal this didn't seem to actually override any cases in the super class as would be needed.

What I ended up doing was adapting some macros to iterate over variadic macro arguments in the way I wanted. This is a fairly sensible solution given that we already decided to use variadic C99 macro arguments for other reasons. This mechanism gets used four times in total in my solution, once in the function declaration and once in the delgated call for both Java and C++. (C++ has perfect forwarding properties retained and Java needs a typemap lookup to be performed, so they are distinct in each and every case).

There's also a custom typemap to simplify some of the Java code - in a void function it's not legal to write return other_void_function();, so we would need to special case void functions if it weren't for that.

So let's see what that looks like in reality. First up is the run.java I used for testing, it's only slightly modified from previous examples to add a Java implementation of the std::function object.

public class run extends Functor {
    public static void main(String[] argv) {
        System.loadLibrary("test");
        test.make_functor().call(1,2.5);

        new Functor(test.add_and_print_cb).call(3,4.5);

        Functor f = new run();
        test.do_things(f);
    }

    @Override
    public void call(int a, double b) {
        System.out.println("Java: " + a + ", " + b);
    }
}

The std_function.i is now substantially larger with all the changes outlined above:

%{
  #include <functional>
  #include <iostream>

  #ifndef SWIG_DIRECTORS
  #error "Directors must be enabled in your SWIG module for std_function.i to work correctly"
  #endif
%}

// These are the things we actually use
#define param(num,type) $typemap(jstype,type) arg ## num
#define unpack(num,type) arg##num
#define lvalref(num,type) type&& arg##num
#define forward(num,type) std::forward<type>(arg##num)

// This is the mechanics
#define FE_0(...)
#define FE_1(action,a1) action(0,a1)
#define FE_2(action,a1,a2) action(0,a1), action(1,a2)
#define FE_3(action,a1,a2,a3) action(0,a1), action(1,a2), action(2,a3)
#define FE_4(action,a1,a2,a3,a4) action(0,a1), action(1,a2), action(2,a3), action(3,a4)
#define FE_5(action,a1,a2,a3,a4,a5) action(0,a1), action(1,a2), action(2,a3), action(3,a4), action(4,a5)

#define GET_MACRO(_1,_2,_3,_4,_5,NAME,...) NAME
%define FOR_EACH(action,...) 
  GET_MACRO(__VA_ARGS__, FE_5, FE_4, FE_3, FE_2, FE_1, FE_0)(action,__VA_ARGS__)
%enddef

%define %std_function(Name, Ret, ...)

%feature("director") Name##Impl;
%typemap(javaclassmodifiers) Name##Impl "abstract class";

%{
  struct Name##Impl {
    virtual ~Name##Impl() {}
    virtual Ret call(__VA_ARGS__) = 0;
  };
%}

%javamethodmodifiers Name##Impl::call "abstract protected";
%typemap(javaout) Ret Name##Impl::call ";" // Suppress the body of the abstract method

struct Name##Impl {
  virtual ~Name##Impl();
protected:
  virtual Ret call(__VA_ARGS__) = 0;
};

%typemap(maybereturn) SWIGTYPE "return ";
%typemap(maybereturn) void "";

%typemap(javain) std::function<Ret(__VA_ARGS__)> "$javaclassname.getCPtr($javaclassname.makeNative($javainput))"
%typemap(javacode) std::function<Ret(__VA_ARGS__)> %{
  protected Name() {
    wrapper = new Name##Impl(){
      public $typemap(jstype, Ret) call(FOR_EACH(param, __VA_ARGS__)) {
    $typemap(maybereturn, Ret)Name.this.call(FOR_EACH(unpack, __VA_ARGS__));
      }
    };
    proxy = new $javaclassname(wrapper);
  }

  static $javaclassname makeNative($javaclassname in) {
    if (null == in.wrapper) return in;
    return in.proxy;
  }

  // Bot of these are retained to prevent garbage collection from happenign to early
  private Name##Impl wrapper;
  private $javaclassname proxy;
%}

%rename(Name) std::function<Ret(__VA_ARGS__)>;
%rename(call) std::function<Ret(__VA_ARGS__)>::operator();

namespace std {
  struct function<Ret(__VA_ARGS__)> {
    // Copy constructor
    function<Ret(__VA_ARGS__)>(const std::function<Ret(__VA_ARGS__)>&);

    // Call operator
    Ret operator()(__VA_ARGS__) const;

    // Conversion constructor from function pointer
    function<Ret(__VA_ARGS__)>(Ret(*const)(__VA_ARGS__));

    %extend {
      function<Ret(__VA_ARGS__)>(Name##Impl *in) {
    return new std::function<Ret(__VA_ARGS__)>([=](FOR_EACH(lvalref,__VA_ARGS__)){
          return in->call(FOR_EACH(forward,__VA_ARGS__));
    });
      }
    }
  };
}

%enddef

And test.i is slightly expanded to validate the Java -> C++ passing of std::function objects and enable directors:

%module(directors="1") test
%include "std_function.i"

%std_function(Functor, void, int, double);

%{
#include <iostream>
void add_and_print(int a, double b) {
  std::cout << a+b << "\n";
}
%}

%callback("%s_cb");
void add_and_print(int a, double b);
%nocallback;

%inline %{
  std::function<void(int,double)> make_functor() {
    return [](int x, double y){
      std::cout << x << ", " << y << "\n";
    };
  }

  void do_things(std::function<void(int,double)> in) {
    in(-1,666.6);
  }
%}

This compiled and ran as with the previous examples. It's worth noting that we've crossed into writing a lot of Java specific code - although the design would work for other languages if you were targeting Python it's a lot simpler to fix some of these issues using Python specific features.

There are two things I'd like to improve:

Use C++14 variadic lambdas to avoid the macro preprocessor magic I've used to keep them compatible with C++11. If you have C++ 14 the %extend constructor becomes:

%extend {
  function<Ret(__VA_ARGS__)>(Name##Impl *in) {
    return new std::function<Ret(__VA_ARGS__)>([=](auto&& ...param){
      return in->call(std::forward<decltype(param)>(param)...);
    });
  }
}

When it comes to using this macro with std::shared_ptr as expected the macro itself needs no changes. There is however a problem with the implementation of the javadirectorin and directorin typemaps that get applied, which do prevent things from "just working". This is true even with a build of SWIG from "trunk". (There's an outstanding question on combining directors and shared_ptr)

We can work around that though, by adding two additional typemaps in the main .i file of our module right after the call to %shared_ptr:

%shared_ptr(some::ns::TheThing);
%typemap(javadirectorin) std::shared_ptr<some::ns::TheThing> "new $typemap(jstype, some::ns::TheThing)($1,false)";
%typemap(directorin,descriptor="L$typemap(jstype, some::ns::TheThing);") std::shared_ptr<some::ns::TheThing> %{ 
  *($&1_type*)&j$1 = &$1;
%}

The first of these two typemaps is actually dead code because we forced the "call" method to be abstract in our abstract class, but it's easier to fix the compilation of this method than it is to suppress it. The second typemap is important. It's substantially similar to the normal "out" typemap in that it creates a jlong which is really just a representation of a C++ pointer, i.e. it prepares an object to go from C++, to Java.

Note that you might need to modify the descriptor attribute of the directorin typemap if you use packages in your module, either to "L$packagepath/$typemap(...);" or simply write it by hand.

This should remove the spurious "SWIGTYPE_p_sstd__shared_ptr..." type generated now as well. If you have virtual functions that return shared_ptr objects you'll need to write directorout and javadirectorout typemaps for them too. These can be base on the normal "in" typemap.

This was sufficient for my own simple testing with a modified Functor to work, at least with my version of SWIG checked out from the trunk today. (My test with 2.0.x failed and I didn't put much effort into making it work since this is a known work in progress area).