How to Catch Python Stdout in C++ Code

How To catch python stdout in c++ code

If I'm reading your question correctly, you want to capture stdout/stderr into a variable within your C++? You can do this by redirecting stdout/stderr into a python variable and then querying this variable into your C++. Please not that I have not done the proper ref counting below:

#include <Python.h>
#include <string>

int main(int argc, char** argv)
{
    std::string stdOutErr =
"import sys\n\
class CatchOutErr:\n\
    def __init__(self):\n\
        self.value = ''\n\
    def write(self, txt):\n\
        self.value += txt\n\
catchOutErr = CatchOutErr()\n\
sys.stdout = catchOutErr\n\
sys.stderr = catchOutErr\n\
"; //this is python code to redirect stdouts/stderr

    Py_Initialize();
    PyObject *pModule = PyImport_AddModule("__main__"); //create main module
    PyRun_SimpleString(stdOutErr.c_str()); //invoke code to redirect
    PyRun_SimpleString("print(1+1)"); //this is ok stdout
    PyRun_SimpleString("1+a"); //this creates an error
    PyObject *catcher = PyObject_GetAttrString(pModule,"catchOutErr"); //get our catchOutErr created above
    PyErr_Print(); //make python print any errors

    PyObject *output = PyObject_GetAttrString(catcher,"value"); //get the stdout and stderr from our catchOutErr object

    printf("Here's the output:\n %s", PyString_AsString(output)); //it's not in our C++ portion

    Py_Finalize();

    return 0;

}

How to catch Python 3 stdout in C++ code

Your issue is that .value isn't a bytes object, it is a string (i.e. Python2 unicode) object. Therefore PyBytes_AsString fails. We can convert it to a bytes object with PyUnicode_AsEncodedString.

PyObject *output = PyObject_GetAttrString(catcher,"value"); //get the stdout and stderr from our catchOutErr
PyObject* encoded = PyUnicode_AsEncodedString(output,"utf-8","strict");
printf("Here's the output:\n %s", PyBytes_AsString(encoded));

Note that you should be checking these result PyObject* against NULL to see if an error has occurred.

In python, how to capture the stdout from a c++ shared library to a variable

Python's sys.stdout object is simply a Python wrapper on top of the usual stdout file descriptor—changing it only affects the Python process, not the underlying file descriptor. Any non-Python code, whether it be another executable which was exec'ed or a C shared library which was loaded, won't understand that and will continue using the ordinary file descriptors for I/O.

So, in order for the shared library to output to a different location, you need to change the underlying file descriptor by opening a new file descriptor and then replacing stdout using os.dup2(). You could use a temporary file for the output, but it's a better idea to use a pipe created with os.pipe(). However, this has the danger for deadlock, if nothing is reading the pipe, so in order to prevent that we can use another thread to drain the pipe.

Below is a full working example which does not use temporary files and which is not susceptible to deadlock (tested on Mac OS X).

C shared library code:

// test.c
#include <stdio.h>

void hello(void)
{
  printf("Hello, world!\n");
}

Compiled as:

$ clang test.c -shared -fPIC -o libtest.dylib

Python driver:

import ctypes
import os
import sys
import threading

print 'Start'

liba = ctypes.cdll.LoadLibrary('libtest.dylib')

# Create pipe and dup2() the write end of it on top of stdout, saving a copy
# of the old stdout
stdout_fileno = sys.stdout.fileno()
stdout_save = os.dup(stdout_fileno)
stdout_pipe = os.pipe()
os.dup2(stdout_pipe[1], stdout_fileno)
os.close(stdout_pipe[1])

captured_stdout = ''
def drain_pipe():
    global captured_stdout
    while True:
        data = os.read(stdout_pipe[0], 1024)
        if not data:
            break
        captured_stdout += data

t = threading.Thread(target=drain_pipe)
t.start()

liba.hello()  # Call into the shared library

# Close the write end of the pipe to unblock the reader thread and trigger it
# to exit
os.close(stdout_fileno)
t.join()

# Clean up the pipe and restore the original stdout
os.close(stdout_pipe[0])
os.dup2(stdout_save, stdout_fileno)
os.close(stdout_save)

print 'Captured stdout:\n%s' % captured_stdout

Capture stdout from a running C program with Python

The output is being buffered so you need to useiter(p.stdout.readline,"")

p = Popen("a.exe", stdout=PIPE)

for line in iter(p.stdout.readline,""):
     print line

Try flushing stdout from c if sys.stdout.flush() if is not working, as far as I know lines are block buffered when writing to a pipe:

int main() {
    for (int i=0;; i++) {
        printf("%i\n",i);
        Sleep(100);
        fflush(stdout);

    }
    return 0;
}

Redirect stdout from python for C calls

I've written below a few additional comments that should make clearer what it's going on inside the redirect_stdout function:

def redirect_stdout():
    print "Redirecting stdout"
    sys.stdout.flush() # <--- important when redirecting to files

    # Duplicate stdout (file descriptor 1)
    # to a different file descriptor number
    newstdout = os.dup(1)

    # /dev/null is used just to discard what is being printed
    devnull = os.open('/dev/null', os.O_WRONLY)

    # Duplicate the file descriptor for /dev/null
    # and overwrite the value for stdout (file descriptor 1)
    os.dup2(devnull, 1)

    # Close devnull after duplication (no longer needed)
    os.close(devnull)

    # Use the original stdout to still be able
    # to print to stdout within python
    sys.stdout = os.fdopen(newstdout, 'w')

One important thing to note is that a process gets three different file descriptors from the OS when launched:

• stdin: 0
• stdout: 1
• stderr: 2

As explained in the comments, the code above takes advantage of the file descriptor for stdout and the file descriptor duplication functions to make trick the C code into using a different stdout while still keeping a reference to the original stdout in the python code to be able to print.

How to capture stdout output from a Python function call?

Try this context manager:

from io import StringIO 
import sys

class Capturing(list):
    def __enter__(self):
        self._stdout = sys.stdout
        sys.stdout = self._stringio = StringIO()
        return self
    def __exit__(self, *args):
        self.extend(self._stringio.getvalue().splitlines())
        del self._stringio    # free up some memory
        sys.stdout = self._stdout

Usage:

with Capturing() as output:
    do_something(my_object)

output is now a list containing the lines printed by the function call.

Advanced usage:

What may not be obvious is that this can be done more than once and the results concatenated:

with Capturing() as output:
    print('hello world')

print('displays on screen')

with Capturing(output) as output:  # note the constructor argument
    print('hello world2')

print('done')
print('output:', output)

Output:

displays on screen                     
done                                   
output: ['hello world', 'hello world2']

Update: They added redirect_stdout() to contextlib in Python 3.4 (along with redirect_stderr()). So you could use io.StringIO with that to achieve a similar result (though Capturing being a list as well as a context manager is arguably more convenient).

Python C API: how to flush stdout and stderr?

You can do two things from what I remember based on the PySys_WriteStdout() call which simply calls sysmodule.sys_write(_Py_Identifier *key, FILE *fp, const char *format, va_list va):

• unset the sys.stdout, thus you'll access the raw STDOUT, not just Python's wrapper and for that you should be able to use standard C flushing
• get the reference for sys.stdout and flush it directly

Optionally, just set PYTHONUNBUFFERED as env var, but it's not as flexible and isn't present by default in the system

For the sys.stdout flushing, you might also be able to utilize PySys_GetObject(const char *name) directly and then call the io.IOBase.flush() in the sys.stdout reference.

PySys_GetObject("__stdout__") according to the implementation in the pylifecycle.init_sys_streams(PyThreadState *tstate)

Or if you want to live dangerously, just pull the private API (_PySys_GetObjectId(&PyId_stdout)) to utilize the same way CPython does. It's defined in bltinmodule.c via _Py_IDENTIFIER(varname).

Here is how CPython does the flushing via its pylifecycle.flush_std_files() function.

---

Related Topics