Getting Another Program's Output as Input on the Fly

Getting another program's output as input on the fly

Just set stdout to be line buffered at the beginning of your C program (before performing any output), like this:

#include <stdio.h>
setvbuf(stdout, NULL, _IOLBF, 0);

or

#include <stdio.h>
setlinebuf(stdout);

Either one will work on Linux, but setvbuf is part of the C standard so it will work on more systems.

By default stdout will be block buffered for a pipe or file, or line buffered for a terminal. Since stdout is a pipe in this case, the default will be block buffered. If it is block buffered then the buffer will be flushed when it is full, or when you call fflush(stdout). If it is line buffered then it will be flushed automatically after each line.

Parsing program output line by line on-the-fly

To read and write data in the script you want to use to filter the data just read and write from/to standard input/output.

./myProgram model test | ./filter.py > myOutput

filter.py:

import sys

for line in sys.stdin:
    if some_condition:
        sys.stdout.write(line)

If the condition is just to have some pattern in the data you don't need a script, you can simply use grep to filter the lines:

 ./myProgram model test | grep 'interesting_pattern' > myOutput

on-the-fly output redirection, seeing the file redirection output while the program is still running

From the stdout manual page:

The stream stderr is unbuffered.
The stream stdout is line-buffered
when it points to a terminal.
Partial lines will not appear until
fflush(3) or exit(3) is called, or
a new‐line is printed.

Bottom line: Unless the output is a terminal, your program will have its standard output in fully buffered mode by default. This essentially means that it will output data in large-ish blocks, rather than line-by-line, let alone character-by-character.

Ways to work around this:

• Fix your program: If you need real-time output, you need to fix your program. In C you can use fflush(stdout) after each output statement, or setvbuf() to change the buffering mode of the standard output. For Python there is sys.stdout.flush() of even some of the suggestions here.

• Use a utility that can record from a PTY, rather than outright stdout redirections. GNU Screen can do this for you:

  screen -d -m -L python test.py
  

  would be a start. This will log the output of your program to a file called screenlog.0 (or similar) in your current directory with a default delay of 10 seconds, and you can use screen to connect to the session where your command is running to provide input or terminate it. The delay and the name of the logfile can be changed in a configuration file or manually once you connect to the background session.

EDIT:

On most Linux system there is a third workaround: You can use the LD_PRELOAD variable and a preloaded library to override select functions of the C library and use them to set the stdout buffering mode when those functions are called by your program. This method may work, but it has a number of disadvantages:

• It won't work at all on static executables

• It's fragile and rather ugly.

• It won't work at all with SUID executables - the dynamic loader will refuse to read the LD_PRELOAD variable when loading such executables for security reasons.

• It's fragile and rather ugly.

• It requires that you find and override a library function that is called by your program after it initially sets the stdout buffering mode and preferably before any output. getenv() is a good choice for many programs, but not all. You may have to override common I/O functions such as printf() or fwrite() - if push comes to shove you may just have to override all functions that control the buffering mode and introduce a special condition for stdout.

• It's fragile and rather ugly.

• It's hard to ensure that there are no unwelcome side-effects. To do this right you'd have to ensure that only stdout is affected and that your overrides will not crash the rest of the program if e.g. stdout is closed.

• Did I mention that it's fragile and rather ugly?

That said, the process it relatively simple. You put in a C file, e.g. linebufferedstdout.c the replacement functions:

#define _GNU_SOURCE
#include <stdlib.h>
#include <stdio.h>
#include <dlfcn.h>

char *getenv(const char *s) {
    static char *(*getenv_real)(const char *s) = NULL;

    if (getenv_real == NULL) {
        getenv_real = dlsym(RTLD_NEXT, "getenv");

        setlinebuf(stdout);
    }

    return getenv_real(s);
}

Then you compile that file as a shared object:

gcc -O2 -o linebufferedstdout.so -fpic -shared linebufferedstdout.c -ldl -lc

Then you set the LD_PRELOAD variable to load it along with your program:

$ LD_PRELOAD=./linebufferedstdout.so python test.py | tee -a test.out 
0
1000
2000
3000
4000

If you are lucky, your problem will be solved with no unfortunate side-effects.

You can set the LD_PRELOAD library in the shell, if necessary, or even specify that library system-wide (definitely NOT recommended) in /etc/ld.so.preload.

Read from another process' output stream

stdout is buffered and it's not automatically flushed if you are not writing to screen. Add:

fflush(stdout);

after:

printf("%#lx %s\n", (UInt32)event.elementCookie,
    (event.value == 0) ? "depressed" : "pressed");

Is there a way to output the tail on the fly?

Try

 program -flag1 -flag2 argument | tail

Taking Output of function and putting it into another function in C

1) You need to use a prototype or declare your functions forward.

2) You can not return 2 variables form a function in C, but you can pass an array and read/write his values:

#include <stdio.h>

void guess(int arr[])
{
    printf("\nEnter you guess: ");
    scanf("%d,%d", &arr[0], &arr[1]);
}

void check(int arr[])
{
    printf("%d %d ",arr[0], arr[1]);
}

int main(void)
{
    int arr[2];

    guess(arr);
    check(arr);
    return 0;
}

or you can pass a reference

void guess(int *a, int *b)
{
    printf("\nEnter you guess: ");
    scanf("%d,%d", a, b);
}

guess(&a, &b);

Java reading standard output from an external program using inputstream

You have to consume both the program's stdout and stderr concurrently to avoid blocking scenarios.

See this article for more info, and in particular note the StreamGobbler mechanism that captures stdout/err in separate threads. This is essential to prevent blocking and is the source of numerous errors if you don't do it properly!

How to pass output from a python program into a processing program

I've never used the Sense HAT before, but I'm guessing it's using I2C behind the scenes. In theory it should be possible to reimplement the code in Processing using it's I2C io library, but in practice it may take quite a bit of effort, looking at the sense-hat library uses RTIMU and all the fancy filtering that does on it's own.

To get your Python program to talk to Processing you have at least two options:

1. pipe the output from the python program into Processing's stdin and parse what's coming through
2. Use sockets.

The second option should be simpler and I can think of mutiple options:

1. raw UDP sockets
2. OSC using PyOSC for the Python and oscP5 for Processing
3. Using WebSockets

I'd recommend the second option again: UDP is pretty fast and OSC on top of that makes it east to pass messages with arguments.

The Python script would:

• poll orientation data
• share orientation values via a message like /orientation

The Processing sketch would:

• be an OSC Server server and wait for data
• fetch the 3 float arguments from the /orientation message received and draw

Here's an untested proof of concept sender script in Python:

import time
from sense_hat import SenseHat
from OSC import OSCClient, OSCMessage

#update 60 times a second -> feel free to adjust this what works best
delay = 1.0/60.0 
# sense hat
sense = SenseHat()
# OSC client -> Processing
client = OSCClient()
client.connect( ("localhost", 12000) )

while True:
    # read sense hat
    orientation = sense.get_orientation_degrees()
    print("p: {pitch}, r: {roll}, y: {yaw}".format(**orientation))
    # send data to Processing
    client.send( OSCMessage("/orientation", [orientation["pitch"],orientation["roll"],orientation["yaw"] ] ) )
    # wait
    time.sleep(delay)

and the Processing receiver:

import oscP5.*;
import netP5.*;

OscP5 oscP5;

float pitch,roll,yaw;

void setup() {
  size(400,400,P3D);
  frameRate(25);
  /* start oscP5, listening for incoming messages at port 12000 */
  oscP5 = new OscP5(this,12000);
}

void draw() {
  background(0);  
  text("pitch: " + pitch + "\nroll: " + roll + "\nyaw: " + yaw,10,15);
}

/* incoming osc message are forwarded to the oscEvent method. */
void oscEvent(OscMessage message) {
  message.print();
  if(message.checkAddrPattern("/orientation")==true) {
    /* check if the typetag is the right one. -> expecting float (pitch),float (roll), float (yaw)*/
    if(message.checkTypetag("fff")) {
      pitch = message.get(0).floatValue();
      roll  = message.get(1).floatValue();
      yaw   = message.get(2).floatValue();
    }
  }
}

Note that you'll need to install PyOSC and run the Processing sketch before hand, otherwise you might get a Python error about the OSCClient not being able to connect. The idea is Processing becomes an OSC Server and the Python Script is an OSCClient and the server need to be available for the client to connect. (You can make the Python script an OSC Server if you want and the Processing sketch a client if that works better for you)

To install PyOSC try:

sudo pip install pyosc

Otherwise:

cd ~/Downloads
wget https://pypi.python.org/packages/7c/e4/6abb118cf110813a7922119ed0d53e5fe51c570296785ec2a39f37606d85/pyOSC-0.3.5b-5294.tar.gz
tar xvzf pyOSC-0.3.5b-5294.tar.gz
cd pyOSC-0.3.5b-5294
sudo python setup.py install

Again, the above is untested, but the idea is to:

1. Download the library
2. Unzip it
3. Navigate to the unzipped folder
4. Install it via sudo python setup.py install

Take C++ program output as Java input

Looks like you have all the Java and C++ code already in place. The missing puzzle piece seems to be the shell redirection. You should consult the manual of the shell you are using and look for redirection. In case you are using bash shell the redirection should look like this:

$ ./runActivities.sh | java -jar program.jar arg1 arg2 arg3

piping strings from program output

Here is a solution (edited: it doesn't append the two lines into one line), but for the second part the program.exe will have to be stopped and restarted to change the output filename.

This uses a helper batch file called repl.bat - download from: https://www.dropbox.com/s/qidqwztmetbvklt/repl.bat

Place repl.bat in the same folder as the batch file or in a folder that is on the path.

program.exe -flags | repl "^(cmd|prc): " "" a >output.txt

---

Related Topics