Bash: Best Architecture for Reading from Two Input Streams

BASH: Best architecture for reading from two input streams

Bash is not the right tool for this. This problem is typically solved using the select(2) or poll(2) system calls, which allow you to wait on multiple file descriptors simultaneously without spinning. Bash does not have an interface to either of those.

I'd recommend using a scripting language such as Python or Perl (whatever you're comfortable with, really) that provides an interface with select or poll (e.g. Python's select module).

Pthreads as a standard solution for simultaneous console input in UNIX?

You can take a look at the select(2) system call and read about multiplexing IO.

bash - redirect specific output from 2nd script back to stdin of 1st program?

An additional expect-like software tool would be empty!

http://empty.sourceforge.net

Linux non-blocking fifo (on demand logging)

Inspired by your question I've written a simple program that will let you do this:

$ myprogram 2>&1 | ftee /tmp/mylog

It behaves similarly to tee but clones the stdin to stdout and to a named pipe (a requirement for now) without blocking. This means that if you want to log this way it may happen that you're gonna lose your log data, but I guess it's acceptable in your scenario.
The trick is to block SIGPIPE signal and to ignore error on writing to a broken fifo. This sample may be optimized in various ways of course, but so far, it does the job I guess.

/* ftee - clone stdin to stdout and to a named pipe 
(c) racic@stackoverflow
WTFPL Licence */

#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <sys/types.h>
#include <sys/stat.h>
#include <fcntl.h>
#include <errno.h>
#include <signal.h>
#include <unistd.h>

int main(int argc, char *argv[])
{
    int readfd, writefd;
    struct stat status;
    char *fifonam;
    char buffer[BUFSIZ];
    ssize_t bytes;
    
    signal(SIGPIPE, SIG_IGN);

    if(2!=argc)
    {
        printf("Usage:\n someprog 2>&1 | %s FIFO\n FIFO - path to a"
            " named pipe, required argument\n", argv[0]);
        exit(EXIT_FAILURE);
    }
    fifonam = argv[1];

    readfd = open(fifonam, O_RDONLY | O_NONBLOCK);
    if(-1==readfd)
    {
        perror("ftee: readfd: open()");
        exit(EXIT_FAILURE);
    }

    if(-1==fstat(readfd, &status))
    {
        perror("ftee: fstat");
        close(readfd);
        exit(EXIT_FAILURE);
    }

    if(!S_ISFIFO(status.st_mode))
    {
        printf("ftee: %s in not a fifo!\n", fifonam);
        close(readfd);
        exit(EXIT_FAILURE);
    }

    writefd = open(fifonam, O_WRONLY | O_NONBLOCK);
    if(-1==writefd)
    {
        perror("ftee: writefd: open()");
        close(readfd);
        exit(EXIT_FAILURE);
    }

    close(readfd);

    while(1)
    {
        bytes = read(STDIN_FILENO, buffer, sizeof(buffer));
        if (bytes < 0 && errno == EINTR)
            continue;
        if (bytes <= 0)
            break;

        bytes = write(STDOUT_FILENO, buffer, bytes);
        if(-1==bytes)
            perror("ftee: writing to stdout");
        bytes = write(writefd, buffer, bytes);
        if(-1==bytes);//Ignoring the errors
    }
    close(writefd); 
    return(0);
}

You can compile it with this standard command:

$ gcc ftee.c -o ftee

You can quickly verify it by running e.g.:

$ ping www.google.com | ftee /tmp/mylog

$ cat /tmp/mylog

Also note - this is no multiplexer. You can only have one process doing $ cat /tmp/mylog at a time.

Parsing numbers from standard input stream using sscanf in C

Ethan, some times you just need to slow down to help C fall into place. I your case you have data that evidently will contain a classification character in the beginning row of each line. That can be following by one or two number you need to read and handle accordingly. Your problem is defined by your dataset.

Thankfully, you can utilize line-oriented input functions fgets or getline and then a single call to sscanf for each like to handle all cases. After reading your line, you pass it to a sscanf call similar to the following:

rtn = sscanf (buf, "%c %d %d", &c, &v1, &v2);  /* always check return */

Preserving the return (i.e. the number of successful conversions that took place based upon your format string), you can began answering logical questions about the data you can expect. However before blindly comparing whether there are three, or two, or less conversions, you can answer simple questions just by looking at the first char in your buffer. If you require a lowercase letter to begin a valid line, then all you need to do is dereference your buffer to get the first char and then check if it is between 'a' - 'z'. If not, it's not a valid line. Similarly, if you check the first char and find it 0 (the terminating char) or '\n', then you know the line is empty and you need not bother with it any more. Those two test will filter out all the crud leaving you with at least lines that may match your criteria. For instance simply checking:

    if (rtn == 0) { /* no successful conversions took place */
        fprintf (stderr, "error: no values parsed from line.\n");
        continue;
    }

    if (!*buf || *buf == '\n') {    /* check if buf was empty line */
        fprintf (stderr, "error: line is empty or contians only newline.\n");
        continue;
    }

    if (*buf < 'a' || 'z' < *buf) {  /* check first char not a-z */
        fprintf (stderr, "error: no lowercase char beginning line.\n");
        continue;
    }

gives you confidence that at least the line you operated on had a reasonable chance of matching what you are looking for.

Now you can either do and if ... else if .... else if .... (which is perfectly fine), or since you know you are primarily concerned with checking values 1-3, a switch statement makes sense for handling the remaining lines. For example, you could do something like the following to parse the value from those line that warrant a look:

    switch (rtn) {  /* switch on number of successful conversions */

        case 3:     /* three successful conversions */
            printf ("all values: '%c'  %d  %d\n", c, v1, v2);
            break;

        case 2:     /* two successful conversions */
            printf ("two values: '%c'  %d\n", c, v1);
            break;

        default:    /* one or less (need at least two */
            fprintf (stderr, "error: no character and value on line.\n");
    }

Putting all those pieces together, you could do something similar to:

#include <stdio.h>

#define MAXC 256

int main (void) {

    char buf[MAXC] = "";

    while (fgets (buf, MAXC, stdin))    /* read each line into buf */
    {
        char c;
        int v1, v2, rtn;     /* vars for values and sscanf return */

        rtn = sscanf (buf, "%c %d %d", &c, &v1, &v2); 

        if (rtn == 0) { /* no successful conversions took place */
            fprintf (stderr, "error: no values parsed from line.\n");
            continue;
        }

        if (!*buf || *buf == '\n') {    /* check if buf was empty line */
            fprintf (stderr, "error: line is empty or contians only newline.\n");
            continue;
        }

        if (*buf < 'a' || 'z' < *buf) {  /* check first char not a-z */
            fprintf (stderr, "error: no lowercase char beginning line.\n");
            continue;
        }

        switch (rtn) {  /* switch on number of successful conversions */

            case 3:     /* three successful conversions */
                printf ("all values: '%c'  %d  %d\n", c, v1, v2);
                break;

            case 2:     /* two successful conversions */
                printf ("two values: '%c'  %d\n", c, v1);
                break;

            default:    /* one or less (need at least two */
                fprintf (stderr, "error: no character and value on line.\n");
        }
    }

    return 0;
}

Example Input Files

$ cat dat/input.txt
a -1109180 699692587
a 1213834231 -226769626
c 994957275 2082945813
e 1213834231

And a little messier file:

$ cat dat/rows.txt
a -1109180 699692587
a quick brown fox jumps over the lazy dog
a 1213834231 -226769626

c 994957275 2082945813
227 isn't good
e 1213834231

Should provide a good test to see if the code does what you need it to do. Running your data through the code provides:

Example Use/Output

$ /bin/sscanf_mixed <dat/input.txt
all values: 'a'  -1109180  699692587
all values: 'a'  1213834231  -226769626
all values: 'c'  994957275  2082945813
two values: 'e'  1213834231

The messier file output:

$ ./bin/sscanf_mixed <dat/rows.txt
all values: 'a'  -1109180  699692587
error: no character and value on line.
all values: 'a'  1213834231  -226769626
error: line is empty or contians only newline.
all values: 'c'  994957275  2082945813
error: no lowercase char beginning line.
two values: 'e'  1213834231

As you can see, both input files provided the same number of conversions, the second simply noting the error or problem that took place when non-conforming line are encountered.

Look things over and let me know if you have any questions. There are many ways to approach the logic of the problem, but which ever way you choose, you want to make sure you build all input routines flexible enough to handle those conditions (empty line, etc...) that you should reasonably expect to encounter in real world use.

Reading USB input stream from ruby.

I don't fully understand the solution I found, but it will probably help anyone faced with the same problem, so here goes.

It appears to be a consequence of the 32-bit architecture of Linux Mint that it will not allow the file.read method to work with an argument that's not a power of 2 (e.g. 8, 16, 32). I do not know why this is.

However, re-installing using the 64-bit version of Mint, this works as I was previously seeing on my KDE install (which, in retrospect, was probably a 64-bit architecture).

How do I peek at the first two bytes in an InputStream?

For a general InputStream, I would wrap it in a BufferedInputStream and do something like this:

BufferedInputStream bis = new BufferedInputStream(inputStream);
bis.mark(2);
int byte1 = bis.read();
int byte2 = bis.read();
bis.reset();
// note: you must continue using the BufferedInputStream instead of the inputStream

---

Related Topics