Force Netcat to Send Messages Immediately (Without Buffering)

Force netcat to send messages immediately (without buffering)

You can use the stdbuf command:

stdbuf -o0 python script.py | stdbuf -i0 nc -lk 9999

or the -u Unbuffered option for python:

python -u script.py | stdbuf -i0  nc -lk 9999

I/O buffering is, unless a program explicitly handles it on it's own, handled by the libc. stdbuf influences that behaviour. Further reading man stdbuf.

Increase netcat read buffer from 4KB to 10KB

Quoting the variables fixed the issue.

The solution:

eol=$'[^\n]*' # needed for bash regex
if [[ "$input" =~ ^m\ [0-9]+\ [0-9]+\ [0-9]+${eol} ]];
    then
      base=$(echo "$input" | awk {'print $2'})
      exp=$(echo "$input" | awk {'print $3'})
      mod=$(echo "$input" | awk {'print $4'})
      modPowResult=$(echo "(${base}^${exp})%${mod}" | (export BC_LINE_LENGTH=0; bc))
      printf "%s." "$modPowResult" # echo would work here, too
fi

Cyrus' comment set me on track to solve the issue. I don't know enough about bash to understand it, but when quoting the $input throughout the script solved my issue.

The topics that came up when I searched for these kind of things (stdbuf, socat, bash/kernel buffer settings etc., also mentioned in this answer) where not the culprit here.

Socket not completely filling receive buffer

That's quite to be expected.

A single read can only read as much data as has been received so far (or otherwise it would need to bend time and space). The other end may also not have sent off everything they have to send by the time you issue read, so a single read won't be able to read everything either (because, well, it may not have been sent).

Since TCP is a stream protocol, you will need to devise a way to know how many bytes to expect for a single "message" from the other end, and read exactly that many bytes so you know you've gotten everything the other end has to say; I'd recommend some sort of Type-Length-Value scheme, even if you didn't need more than 1 type at present.

Turn off buffering in pipe

You can use the unbuffer command (which comes as part of the expect package), e.g.

unbuffer long_running_command | print_progress

unbuffer connects to long_running_command via a pseudoterminal (pty), which makes the system treat it as an interactive process, therefore not using the 4-kiB buffering in the pipeline that is the likely cause of the delay.

For longer pipelines, you may have to unbuffer each command (except the final one), e.g.

unbuffer x | unbuffer -p y | z

Send data over telnet without pressing enter

You should be able to do this with telnet option negotiation. The protocol defaults to half-duplex mode, and at a minimum for an interactive session, the server should negotiate the suppress go ahead option and echo option.

At the bare minimum you could just spit out ff fb 01 ff fb 03 (will echo, will suppress-go-ahead) at the begining of the session, then reply to any ff fd 01 (do echo) with ff fb 01 (will echo) and reply to any ff fd 03 (do suppress-go-ahead) with ff fb 03 (will suppress-go-ahead).

Edit to add that the linemode negotiation mentioned by Ben Jackson is a better answer. Suppress go-ahead won't be enough for most clients connecting on ports other than 23.

However I think the other problem you're running into is that Java is sending Unicode characters. For example, when you say (char)0xff, Java assumes you're referring to UTF-16 character U+00ff which is ÿ. It's probably sending it over the socket using UTF-8 encoding, so the telnet client sees two bytes: c3 bf which it passes on and displays as ÿ.

What you can do is explicitly tell Java to use ISO-8859-1 encoding. For example, you may have been doing something like this before:

out = new PrintStream(connection.getOutputStream());
out.print((char)0xff);  // sends 0xc3 0xbf
out.print((char)0xfb);  // sends 0xc3 0xbb
out.print((char)0x01);  // sends 0x01
out.flush();

Instead, you can use the OutputStreamWriter to specify the encoding you want:

out = new OutputStreamWriter(connection.getOutputStream(), "ISO-8859-1");
out.write((char)0xff);  // sends 0xff
out.write((char)0xfb);  // sends 0xfb
out.write((char)0x01);  // sends 0x01
out.flush();

Forcing a TCP socket to flush in Java

Try this create a simple server :

public static void main(String[] args) throws IOException {
    ServerSocket svr = new ServerSocket(1234);
    Socket s = svr.accept();
    byte b4[] = new byte[4];
    new DataInputStream(s.getInputStream()).readFully(b4);
    s.getOutputStream().write(b4);
}

Run your client code, but measure the time:

public static void main(String[] args) throws IOException {
    Socket sock = new Socket("localhost", 1234);
    OutputStream output = sock.getOutputStream();
    long t1 = System.currentTimeMillis();
    output.write(new byte[]{(byte) 0xde, (byte) 0xad, (byte) 0xbe, (byte) 0xef});
    byte b4[] = new byte[4];
    new DataInputStream(sock.getInputStream()).readFully(b4);
    long t2 = System.currentTimeMillis();
    System.out.println("t2-t1="+(t2-t1));
}

When I did this I got an output of 15 milliseconds. This is round trip including both read and write. If you run this and get something dramatically different, then you probably need to fix something in your network configuration. If this gives you about 15 milliseconds you need to look at the difference between this code and your code.The setTcpNoDelay might have an effect but for for me it wasn't noticeable.

C# Read from SslStream continuously (long connection, last for up to days) and Efficiently without infinite loop

Here's my shot at it. Instead of looping forever, I chose recursion. This method will return immediately but will fire an event when EOF is hit and continue to keep reading:

public static void ReadFromSSLStreamAsync(
    SslStream sslStream,
    Action<string> result,
    Action<Exception> error,
    StringBuilder stringBuilder = null)
{
    const string EOFToken = "<EOF>";

    stringBuilder = stringBuilder ?? new StringBuilder();
    var buffer = new byte[4096];

    try
    {
        sslStream.BeginRead(buffer, 0, buffer.Length, asyncResult =>
        {
            // Read all bytes avaliable from stream and then
            // add them to string builder
            {
                int bytesRead;
                try
                {
                    bytesRead = sslStream.EndRead(asyncResult);
                }
                catch (Exception ex)
                {
                    error?.Invoke(ex);
                    return;
                }

                // Use Decoder class to convert from bytes to
                // UTF8 in case a character spans two buffers.
                var decoder = Encoding.UTF8.GetDecoder();
                var buf = new char[decoder.GetCharCount(buffer, 0, bytesRead)];
                decoder.GetChars(buffer, 0, bytesRead, buf, 0);
                stringBuilder.Append(buf);
            }

            // Find the EOFToken, if found copy all data before the token
            // and send it to event, then remove it from string builder
            {
                int tokenIndex;
                while((tokenIndex = stringBuilder.ToString().IndexOf(EOFToken)) != -1)
                {
                    var buf = new char[tokenIndex];
                    stringBuilder.CopyTo(0, buf, 0, tokenIndex);
                    result?.Invoke(new string(buf));
                    stringBuilder.Remove(0, tokenIndex + EOFToken.Length);
                }
            }

            // Continue reading...
            ReadFromSSLStreamAsync(sslStream, result, error, stringBuilder);
        }, null);
    }
    catch(Exception ex)
    {
        error?.Invoke(ex);
    }
}

You could call it as so:

ReadFromSSLStreamAsync(sslStream, sslData =>
{
    Console.WriteLine($"Finished: {sslData}");
}, error =>
{
    Console.WriteLine($"Errored: {error}");
});

It's not TaskAsync, so you don't have to await on it. But it is asynchronous so your thread can go on to do other things.

---

Related Topics