Simplest Way to Send Raw Byte-Arrays Using Ruby's Tcpsocket-Class

simplest way to send raw Byte-arrays using Ruby's TCPSocket-Class

Try using a "*" after the format directive to eat all the elements in the list:

s.write [0x03, 0x00, 0x00, 0x16,
         0x11, 0xE0, 0x00, 0x00, 0x00, 
         0x01, 0x00, 0xC1, 0x02, 0x02, 
         0x02, 0xC2, 0x02, 0x02, 0x02, 
         0xC0, 0x01, 0x0A ].pack('C*')

There are lots of neat tricks possible with string#format so it's worth studying the documentation.

Is there a simple way to send a packet over a raw socket using Ruby's Socket class?

This will do it:

require 'socket'

interface = 'lo'         # loopback interface
interface_index = 0x8933 # SIOCGIFINDEX
geonet_frame = "\x00\x1f\xc6\x51\x07\x07\x07\x07\x07\x07\x07\x07\x07\x07\xc6\x51\x07\x07\x07\x07\x07\x07\xef\x06\x07\x35\x97\x00\x24\x8c\x7a\xdf\x6f\x08\x00\x45\x00\x00\x3d\xf3\x7f\x40\x00\x40\x11\x30\xc6\x0a\x01\x01\x68\x0a\x01\x01\x01\x99\x80\x00\x35\x00\x29\x16\xa5\x01\x76\x01\x00\x00\xff\x00\x00\x01\x00\x00\x00"

socket = Socket.new(Socket::AF_PACKET, Socket::SOCK_RAW, Socket::IPPROTO_RAW)
ifreq = [interface.dup].pack 'a32'
socket.ioctl(interface_index, ifreq)
socket.bind([Socket::AF_PACKET].pack('s') + [Socket::IPPROTO_RAW].pack('n') + ifreq[16..20]+ ("\x00" * 12))

socket.send(geonet_frame, 0)

I used AF_PACKET because it matched the python example, but you can replace it with PF_PACKET if your system doesn't support it.

Replacing the geonet_frame variable as Cocoabean suggested also works:

geonet_frame = [ 0x00, 0x1f, 0xc6, 0x51, 0x07, 0x07, 0x07, 0x07, 0x07, 0x07, 0x07, 0x07, 0x07, 0x07, 0xc6, 0x51, 0x07, 0x07, 0x07, 0x07,0x07, 0x07, 0xef, 0x06, 0x07, 0x35, 0x97, 0x00, 0x24, 0x8c, 0x7a, 0xdf, 0x6f, 0x08, 0x00, 0x45, 0x00, 0x00, 0x3d, 0xf3, 0x7f, 0x40, 0x00, 0x40, 0x11, 0x30, 0xc6, 0x0a, 0x01, 0x01, 0x68, 0x0a, 0x01, 0x01, 0x01, 0x99, 0x80, 0x00, 0x35, 0x00, 0x29, 0x16, 0xa5, 0x01, 0x76, 0x01, 0x00, 0x00, 0xff, 0x00, 0x00, 0x01, 0x00, 0x00, 0x00].pack("C*")

Tested with ruby 1.9.3p484 on CentOS 6.6. Hope it helps.

Cheers.

How to simply serialize complex structures and send them over a network in Qt

You can and should leverage QDataStream for that. Instead from deriving from Serializable, simply implement QDataStream & operator<<(QDataStream &, Type & const) and QDataStream & operator>>(QDataStream &, Type &) for each Type you want to serialize.

You'd then simply dump all data to a QByteArray via a QDataStream. You'd transmit the size of the array, followed by its contents. On the receiving end, you receive the size, then the contents, set a datastream on it, and pull the data out. It should be seamless.

The data block separation is handled automatically for you as long as the individual streaming operators are implemented correctly. The choice of a QHash as a means of serialization is unnecessarily limiting - it may be a good choice for some classes, but not for others.

Instead of serializing into a QHash, you should be serializing into a QDataStream. But the operators for that can be free-standing functions, so you don't need to derive from any special interface class for that. The compiler will remind you of any missing operators.

This is a very simple example, working over UDP.

This is a larger example that shows the details of future-proof versioning, and shows serialization of a rather complex data structure - a QAbstractItemModel.

Generally speaking, the serialization of, say, three objects a,b,c might look as follows:

static const QDataStream::Version kDSVersion = QDataStream::Qt_5_5;

void Foo::send() {
  QByteArray buf;
  QDataStream bds(&buf, QIODevice::WriteOnly));
  bds.setVersion(kDSVersion);
  bds << a << b << c;

  QDataStream ds(socket, QIODevice::WriteOnly);
  ds.setVersion(kDSVersion);
  ds << buf; // buffer size followed by data
}

On the receiving end:

void Foo::readyReadSlot() {
  typedef quint32 QBALength;
  if (socket->bytesAvailable() < sizeof(QBALength)) return;
  auto buf = socket->peek(sizeof(QBALength));
  QDataStream sds(&buf);
  // We use a documented implementation detail:
  // See http://doc.qt.io/qt-5/datastreamformat.html
  // A QByteArray is serialized as a quint32 size followed by raw data.
  QBALength size;
  sds >> size;
  if (size == 0xFFFFFFFF) {
    // null QByteArray, discard
    socket.read(sizeof(QBALength));
    return;
  }
  if (socket->bytesAvailable() < size)  return;
  QByteArray buf;
  QDataStream bds(&socket);
  bds.setVersion(kDSVersion);
  bds >> buf;
  QDataStream ds(&buf);
  ds.setVersion(kDSVersion);
  ds >> a >> b >> c;
}

Air-As3 sockets closing socket stop sending information

Please read The ultimate SO_LINGER page, or: why is my tcp not reliable to understand what's going on at the socket API and kernel stack level.

Then follow the advise and put at least some basic application-level protocol on top of raw TCP byte stream - tell the receiver the length of data it is about to receive, send acknowledgment that everything has been read back to the sender.

Why does Java read random amounts from a socket but not the whole message?

If your in is a BufferedReader then you've run into the common problem with buffering more than needed. The default buffer size of BufferedReader is 8192 characters which is approximately the difference between what you expected and what you got. So the data you are missing is inside BufferedReader's internal buffer, converted to characters (I wonder why it didn't break with some kind of conversion error).

The only workaround is to read the first lines byte-by-byte without using any buffered classes readers. Java doesn't provide an unbuffered InputStreamReader with readLine() capability as far as I know (with the exception of the deprecated DataInputStream.readLine(), as indicated in the comments below), so you have to do it yourself. I would do it by reading single bytes, putting them into a ByteArrayOutputStream until I encounter an EOL, then converting the resulting byte array into a String using the String constructor with the appropriate encoding.

Note that while you can't use a BufferedInputReader, nothing stops you from using a BufferedInputStream from the very beginning, which will make byte-by-byte reads more efficient.

Update

In fact, I am doing something like this right now, only a bit more complicated. It is an application protocol that involves exchanging some data structures that are nicely represented in XML, but they sometimes have binary data attached to them. We implemented this by having two attributes in the root XML: fragmentLength and isLastFragment. The first one indicates how much bytes of binary data follow the XML part and isLastFragment is a boolean attribute indicating the last fragment so the reading side knows that there will be no more binary data. XML is null-terminated so we don't have to deal with readLine(). The code for reading looks like this:

    InputStream ins = new BufferedInputStream(socket.getInputStream());
    while (!finished) {
      ByteArrayOutputStream buf = new ByteArrayOutputStream();
      int b;
      while ((b = ins.read()) > 0) {
        buf.write(b);
      }
      if (b == -1)
        throw new EOFException("EOF while reading from socket");
      // b == 0
      Document xml = readXML(new ByteArrayInputStream(buf.toByteArray()));
      processAnswers(xml);
      Element root = xml.getDocumentElement();
      if (root.hasAttribute("fragmentLength")) {
        int length = DatatypeConverter.parseInt(
                root.getAttribute("fragmentLength"));
        boolean last = DatatypeConverter.parseBoolean(
                root.getAttribute("isLastFragment"));
        int read = 0;
        while (read < length) {
          // split incoming fragment into 4Kb blocks so we don't run 
          // out of memory if the client sent a really large fragment
          int l = Math.min(length - read, 4096);
          byte[] fragment = new byte[l];
          int pos = 0;
          while (pos < l) {
            int c = ins.read(fragment, pos, l - pos);
            if (c == -1)
              throw new EOFException(
                      "Preliminary EOF while reading fragment");
            pos += c;
            read += c;
          }
          // process fragment
        }

Using null-terminated XML for this turned out to be a really great thing as we can add additional attributes and elements without changing the transport protocol. At the transport level we also don't have to worry about handling UTF-8 because XML parser will do it for us. In your case you're probably fine with those two lines, but if you need to add more metadata later you may wish to consider null-terminated XML too.

Extract uri parameters from a HTTP connection on a Ruby TCPSocket

If you want to see a very minimal server, here is one. It handles exactly two parameters, and puts the strings in an array. You'll need to do more to handle variable numbers of parameters.

There is a fuller explanation of the server code at https://practicingruby.com/articles/implementing-an-http-file-server.

require "socket"

server = TCPServer.new('localhost', 8080)
loop do
    socket = server.accept
    request = socket.gets

    # Here is the first line of the request. There are others.
    # Your parsing code will need to figure out which are
    # the ones you need, and extract what you want. Rack will do
    # this for you and give you everything in a nice standard form.

    paramstring = request.split('?')[1]     # chop off the verb
    paramstring = paramstring.split(' ')[0] # chop off the HTTP version
    paramarray  = paramstring.split('&')    # only handles two parameters

    # Do something with the URL parameters which are in the parameter array

    # Build a response!
    # you need to include the Content-Type and Content-Length headers
    # to let the client know the size and type of data
    # contained in the response. Note that HTTP is whitespace
    # sensitive and expects each header line to end with CRLF (i.e. "\r\n")

    response = "Hello world!"

    socket.print "HTTP/1.1 200 OK\r\n" +
                 "Content-Type: text/plain\r\n" +
                 "Content-Length: #{response.bytesize}\r\n" +
                 "Connection: close\r\n"

    # Print a blank line to separate the header from the response body,
    # as required by the protocol.
    socket.print "\r\n"
    socket.print response
    socket.close
end

How can I create a two-way SSL socket in Ruby

I'm not familiar with ruby itself, but with the problems of using select with SSL based sockets. SSL sockets behave differently to TCP sockets since SSL data are transferred in frames and not as a data stream, but nevertheless stream semantics are applied to the socket interface.

Let's explain this with an example, first using a simple TCP connection:

• The server sends 1000 bytes in a single write.
• The data will be delivered to the client and put into the in-kernel socket buffer. Thus select will return that data are available.
• The client application reads only 100 bytes first.
• The other 900 bytes will then be kept in the in-kernel socket buffer. The next call of select will again return that data are available, since select looks at the in-kernel socket buffer.

With SSL this is different:

• The server sends 1000 bytes in a single write. The SSL stack will then encrypt these 1000 bytes into a single SSL frame and send this frame to the client.
• This SSL frame will be delivered to the client and put into the in-kernel socket buffer. Thus select will return that data are available.
• Now the client application reads only 100 bytes. Since the SSL frame contains 1000 bytes and since it needs the full frame to decrypt the data, the SSL stack will read the full frame from the socket, leaving nothing in the in-kernel socket buffer. It will then decrypt the frame and return the requested 100 bytes to the application. The rest of decrypted 900 bytes will be kept in the SSL stack in user space.
• Since the in-kernel socket buffer is empty the next call of select will not return that data are available. Thus if the application only deals with select it will not now that there are still unread data, i.e. the 900 bytes kept in the SSL stack.

How to deal with this difference:

• One way is to always try to read at least 32768 data, because this is the maximum size of an SSL frame. This way one can be sure that no data are still kept in the SSL stack (SSL read will not read over SSL frame boundaries).
• Another way is to check the SSL stack for already decrypted data before calling select. Only if no data are kept in the SSL stack select should be called. To check for such "pending data" use the pending method.
• Try to read more data from the non-blocking socket until no more data are available. This way you can be sure that no data are still pending in the SSL stack. But note that this will also do reads on the underlying TCP socket and thus might prefer data on the SSL socket compared to data on other sockets (which are only read after successful select). This is the recommendation you've cited, but I would prefer the others.

---

Related Topics