Parsing Complete Messages from Serial Port

parsing complete messages from serial port

You can break the read into three parts. Find the start of a message. Then get the LENGTH. Then read the rest of the message.

// Should probably clear these in case data left over from a previous read
input_buffer[0] = input_buffer[1] = 0;

// First make sure first char is 0xB5
do {
    n = read(fd, input_buffer, 1); 
} while (0xB5 != input_buffer[0]);

// Check for 2nd sync char
n = read(fd, &input_buffer[1], 1);

if (input_buffer[1] != 0x62) {
     // Error
     return;
}

// Read up to LENGTH
n = read(fd, &input_buffer[2], 4); 

// Parse length
//int length = *((int *)&input_buffer[4]);
// Since I don't know what size an int is on your system, this way is better
int length = input_buffer[4] | (input_buffer[5] << 8);

// Read rest of message
n = read(fd, &input_buffer[6], length);

// input_buffer should now have a complete message

You should add error checking...

How to read serial port communication into buffer and parse out complete messages

You have to think of Serial Port communications as streaming data. Any time you receive data, you have to expect that it may be a complete message, only a partial message, or multiple messages. It all depends how fast the data is coming in and how fast you application is able to read from the queue. Therefore, you are right in thinking you need a buffer. However, what you may not be realizing yet, is that there is no way to know, strictly via, the Serial Port, where each message begins and ends. That has to be handled via some agreed upon protocol between the sender and the receiver. For instance, many people use the standard start-of-text (STX) and end-of-text (ETX) characters to indicate the beginning and ending of each message send. That way, when you receive the data, you can tell when you have received a complete message.

For instance, if you used STX and ETX characters, you could make a class like this:

Public Class DataBuffer
    Private ReadOnly _startOfText As String = ASCII.GetChars(New Byte() {2})
    Private ReadOnly _endOfText As String = ASCII.GetChars(New Byte() {4})

    Public Event MessageReceived(ByVal message As String)
    Public Event DataIgnored(ByVal text As String)

    Private _buffer As StringBuilder = New StringBuilder

    Public Sub AppendText(ByVal text As String)
        _buffer.Append(text)
        While processBuffer(_buffer)
        End While
    End Sub

    Private Function processBuffer(ByVal buffer As StringBuilder) As Boolean
        Dim foundSomethingToProcess As Boolean = False
        Dim current As String = buffer.ToString()
        Dim stxPosition As Integer = current.IndexOf(_startOfText)
        Dim etxPosition As Integer = current.IndexOf(_endOfText)
        If (stxPosition >= 0) And (etxPosition >= 0) And (etxPosition > stxPosition) Then
            Dim messageText As String = current.Substring(0, etxPosition + 1)
            buffer.Remove(0, messageText.Length)
            If stxPosition > 0 Then
                RaiseEvent DataIgnored(messageText.Substring(0, stxPosition))
                messageText = messageText.Substring(stxPosition)
            End If
            RaiseEvent MessageReceived(messageText)
            foundSomethingToProcess = True
        ElseIf (stxPosition = -1) And (current.Length <> 0) Then
            buffer.Remove(0, current.Length)
            RaiseEvent DataIgnored(current)
            foundSomethingToProcess = True
        End If
        Return foundSomethingToProcess
    End Function

    Public Sub Flush()
        If _buffer.Length <> 0 Then
            RaiseEvent DataIgnored(_buffer.ToString())
        End If
    End Sub
End Class

I should also mention that, in communication protocols, it is typical to have a checksum byte by which you can determine if the message got corrupted during its transmission between the sender and the receiver.

Parsing/formatting data from serial port - C#

The problem is, as you may have guessed, that the event DataReceived is raised as soon as data has been received over the serial port. There may not be a complete record there; the SerialPort object has no clue what you consider to be "enough" data to be significant, or workable.

The usual solution is to maintain another "buffer" of received data, containing any data you have recognized as incomplete. When data comes in over the port and your event fires, it should first take what's in the buffer and append it to what you have already received. Then, you should start at the beginning of this data buffer and inspect the received data, looking for known patterns of atomic "chunks" of data that have meaning to you; for instance, say the first thing you receive is "ID: 12". You take this, put it in the buffer, then scan the buffer looking for a pattern defined by a regex "ID: \d*? ". Because the trailing space is not present in your buffer, your pattern fails to find anything of meaning, and so you now know you haven't received a full message.

Then, on the next raising of the DataReceived event, you pull "453 Sta" out of the serial buffer. You append it to what you already have and get "ID:12453 Sta", and when you apply the regex, you get the match "ID: 12345 ". You pass this into a method for further processing (display to the console, maybe), and remove the same string from the front of the buffer, leaving "Sta". Scanning again you don't find anything else of interest, so you leave what you have, and the cycle repeats aws data continues to come in. Obviously, you'll be testing more patterns than just the ID pattern; you may search for an entire "string" you expect to receive, such as "ID: \d*? State: \w{2} ". You may even keep the data in your buffer until you have both strings for a record: "ID:\d*? State:\w{2} Zip:\d{5} StreetType:\w*? ".

Either way, you will need to identify whether the data being received is either reliably "fixed-length" (meaning each string of a particular type will always have the same number of bytes or characters), or reliably "delimited" (meaning there will be some character or character combination that always separates significant elements of data). If neither of these apply, it may be very difficult to parse the data into single-field chunks.

Here's a sample based on what you have already:

private static StringBuilder receiveBuffer = new StringBuilder();

private static void Port_DataReceived(object sender, SerialDataReceivedEventArgs e)
{

    SerialPort spL = (SerialPort) sender;
    int bufSize = 20;
    Byte[] dataBuffer = new Byte[bufSize];
    Console.WriteLine("Data Received at"+DateTime.Now);
    Console.WriteLine(spL.Read(dataBuffer, 0, bufSize));
    string s = System.Text.ASCIIEncoding.ASCII.GetString(dataBuffer);
    //here's the difference; append what you have to the buffer, then check it front-to-back
    //for known patterns indicating fields
    receiveBuffer.Append(s);

    var regex = new Regex(@"(ID:\d*? State:\w{2} Zip:\d{5} StreetType:\w*? )");
    Match match;
    do{
       match = regex.Match(receiveBuffer.ToString());
       if(match.Success)
       {
          //"Process" the significant chunk of data
          Console.WriteLine(match.Captures[0].Value);
          //remove what we've processed from the StringBuilder.
          receiveBuffer.Remove(match.Captures[0].Index, match.Captures[0].Length);
       }
    } while (match.Success);
}

Fast approach to read and parse serial-data continuously

A very simple way to get ahead is to stop trying to make it faster. There is no point, serial port data rates are very, very low and modern computers are very, very fast. Your Read() call only ever returns a single byte, rarely 2.

Note that this is hard to see, when you debug and single-step through the code then you'll artificially slow down your program a great deal. Allowing more bytes to be received and thus more of them getting returned by the Read() call. But this doesn't happen when the program runs at normal speed.

So use SerialPort.BaseStream.ReadByte() instead. Makes the code very simple.

---

Related Topics