Python Serial: How to Use the Read or Readline Function to Read More Than 1 Character At a Time

readline() in pySerial sometimes captures incomplete values being streamed from Arduino serial port

The problem definitely seems to be caused by very fast reads where the data is read when the serial output from the Arduino hasn't finished sending full data.

Now with this fix, the pySerial will be able to receive the complete data and no data is missed. The main benefit is that it can be used for any type of data length and the sleep time is quite low.

I have fixed this issue with code below.

# This code receives data from serial device and makes sure 
# that full data is received.

# In this case, the serial data always terminates with \n.
# If data received during a single read is incomplete, it re-reads
# and appends the data till the complete data is achieved.

import serial
import time
ser = serial.Serial(port='COM4',
                    baudrate=115200,
                    timeout=0)

print("connected to: " + ser.portstr)

while True:                             # runs this loop forever
    time.sleep(.001)                    # delay of 1ms
    val = ser.readline()                # read complete line from serial output
    while not '\\n'in str(val):         # check if full data is received. 
        # This loop is entered only if serial read value doesn't contain \n
        # which indicates end of a sentence. 
        # str(val) - val is byte where string operation to check `\\n` 
        # can't be performed
        time.sleep(.001)                # delay of 1ms 
        temp = ser.readline()           # check for serial output.
        if not not temp.decode():       # if temp is not empty.
            val = (val.decode()+temp.decode()).encode()
            # requrired to decode, sum, then encode because
            # long values might require multiple passes
    val = val.decode()                  # decoding from bytes
    val = val.strip()                   # stripping leading and trailing spaces.
    print(val)

How to reduce time spent by 'readline()' from serial data

I have a couple of suggestions for you. I write Windows applications that use a serial port and I use a different approach - I assume the principles would be the same across all OS's. I create and open the port first, at the beginning of the program, and leave it open. It's good practice to close the port before your program exists but that's not really necessary, since the OS will clean up afterwards.

But your code will create and initialize the port each time you call the function. You're not explicitly closing it when you're done; perhaps you can get away with that because the port object gets garbage collected. You are trusting the serial library to close the port properly at the OS level before you try to open it again. In any case, if there is overhead in creating the port object, why not incur it once and be done with it?

You don't need to create a TextIOWrapper at all, let alone a bi-directional one. You're wondering if it's the reason for your performance issues, so why not get rid of it? The serial port library has all the functionality you need: check out the read_until function.

I think you ought to start with a framework something like this. I can't run and test this program, so it's a schematic only. I have stripped out all the error handling code. One small issue is that serial ports operate on bytes and you have to convert that to a string.

ser = serial.Serial('/dev/tty.usbserial-00002014', 115200, timeout=15)
def bimu_get_gyroscope_raw():
    while True:
        ser.flushInput()
        b = ser.read_until('\r')
        s = str(b, encoding='latin1')  # convert to str
        if a.startswith('S,'):
            line = s.split(',')
            if len(line)==12: 
                return dict(x = float(line[1]),
                            y = float(line[2]),
                            z = float(line[3]))

I have made ser a global but you could also pass it to the function as an argument.

Keep in mind how serial ports work on a modern OS. You are never reading the characters directly from the hardware - the OS is doing that for you, and placing the characters in an input buffer. When you "read" from the port you are actually retrieving any characters from the buffer, or waiting for their arrival. What you observe - a long delay followed by a rapid succession of lines of data - could be explained by the gyroscope hardware doing nothing for several seconds, and then producing a burst of data that's more than one line long. I don't know how your gyroscope works so I can't say that this is really the case.

The PySerial implementation is actually a wrapper around a set of operating system calls. The Python overhead is very minimal, and much of it is error-handling code. I am sure you will be able to receive thousands of characters per second using Python - I do it all the time. Three seconds is close to eternity on a modern PC. There must be another explanation for it. Don't think for a moment that Python is your bottleneck.

Timing events by looking at the screen and clicking a stopwatch is clumsy. Look at the Python time package. You could simply print the value of time.time() in each of your print statements and put away your chronometer.

You can test the data gathering part of the implementation independently. Just strip out the logic to parse the data, and stay in the while loop forever. Print the data along with time stamps for each received line. If you have another instrument that talks to a serial port you can isolate the performance of the instrument from the performance of the software.

Finally, what event causes the gyroscope to make a data transmission? Is it one of those instruments that just periodically broadcasts its data, or do you have to send it some command to request the data? If the former and the broadcasts are every three seconds, the mystery is solved; likewise if it's the latter and the latency in the response is three seconds. I can imagine that some such thing might be the case, since the instrument will have to read some sensors and translate the results to a character string. You haven't shown us the whole program or told us how the instruments works, so this is just guesswork.

PySerial's readlines() consumes 25x times as much CPU time as read()

My guess is that readlines and readline busily poll the serial line for new characters in order to fulfill your request to get a full line (or lines), whereas .read will only read and return when there indeed is new data. You'll probably have to implement buffering and splitting to lines yourself (code untested since I don't have anything on a serial line right now :-) ):

import serial


def read_lines(s, sep=b"\n"):
    buffer = b""
    while True:
        buffer += s.read(1000)
        while sep in buffer:
            line, _, buffer = buffer.partition(sep)
            yield line


s = serial.Serial("/dev/ttyACM0", 9600)

for line in read_lines(s):
    print(line)

---

Related Topics