Python Parsing Log File to Extract Events in Real Time

Python parsing log file to extract events in real time

C programs usually seek to the current position to clear any “end of file” flags. But as @9000 correctly pointed out, python apparently takes care of this, so you can read from the same file repeatedly even if it has reached end of file.

You might have to take care of incomplete lines, though. If your application writes its log in pieces, then you want to make sure that you handle whole lines, and not those pieces. The following code will accomplish that:

f = open('some.log', 'r')
while True:
    line = ''
    while len(line) == 0 or line[-1] != '\n':
        tail = f.readline()
        if tail == '':
            time.sleep(0.1)          # avoid busy waiting
            # f.seek(0, io.SEEK_CUR) # appears to be unneccessary
            continue
        line += tail
    process(line)

Parse a custom log file in python

You don't need to be that precise with your regex:

import re

log_pattern = re.compile(r"([0-9\-]*)T([0-9\-:.+]*)\s*\[([^]]*)\](.*)")

with open(name, "r") as f:
  for line in f:
      match = log_pattern.match(line)
      if not match:
        continue
      grps = match.groups()
      print("Log line:")
      print(f"  date:{grps[0]},\n  time:{grps[1]},\n  type:{grps[2]},\n  text:{grps[3]}")

You could even imagine being less precise than that, for example r"(.*)T([^\s]*)\s*\[([^]]*)\](.*)" works too. Here is a nice tool to use to test regular expressions: regex101.

Related Topics