Read from a Log File as It's Being Written Using Python

Read from a log file as it's being written using python

You could try with something like this:

import time

while 1:
    where = file.tell()
    line = file.readline()
    if not line:
        time.sleep(1)
        file.seek(where)
    else:
        print line, # already has newline

Example was extracted from here.

Reading from a frequently updated file

I would recommend looking at David Beazley's Generator Tricks for Python, especially Part 5: Processing Infinite Data. It will handle the Python equivalent of a tail -f logfile command in real-time.

# follow.py
#
# Follow a file like tail -f.

import time
def follow(thefile):
    thefile.seek(0,2)
    while True:
        line = thefile.readline()
        if not line:
            time.sleep(0.1)
            continue
        yield line

if __name__ == '__main__':
    logfile = open("run/foo/access-log","r")
    loglines = follow(logfile)
    for line in loglines:
        print line,

Reading a dynamically updated log file via readline

For your specific use case, the explanation is that Vim uses a write-to-temp strategy. This means that all writing operations are performed on a temporary file.

On the contrary, your scripts reads from the original file, so it does not see any change on it.

---

To further test, instead of Vim, you can try to directly write on the file using:

echo "Hello World" >> myfile.log

You should see the new line from python.

Python read log files and get lines containing specific words

This should get you started nicely:

infile = r"D:\Documents and Settings\xxxx\Desktop\test_log.txt"

important = []
keep_phrases = ["test",
              "important",
              "keep me"]

with open(infile) as f:
    f = f.readlines()

for line in f:
    for phrase in keep_phrases:
        if phrase in line:
            important.append(line)
            break

print(important)

It's by no means perfect, for example there is no exception handling or pattern matching, but you can add these to it quite easily. Look into regular expressions, that may be better than phrase matching. If your files are very big, read it line by line to avoid a MemoryError.

Input file:

This line is super important!
don't need this one...
keep me!
bla bla
not bothered
ALWAYS include this test line

Output:

['This line is super important!\n', 'keep me!\n', 'ALWAYS include this test line']

Note: This is Python 3.3.

Python can't read log file correctly unless I paste its content in a new text file

You can't read the file because the it's encoded in UTF-16, you can tell by the first characters which are the BOM. 0xff is part of the BOM for UTF-16. So when reading just add encoding='utf16' (or use codecs.open with utf16 in python2)

reading log files in Python and outputing specific text

Initialize a flag before the loop:

in_error = False

Then switch it on and off as needed:

if line.startswith('Error'):
    in_error = True
elif line.startswith('End of Error'):
    print(line)
    in_error = False
if in_error:
    print(line)

Create a log file

You can use the logging module to accomplish this.

At the very easiest level, it will be set up like this:

logging.basicConfig(filename="logfilename.log", level=logging.INFO)

There are a number of different levels that you can use to write to the file, such as:

logging.info('your text goes here')
logging.error('your text goes here')
logging.debug('your text goes here')

You can use these lines anywhere that you want to log to the file. If you want to replace the console printing with logging all together, just replace the print lines with logging.info(.......)

For more info on the topic, such as more configurable options (such as timestamps), check the docs (python 3): https://docs.python.org/3/library/logging.html

---

Related Topics