Reading from a Frequently Updated File

Reading from a frequently updated file

I would recommend looking at David Beazley's Generator Tricks for Python, especially Part 5: Processing Infinite Data. It will handle the Python equivalent of a tail -f logfile command in real-time.

# follow.py
#
# Follow a file like tail -f.

import time
def follow(thefile):
thefile.seek(0,2)
while True:
line = thefile.readline()
if not line:
time.sleep(0.1)
continue
yield line

if __name__ == '__main__':
logfile = open("run/foo/access-log","r")
loglines = follow(logfile)
for line in loglines:
print line,

Reading entire file data from a constantly updated file in Python

The for loop will read until it hits the current end of the file and then terminate. Maybe do something like this:

#!/usr/bin/env python                                                           
import os
import sys
import time

def process_line(line):
print(line.rstrip("\n"))

def process_file(f):
for line in f:
process_line(line)

def tail(path):
old_size = 0
pos = 0
while True:
new_size = os.stat(path).st_size
if new_size > old_size:
with open(path, "U") as f:
f.seek(pos)
process_file(f)
pos = f.tell()
old_size = new_size
time.sleep(1)

if __name__ == "__main__":
tail(sys.argv[1])

Of course, this assumes the file doesn't roll and get its size reset to zero.

Javascript - Reading from a frequently updated file

If you're going to use Node.js I recommend you use node-tail.

It allows you to "tail" the file in a straight forward manner. According to the documentation this is how you'd do it:

Tail = require('tail').Tail;

tail = new Tail("fileToTail");

tail.on("line", function(data) {
console.log(data);
});

tail.on("error", function(error) {
console.log('ERROR: ', error);
});

Also, you could use fs.watchFile which comes with Node.js.

Reading a dynamically updated log file via readline

For your specific use case, the explanation is that Vim uses a write-to-temp strategy. This means that all writing operations are performed on a temporary file.

On the contrary, your scripts reads from the original file, so it does not see any change on it.


To further test, instead of Vim, you can try to directly write on the file using:

echo "Hello World" >> myfile.log

You should see the new line from python.



Related Topics



Leave a reply



Submit