Watch/Read a Growing Log File

Read from a log file as it's being written using python

You could try with something like this:

import time

while 1:
    where = file.tell()
    line = file.readline()
    if not line:
        time.sleep(1)
        file.seek(where)
    else:
        print line, # already has newline

Example was extracted from here.

PHP: How to read a file live that is constantly being written to

You need to loop with sleep:

$file='/home/user/youfile.txt';
$lastpos = 0;
while (true) {
    usleep(300000); //0.3 s
    clearstatcache(false, $file);
    $len = filesize($file);
    if ($len < $lastpos) {
        //file deleted or reset
        $lastpos = $len;
    }
    elseif ($len > $lastpos) {
        $f = fopen($file, "rb");
        if ($f === false)
            die();
        fseek($f, $lastpos);
        while (!feof($f)) {
            $buffer = fread($f, 4096);
            echo $buffer;
            flush();
        }
        $lastpos = ftell($f);
        fclose($f);
    }
}

(tested.. it works)

Reading from a frequently updated file

I would recommend looking at David Beazley's Generator Tricks for Python, especially Part 5: Processing Infinite Data. It will handle the Python equivalent of a tail -f logfile command in real-time.

# follow.py
#
# Follow a file like tail -f.

import time
def follow(thefile):
    thefile.seek(0,2)
    while True:
        line = thefile.readline()
        if not line:
            time.sleep(0.1)
            continue
        yield line

if __name__ == '__main__':
    logfile = open("run/foo/access-log","r")
    loglines = follow(logfile)
    for line in loglines:
        print line,

In Colab : Reading / Watching logs as they grow

open a ssh into Colab and do the tail -f file.log just as you used to
do.

This option is very comfortable as you can resize your local console window and move it around, being it outside Colab.

Thanks to some people, nowadays is really easy to get there.

Run this on a cell. Code comes from here.

# Install colab_ssh
!pip install colab_ssh --upgrade

from colab_ssh import launch_ssh
launch_ssh('YOUR_NGROK_AUTH_TOKEN', 'SOME_PASSWORD')

This will output something like (as of 2022-02-10)

Collecting colab_ssh
  Downloading colab_ssh-0.3.27-py3-none-any.whl (26 kB)
Installing collected packages: colab-ssh
Successfully installed colab-ssh-0.3.27
Warning: Due to some issues with ngrok on Google Colab, reported in the issue https://github.com/WassimBenzarti/colab-ssh/issues/45, 
we highly recommend that update your code by following this documentation https://github.com/WassimBenzarti/colab-ssh#getting-started
Successfully running X.tcp.ngrok.io:12345
[Optional] You can also connect with VSCode SSH Remote extension using this configuration:

  Host google_colab_ssh
    HostName X.tcp.ngrok.io
    User root
    Port 12345
    

Now, on your local machine (or wherever you want) open a console and type.

ssh -p 12345 root@X.tcp.ngrok.io

After that you are INSIDE Colab and you can do.

tail -f /content/file.log

And watch it grow.

---

After some inactivity the connection may get closed.

Because the host changes frequently, using RSA keys to avoid typing the password doesn't seem like a good solution.

So you may want to install sshpass

# sudo apt-get install sshpass

So you can simply reuse the command for connection after getting disconnected

# sshpass -p <PASSWORD> ssh -p <PORT>  root@X.tcp.ngrok.io

Display the contents of a log file as it is updated

Use a Flask view to continuously read from the file forever and stream the response. Use JavaScript to read from the stream and update the page. This example sends the entire file, you may want to truncate that at some point to save bandwidth and memory. This example sleeps between reads to reduce cpu load from the endless loop and allow other threads more active time.

from time import sleep
from flask import Flask, render_template

app = Flask(__name__)

@app.route('/')
def index():
    return render_template('index.html')

@app.route('/stream')
def stream():
    def generate():
        with open('job.log') as f:
            while True:
                yield f.read()
                sleep(1)

    return app.response_class(generate(), mimetype='text/plain')

app.run()

<pre id="output"></pre>
<script>
    var output = document.getElementById('output');

    var xhr = new XMLHttpRequest();
    xhr.open('GET', '{{ url_for('stream') }}');
    xhr.send();

    setInterval(function() {
        output.textContent = xhr.responseText;
    }, 1000);
</script>

This is almost the same as this answer, which describes how to stream and parse messages, although reading from an external file forever was novel enough to be it's own answer. The code here is simpler because we don't care about parsing messages or ending the stream, just tailing the file forever.

Reading log files as they're updated in Go

I have written a Go package -- github.com/hpcloud/tail -- to do exactly this.

t, err := tail.TailFile("/var/log/nginx.log", tail.Config{Follow: true})
for line := range t.Lines {
    fmt.Println(line.Text)
}

...

Quoting kostix's answer:

in real life files might be truncated, replaced or renamed (because that's what tools like logrotate are supposed to do).

If a file gets truncated, it will automatically be re-opened. To support re-opening renamed files (due to logrotate, etc.), you can set Config.ReOpen, viz.:

t, err := tail.TailFile("/var/log/nginx.log", tail.Config{
    Follow: true,
    ReOpen: true})
for line := range t.Lines {
    fmt.Println(line.Text)
}

Config.ReOpen is analogous to tail -F (capital F):

 -F      The -F option implies the -f option, but tail will also check to see if the file being followed has been
         renamed or rotated.  The file is closed and reopened when tail detects that the filename being read from
         has a new inode number.  The -F option is ignored if reading from standard input rather than a file.

---

Related Topics