Display Data Streamed from a Flask View as It Updates

Display data streamed from a Flask view as it updates

You can stream data in a response, but you can't dynamically update a template the way you describe. The template is rendered once on the server side, then sent to the client.

One solution is to use JavaScript to read the streamed response and output the data on the client side. Use XMLHttpRequest to make a request to the endpoint that will stream the data. Then periodically read from the stream until it's done.

This introduces complexity, but allows updating the page directly and gives complete control over what the output looks like. The following example demonstrates that by displaying both the current value and the log of all values.

This example assumes a very simple message format: a single line of data, followed by a newline. This can be as complex as needed, as long as there's a way to identify each message. For example, each loop could return a JSON object which the client decodes.

from math import sqrt
from time import sleep
from flask import Flask, render_template

app = Flask(__name__)

@app.route("/")
def index():
    return render_template("index.html")

@app.route("/stream")
def stream():
    def generate():
        for i in range(500):
            yield "{}\n".format(sqrt(i))
            sleep(1)

    return app.response_class(generate(), mimetype="text/plain")

<p>This is the latest output: <span id="latest"></span></p>
<p>This is all the output:</p>
<ul id="output"></ul>
<script>
    var latest = document.getElementById('latest');
    var output = document.getElementById('output');

    var xhr = new XMLHttpRequest();
    xhr.open('GET', '{{ url_for('stream') }}');
    xhr.send();
    var position = 0;

    function handleNewData() {
        // the response text include the entire response so far
        // split the messages, then take the messages that haven't been handled yet
        // position tracks how many messages have been handled
        // messages end with a newline, so split will always show one extra empty message at the end
        var messages = xhr.responseText.split('\n');
        messages.slice(position, -1).forEach(function(value) {
            latest.textContent = value;  // update the latest value in place
            // build and append a new item to a list to log all output
            var item = document.createElement('li');
            item.textContent = value;
            output.appendChild(item);
        });
        position = messages.length - 1;
    }

    var timer;
    timer = setInterval(function() {
        // check the response for new data
        handleNewData();
        // stop checking once the response has ended
        if (xhr.readyState == XMLHttpRequest.DONE) {
            clearInterval(timer);
            latest.textContent = 'Done';
        }
    }, 1000);
</script>

---

An <iframe> can be used to display streamed HTML output, but it has some downsides. The frame is a separate document, which increases resource usage. Since it's only displaying the streamed data, it might not be easy to style it like the rest of the page. It can only append data, so long output will render below the visible scroll area. It can't modify other parts of the page in response to each event.

index.html renders the page with a frame pointed at the stream endpoint. The frame has fairly small default dimensions, so you may want to to style it further. Use render_template_string, which knows to escape variables, to render the HTML for each item (or use render_template with a more complex template file). An initial line can be yielded to load CSS in the frame first.

from flask import render_template_string, stream_with_context

@app.route("/stream")
def stream():
    @stream_with_context
    def generate():
        yield render_template_string('<link rel=stylesheet href="{{ url_for("static", filename="stream.css") }}">')

        for i in range(500):
            yield render_template_string("<p>{{ i }}: {{ s }}</p>\n", i=i, s=sqrt(i))
            sleep(1)

    return app.response_class(generate())

<p>This is all the output:</p>
<iframe src="{{ url_for("stream") }}"></iframe>

updating streamed data from flask in real time

JavaScript works asynchronously and after running send() it doesn't wait for response and it executes next line before it gets response with text.

To get one response you have to use xhr.onload (before send()) to define function which xhr will run when it gets response

 xhr.onload = function () {
    latest.textContent = xhr.responseText;
 }

 xhr.send()

---

Minimal working code

from flask import Flask, render_template_string
import time

app = Flask(__name__)

@app.route('/')
def index():
    return render_template_string('''<p>This is the current value: <span id="latest_value"></span></p>
<script>
    var latest = document.getElementById('latest_value');

    var xhr = new XMLHttpRequest();
    xhr.open('GET', '{{ url_for('stream') }}');

    xhr.onload = function() {
        latest.textContent = xhr.responseText;
    }

    xhr.send();

    /* this lines will give `><` because `xhr.responseText` is still empty */
    /* you can remove these lines */
    console.log(">" + xhr.responseText + "<")
    latest.textContent = ">" + xhr.responseText + "<";

</script>''')

@app.route('/stream_time')
def stream():
    def generate():
        current_time = time.strftime("%H:%M:%S\n")
        print(current_time)
        yield current_time

    return app.response_class(generate(), mimetype='text/plain')

app.run()

---

To update periodically text in browser you have to first use while loop in generator to send many values.

For test I use also time.sleep(1) to send less data

    def generate():
        while True:
            current_time = time.strftime("%H:%M:%S\n")
            print(current_time)
            yield current_time
            time.sleep(1)

Now in JavaScript you have to use xhr.onreadystatechange to assign function which will update text on page when it gets new data from server

xhr.onreadystatechange = function() {
    var all_messages = xhr.responseText.split('\n');
    last_message = all_messages.length - 2
    latest.textContent = all_messages[last_message]
}

Because xhr.responseText ends with \n so split('\n') creates empty message as last element on list all_messages so I use length - 2 instead of length - 1

---

Minimal working code

from flask import Flask, render_template_string
import time

app = Flask(__name__)

@app.route('/')
def index():
    return render_template_string('''<p>This is the current value: <span id="latest_value"></span></p>
<script>

    var latest = document.getElementById('latest_value');

    var xhr = new XMLHttpRequest();
    xhr.open('GET', '{{ url_for('stream') }}');

    xhr.onreadystatechange = function() {
        var all_lines = xhr.responseText.split('\\n');
        last_line = all_lines.length - 2
        latest.textContent = all_lines[last_line]

        if (xhr.readyState == XMLHttpRequest.DONE) {
            /*alert("The End of Stream");*/
            latest.textContent = "The End of Stream"
        }
    }

    xhr.send();

</script>''')

@app.route('/stream_time')
def stream():
    def generate():
        while True:
            current_time = time.strftime("%H:%M:%S\n")
            print(current_time)
            yield current_time
            time.sleep(1)

    return app.response_class(generate(), mimetype='text/plain')

app.run()

---

BTW: because I used render_template_string(html) instead of render_template(filename) so I had to use \\n instead of \n

update pyserial data in real time with flask

It depends on what you mean by "real-time". Let us take a look at how Flask works first.

Why it doesn't work

Everything outside of the functions with @app.route decorator gets executed only once, upon the start of the server. That is why your data is currently fetched just once, your data fetching code is outside those functions:

water = serial.Serial("COM4",9600)
water1 = water.readline().decode('ascii')

Refresh data on page refresh

Now, the easiest way how to fix it is to fetch the data each time you refresh the page like this:

@app.route('/plant')
def plant():
    water = serial.Serial("COM4",9600)
    water1 = water.readline().decode('ascii')
    # while True: // don't use while True though, it'll freeze your program forever.
        return '<h2>soil is:</h2> ' + water1 + '%'

Since plan() is executed upon each page refresh, your data also updates such.

Refresh data in "real" real-time

There is the third option though, to achieve the result that usually is called "real-time". You won't get around it without using Javascript though. In this case you set up an API, which will return your data, which you'll call from Javascript.

I, however, suspect, that what you really want to do is refresh data on page refresh.

Streaming data with Python and Flask

To replace existing content on the page you might need javascript i.e., you could send it or make it to make requests for you, use long polling, websockets, etc. There are many ways to do it, here's one that uses server send events:

#!/usr/bin/env python
import itertools
import time
from flask import Flask, Response, redirect, request, url_for

app = Flask(__name__)

@app.route('/')
def index():
    if request.headers.get('accept') == 'text/event-stream':
        def events():
            for i, c in enumerate(itertools.cycle('\|/-')):
                yield "data: %s %d\n\n" % (c, i)
                time.sleep(.1)  # an artificial delay
        return Response(events(), content_type='text/event-stream')
    return redirect(url_for('static', filename='index.html'))

if __name__ == "__main__":
    app.run(host='localhost', port=23423)

Where static/index.html:

<!doctype html>
<title>Server Send Events Demo</title>
<style>
  #data {
    text-align: center;
  }
</style>
<script src="http://code.jquery.com/jquery-latest.js"></script>
<script>
if (!!window.EventSource) {
  var source = new EventSource('/');
  source.onmessage = function(e) {
    $("#data").text(e.data);
  }
}
</script>
<div id="data">nothing received yet</div>

The browser reconnects by default in 3 seconds if the connection is lost. if there is nothing more to send the server could return 404 or just send some other than 'text/event-stream' content type in response to the next request. To stop on the client side even if the server has more data you could call source.close().

Note: if the stream is not meant to be infinite then use other techniques (not SSE) e.g., send javascript snippets to replace the text (infinite <iframe> technique):

#!/usr/bin/env python
import time
from flask import Flask, Response

app = Flask(__name__)

@app.route('/')
def index():
    def g():
        yield """<!doctype html>
<title>Send javascript snippets demo</title>
<style>
  #data {
    text-align: center;
  }
</style>
<script src="http://code.jquery.com/jquery-latest.js"></script>
<div id="data">nothing received yet</div>
"""

        for i, c in enumerate("hello"):
            yield """
<script>
  $("#data").text("{i} {c}")
</script>
""".format(i=i, c=c)
            time.sleep(1)  # an artificial delay
    return Response(g())

if __name__ == "__main__":
    app.run(host='localhost', port=23423)

I've inlined the html here to show that there is nothing more to it (no magic). Here's the same as above but using templates:

#!/usr/bin/env python
import time
from flask import Flask, Response

app = Flask(__name__)

def stream_template(template_name, **context):
    # http://flask.pocoo.org/docs/patterns/streaming/#streaming-from-templates
    app.update_template_context(context)
    t = app.jinja_env.get_template(template_name)
    rv = t.stream(context)
    # uncomment if you don't need immediate reaction
    ##rv.enable_buffering(5)
    return rv

@app.route('/')
def index():
    def g():
        for i, c in enumerate("hello"*10):
            time.sleep(.1)  # an artificial delay
            yield i, c
    return Response(stream_template('index.html', data=g()))

if __name__ == "__main__":
    app.run(host='localhost', port=23423)

Where templates/index.html:

<!doctype html>
<title>Send javascript with template demo</title>
<style>
  #data {
    text-align: center;
  }
</style>
<script src="http://code.jquery.com/jquery-latest.js"></script>
<div id="data">nothing received yet</div>
{% for i, c in data: %}
<script>
  $("#data").text("{{ i }} {{ c }}")
</script>
{% endfor %}

Display the contents of a log file as it is updated

Use a Flask view to continuously read from the file forever and stream the response. Use JavaScript to read from the stream and update the page. This example sends the entire file, you may want to truncate that at some point to save bandwidth and memory. This example sleeps between reads to reduce cpu load from the endless loop and allow other threads more active time.

from time import sleep
from flask import Flask, render_template

app = Flask(__name__)

@app.route('/')
def index():
    return render_template('index.html')

@app.route('/stream')
def stream():
    def generate():
        with open('job.log') as f:
            while True:
                yield f.read()
                sleep(1)

    return app.response_class(generate(), mimetype='text/plain')

app.run()

<pre id="output"></pre>
<script>
    var output = document.getElementById('output');

    var xhr = new XMLHttpRequest();
    xhr.open('GET', '{{ url_for('stream') }}');
    xhr.send();

    setInterval(function() {
        output.textContent = xhr.responseText;
    }, 1000);
</script>

This is almost the same as this answer, which describes how to stream and parse messages, although reading from an external file forever was novel enough to be it's own answer. The code here is simpler because we don't care about parsing messages or ending the stream, just tailing the file forever.

---

Related Topics