Streaming data with Python and Flask
To replace existing content on the page you might need javascript i.e., you could send it or make it to make requests for you, use long polling, websockets, etc. There are many ways to do it, here's one that uses server send events:
#!/usr/bin/env python
import itertools
import time
from flask import Flask, Response, redirect, request, url_for
app = Flask(__name__)
@app.route('/')
def index():
if request.headers.get('accept') == 'text/event-stream':
def events():
for i, c in enumerate(itertools.cycle('\|/-')):
yield "data: %s %d\n\n" % (c, i)
time.sleep(.1) # an artificial delay
return Response(events(), content_type='text/event-stream')
return redirect(url_for('static', filename='index.html'))
if __name__ == "__main__":
app.run(host='localhost', port=23423)
Where static/index.html
:
<!doctype html>
<title>Server Send Events Demo</title>
<style>
#data {
text-align: center;
}
</style>
<script src="http://code.jquery.com/jquery-latest.js"></script>
<script>
if (!!window.EventSource) {
var source = new EventSource('/');
source.onmessage = function(e) {
$("#data").text(e.data);
}
}
</script>
<div id="data">nothing received yet</div>
The browser reconnects by default in 3 seconds if the connection is lost. if there is nothing more to send the server could return 404 or just send some other than 'text/event-stream'
content type in response to the next request. To stop on the client side even if the server has more data you could call source.close()
.
Note: if the stream is not meant to be infinite then use other techniques (not SSE) e.g., send javascript snippets to replace the text (infinite <iframe>
technique):
#!/usr/bin/env python
import time
from flask import Flask, Response
app = Flask(__name__)
@app.route('/')
def index():
def g():
yield """<!doctype html>
<title>Send javascript snippets demo</title>
<style>
#data {
text-align: center;
}
</style>
<script src="http://code.jquery.com/jquery-latest.js"></script>
<div id="data">nothing received yet</div>
"""
for i, c in enumerate("hello"):
yield """
<script>
$("#data").text("{i} {c}")
</script>
""".format(i=i, c=c)
time.sleep(1) # an artificial delay
return Response(g())
if __name__ == "__main__":
app.run(host='localhost', port=23423)
I've inlined the html here to show that there is nothing more to it (no magic). Here's the same as above but using templates:
#!/usr/bin/env python
import time
from flask import Flask, Response
app = Flask(__name__)
def stream_template(template_name, **context):
# http://flask.pocoo.org/docs/patterns/streaming/#streaming-from-templates
app.update_template_context(context)
t = app.jinja_env.get_template(template_name)
rv = t.stream(context)
# uncomment if you don't need immediate reaction
##rv.enable_buffering(5)
return rv
@app.route('/')
def index():
def g():
for i, c in enumerate("hello"*10):
time.sleep(.1) # an artificial delay
yield i, c
return Response(stream_template('index.html', data=g()))
if __name__ == "__main__":
app.run(host='localhost', port=23423)
Where templates/index.html
:
<!doctype html>
<title>Send javascript with template demo</title>
<style>
#data {
text-align: center;
}
</style>
<script src="http://code.jquery.com/jquery-latest.js"></script>
<div id="data">nothing received yet</div>
{% for i, c in data: %}
<script>
$("#data").text("{{ i }} {{ c }}")
</script>
{% endfor %}
Flask Load New Page After Streaming Data
Consider using send_file()
or send_from_directory()
for sending files.
Getting 2 responses from 1 request is not possible, but you can split the problem into chunks with the help of some JS, following this simple diagram (not very UML precise, but that's it):
this diagram refers to a previous and simpler version of the code, which was later updated after the OP asked for flash()
to be called
POST to
/uploader
through a function called from the form byonsubmit
, so that besides saving the file you can also have some logic there, like checking the response statusprocess the file (I did a mockup of your processing through
upper()
)if the server responds with
201
("Created") then you can save the file and print "Download Complete" (I usedwindow.document.body.innerHTML
because it's only one tag and we can replace all the previous DOM; it shouldn't be used to change complex HTML)else, if the server responds with other status codes (like
500
), POST to/something-went-wrong
to get the new - possibly flashed - HTML to be rendered. The POST step is not shown in the diagram.
To test the error page, make some syntax error in the processing inside upload_file()
, like data_df = pd.read_csv(BytesIO(uploaded_file.
aread()))
In the something-went-wrong
response I addeed a CSP header to mitigate a possible malicious attack, because we can't trust the user enough.
Here's the code:
main.py
from flask import (Flask,
request,
redirect,
render_template,
send_file,
url_for,
Response, jsonify, flash, make_response)
from flask_wtf.csrf import CSRFProtect
from io import BytesIO
import pandas as pd
app = Flask(__name__)
app.secret_key = "your_secret"
csrf = CSRFProtect(app)
@app.route('/')
def index():
return render_template("index.html")
@app.route("/uploader", methods=['POST'])
def upload_file():
try:
uploaded_file = request.files['input_file']
data_df = pd.read_csv(BytesIO(uploaded_file.read()))
# do stuff
data_df["foo"] = data_df["foo"].str.upper()
# Stream buffer:
io_buffer = BytesIO()
data_df.to_csv(io_buffer)
io_buffer.seek(0)
except Exception as ex:
print(ex) # and log if needed
# Here I return the exception string just as an example! Not good in real usage.
return jsonify({"message": str(ex)}), 500
else:
return send_file(io_buffer,
download_name="result.csv",
as_attachment=True), 201
@app.route("/something-went-wrong", methods=["POST"])
def something_went_wrong():
flash(request.get_json()["message"])
response = make_response(render_template("something-went-wrong.html"), 200)
response.headers['Content-Security-Policy'] = "default-src 'self'"
return response
The form with the JS handler:
<form id="myForm" enctype="multipart/form-data" onsubmit="return submitHandler()">
<input type="hidden" name="csrfToken" value="{{ csrf_token() }}"/>
<label>CSV file</label><br>
<input type="file" id="inputFile" name="input_file" required/><br><br>
<!-- some other inputs -->
<div id="submitBtnContainer">
<input id="submitBtn" type="submit"/>
</div>
</form>
<script>
function submitHandler() {
const csrf_token = "{{ csrf_token() }}";
let formData = new FormData();
const file = document.getElementById('inputFile').files[0];
formData.append("input_file", file);
fetch("/uploader", {
method: "POST",
body: formData,
headers: {
"X-CSRFToken": csrf_token,
},
})
.then(response => {
if (response.status != 201) {
response.json().then(data => {
fetch("/something-went-wrong", {
method: "POST",
body: JSON.stringify({"message": data["message"]}),
headers: {
"Content-Type": "application/json",
"X-CSRFToken": csrf_token,
},
})
.then(response => response.text())
.then(text => {
window.document.body.innerHTML = text;
})
});
}
else {
return response.blob().then(blob => {
const file = new Blob([blob], { type: 'text/csv' });
const fileURL = URL.createObjectURL(file);
let fileLink = document.createElement('a');
fileLink.href = fileURL;
fileLink.download = "result.csv";
fileLink.click();
window.document.body.innerHTML = "<h1>Download Complete</h1>";
});
}
})
return false;
}
</script>
For completeness, my dummy csv "file.csv"
:
foo |
---|
bar |
Monitoring a real-time data stream with a flask web-app
I just needed to change this line
t = threading.Thread(target=stream.monitor())
to this:
t = threading.Thread(target=stream.monitor)
Display data streamed from a Flask view as it updates
You can stream data in a response, but you can't dynamically update a template the way you describe. The template is rendered once on the server side, then sent to the client.
One solution is to use JavaScript to read the streamed response and output the data on the client side. Use XMLHttpRequest
to make a request to the endpoint that will stream the data. Then periodically read from the stream until it's done.
This introduces complexity, but allows updating the page directly and gives complete control over what the output looks like. The following example demonstrates that by displaying both the current value and the log of all values.
This example assumes a very simple message format: a single line of data, followed by a newline. This can be as complex as needed, as long as there's a way to identify each message. For example, each loop could return a JSON object which the client decodes.
from math import sqrt
from time import sleep
from flask import Flask, render_template
app = Flask(__name__)
@app.route("/")
def index():
return render_template("index.html")
@app.route("/stream")
def stream():
def generate():
for i in range(500):
yield "{}\n".format(sqrt(i))
sleep(1)
return app.response_class(generate(), mimetype="text/plain")
<p>This is the latest output: <span id="latest"></span></p>
<p>This is all the output:</p>
<ul id="output"></ul>
<script>
var latest = document.getElementById('latest');
var output = document.getElementById('output');
var xhr = new XMLHttpRequest();
xhr.open('GET', '{{ url_for('stream') }}');
xhr.send();
var position = 0;
function handleNewData() {
// the response text include the entire response so far
// split the messages, then take the messages that haven't been handled yet
// position tracks how many messages have been handled
// messages end with a newline, so split will always show one extra empty message at the end
var messages = xhr.responseText.split('\n');
messages.slice(position, -1).forEach(function(value) {
latest.textContent = value; // update the latest value in place
// build and append a new item to a list to log all output
var item = document.createElement('li');
item.textContent = value;
output.appendChild(item);
});
position = messages.length - 1;
}
var timer;
timer = setInterval(function() {
// check the response for new data
handleNewData();
// stop checking once the response has ended
if (xhr.readyState == XMLHttpRequest.DONE) {
clearInterval(timer);
latest.textContent = 'Done';
}
}, 1000);
</script>
An <iframe>
can be used to display streamed HTML output, but it has some downsides. The frame is a separate document, which increases resource usage. Since it's only displaying the streamed data, it might not be easy to style it like the rest of the page. It can only append data, so long output will render below the visible scroll area. It can't modify other parts of the page in response to each event.
index.html
renders the page with a frame pointed at the stream
endpoint. The frame has fairly small default dimensions, so you may want to to style it further. Use render_template_string
, which knows to escape variables, to render the HTML for each item (or use render_template
with a more complex template file). An initial line can be yielded to load CSS in the frame first.
from flask import render_template_string, stream_with_context
@app.route("/stream")
def stream():
@stream_with_context
def generate():
yield render_template_string('<link rel=stylesheet href="{{ url_for("static", filename="stream.css") }}">')
for i in range(500):
yield render_template_string("<p>{{ i }}: {{ s }}</p>\n", i=i, s=sqrt(i))
sleep(1)
return app.response_class(generate())
<p>This is all the output:</p>
<iframe src="{{ url_for("stream") }}"></iframe>
fun clock streaming text with python and flask
in view you can avoid while
and sleep
in that example:
@app.route('/time_feed')
def time_feed():
def generate():
yield datetime.now().strftime("%Y.%m.%d|%H:%M:%S") # return also will work
return Response(generate(), mimetype='text')
in template:
<p id="clock">Here will be date|time</p>
<script>
var clock = document.getElementById("clock");
setInterval(() => {
fetch("{{ url_for('time_feed') }}")
.then(response => {
response.text().then(t => {clock.innerHTML = t})
});
}, 1000);
</script>
in your example with video streem it is just trick, it is not solution for real video streaming, just because not provide audio. If you need for real video stream you need to use webRTC with Kurento-media-server for example.
For python look on aiortc lib.
Related Topics
Typeerror: Unsupported Operand Type(S) for /: 'Str' and 'Str'
Scipy: Savefig Without Frames, Axes, Only Content
Automating Pydrive Verification Process
How Can This Function Be Rewritten to Implement Ordereddict
Sqlite Insert Query Not Working with Python
Sorting Python List Based on the Length of the String
Python Read JSON File and Modify
Multithreaded Blas in Python/Numpy
How to "Properly" Print a List
Checking If All Elements in a List Are Unique
Interpolate Nan Values in a Numpy Array
Scrape Website with Dynamic Mouseover Event
Meaning of Inter_Op_Parallelism_Threads and Intra_Op_Parallelism_Threads
How to Encrypt and Decrypt a String in Python
Running Infinite Loops Using Threads in Python
Python - Difference Between Two Strings