Best Approach to Real Time Http Streaming to Html5 Video Client

Best approach to real time http streaming to HTML5 video client

EDIT 3: As of IOS 10, HLS will support fragmented mp4 files. The answer
now, is to create fragmented mp4 assets, with a DASH and HLS manifest. > Pretend flash, iOS9 and below and IE 10 and below don't exist.

Everything below this line is out of date. Keeping it here for posterity.

---

EDIT 2: As people in the comments are pointing out, things change.
Almost all browsers will support AVC/AAC codecs.
iOS still requires HLS. But via adaptors like hls.js you can play
HLS in MSE. The new answer is HLS+hls.js if you need iOS. or just
Fragmented MP4 (i.e. DASH) if you don't

There are many reasons why video and, specifically, live video is very difficult. (Please note that the original question specified that HTML5 video is a requirement, but the asker stated Flash is possible in the comments. So immediately, this question is misleading)

First I will restate: THERE IS NO OFFICIAL SUPPORT FOR LIVE STREAMING OVER HTML5. There are hacks, but your mileage may vary.

EDIT: since I wrote this answer Media Source Extensions have matured,
and are now very close to becoming a viable option. They are supported
on most major browsers. IOS continues to be a hold out.

Next, you need to understand that Video on demand (VOD) and live video are very different. Yes, they are both video, but the problems are different, hence the formats are different. For example, if the clock in your computer runs 1% faster than it should, you will not notice on a VOD. With live video, you will be trying to play video before it happens. If you want to join a a live video stream in progress, you need the data necessary to initialize the decoder, so it must be repeated in the stream, or sent out of band. With VOD, you can read the beginning of the file them seek to whatever point you wish.

Now let's dig in a bit.

Platforms:

• iOS
• PC
• Mac
• Android

Codecs:

• vp8/9
• h.264
• thora (vp3)

Common Delivery methods for live video in browsers:

• DASH (HTTP)
• HLS (HTTP)
• flash (RTMP)
• flash (HDS)

Common Delivery methods for VOD in browsers:

• DASH (HTTP Streaming)
• HLS (HTTP Streaming)
• flash (RTMP)
• flash (HTTP Streaming)
• MP4 (HTTP pseudo streaming)
• I'm not going to talk about MKV and OOG because I do not know them very well.

html5 video tag:

• MP4
• webm
• ogg

---

Lets look at which browsers support what formats

Safari:

• HLS (iOS and mac only)
• h.264
• MP4

Firefox

• DASH (via MSE but no h.264)
• h.264 via Flash only!
• VP9
• MP4
• OGG
• Webm

IE

• Flash
• DASH (via MSE IE 11+ only)
• h.264
• MP4

Chrome

• Flash
• DASH (via MSE)
• h.264
• VP9
• MP4
• webm
• ogg

MP4 cannot be used for live video (NOTE: DASH is a superset of MP4, so don't get confused with that). MP4 is broken into two pieces: moov and mdat. mdat contains the raw audio video data. But it is not indexed, so without the moov, it is useless. The moov contains an index of all data in the mdat. But due to its format, it can not be 'flattened' until the timestamps and size of EVERY frame is known. It may be possible to construct an moov that 'fibs' the frame sizes, but is is very wasteful bandwidth wise.

So if you want to deliver everywhere, we need to find the least common denominator. You will see there is no LCD here without resorting to flash
example:

• iOS only supports h.264 video. and it only supports HLS for live.
• Firefox does not support h.264 at all, unless you use flash
• Flash does not work in iOS

The closest thing to an LCD is using HLS to get your iOS users, and flash for everyone else.
My personal favorite is to encode HLS, then use flash to play HLS for everyone else. You can play HLS in flash via JW player 6, (or write your own HLS to FLV in AS3 like I did)

Soon, the most common way to do this will be HLS on iOS/Mac and DASH via MSE everywhere else (This is what Netflix will be doing soon). But we are still waiting for everyone to upgrade their browsers. You will also likely need a separate DASH/VP9 for Firefox (I know about open264; it sucks. It can't do video in main or high profile. So it is currently useless).

Real Time Streaming to HTML5 (with out webrtc) just using video tag

Evren,

Since you have asked this question initially, the Media Source Extensions
https://www.w3.org/TR/media-source/ have matured enough to be able to play very short (30ms) ISO-BMFF video/mp4 segments with just a little buffering.

Refer to
HTML5 live streaming

So your statement

(one can not wrap h264 and mp3 in real time and wrap it and send it to the client)

is out of date now. Yes you can do it with h264 + AAC.

There are several implementations out there; take a look at Unreal Media Server.
From Unreal Media Server FAQ: http://umediaserver.net/umediaserver/faq.html

How is Unreal HTML5 live streaming different from MPEG-DASH?
Unlike MPEG-DASH, Unreal Media Server uses a WebSocket protocol for live streaming to HTML5 MSE element in web browsers. This is much more efficient than fetching segments via HTTP requests per MPEG-DASH. Also, Unreal Media Server sends segments of minimal duration, as low as 30 ms. That allows for low, sub-second latency streaming, while MPEG-DASH, like other HTTP chunk-based live streaming protocols, cannot provide low latency live streaming.

Their demos webpage has a live HTML5 feed from RTSP camera:
http://umediaserver.net/umediaserver/demos.html
Notice that the latency in HTML5 player is comparable to that in Flash player.

html5-Video-Tag vs. Streaming Server

At the moment the main benefits of flash are:

• Easy to add Ads
• Browser compatibility
• No need to support many different codecs

So if you need any of this you might choose Flash

low-latency html5 video on a LAN

You won't achieve low latency with any of the segmented distribution methods (HLS, DASH, or similar). The very nature of these protocols is that the data is chunked into relatively large pieces. 4 seconds with HLS is amazingly low, and with chunks that small you have quite a bit of overhead... a waste of bandwidth and not really HLS and DASH are good for.

The first one is working under the assumption that the content consumers will not be on site.

My answer there (https://stackoverflow.com/a/37475943/362536) doesn't assume that the consumers will not be on your site... that's not the case at all. What I'm suggesting there is that you take advantage of YouTube and embed their viewer when low latency isn't needed, saving you mountains of money.

If all of your viewers require low latency video to make this work, you're going to have to get crafty on the server side. If you told us what sort of scale you were working with, perhaps we could suggest something more specific. Since you didn't, let's focus on the possibilities client-side.

WebRTC is one of the best options. Everything in the whole WebRTC stack is built with low latency in mind. With WebRTC, you can get those sub-second latencies in normal operation. Note that aren't a lot of good choices for streaming servers that support WebRTC today.

You can also use Media Source Extensions and Web Sockets. This gives you quite a bit of control and allows very fast streaming of data to the clients, at a slightly higher cost of latency. It's much easier to do this than it is to implement your own server-side WebRTC that supports media streams.

I strongly recommend reading over my answer on that other question again as well. There are a lot of considerations here... make very sure that this low latency is actually worth the reduction in quality and the financial costs involved. This is rarely the case, expecially for 10s of thousands of users or more.

---

Related Topics