Capture Frames from Video with HTML5 and JavaScript

Capture frames from video with HTML5 and JavaScript

Cause

The problem is that seeking video (by setting it's currentTime) is asynchronous.

You need to listen to the seeked event or else it will risk take the actual current frame which is likely your old value.

As it is asynchronous you must not use the setInterval() as it is asynchronous too and you will not be able to properly synchronize when the next frame is seeked to. There is no need to use setInterval() as we will utilize the seeked event instead which will keep everything is sync.

Solution

By re-writing the code a little you can use the seeked event to go through the video to capture the correct frame as this event ensures us that we are actually at the frame we requested by setting the currentTime property.

Example

// global or parent scope of handlers
var video = document.getElementById("video"); // added for clarity: this is needed
var i = 0;

video.addEventListener('loadeddata', function() {
    this.currentTime = i;
});

Add this event handler to the party:

video.addEventListener('seeked', function() {

  // now video has seeked and current frames will show
  // at the time as we expect
  generateThumbnail(i);

  // when frame is captured, increase here by 5 seconds
  i += 5;

  // if we are not past end, seek to next interval
  if (i <= this.duration) {
    // this will trigger another seeked event
    this.currentTime = i;
  }
  else {
    // Done!, next action
  }
});

JavaScript: Extract video frames reliably

[2021 update]: Since this question (and answer) has first been posted, things have evolved in this area, and it is finally time to make an update; the method that was exposed here went out-of-date, but luckily a few new or incoming APIs can help us better in extracting video frames:

The most promising and powerful one, but still under development, with a lot of restrictions: WebCodecs

This new API unleashes access to the media decoders and encoders, enabling us to access raw data from video frames (YUV planes), which may be a lot more useful for many applications than rendered frames; and for the ones who need rendered frames, the VideoFrame interface that this API exposes can be drawn directly to a <canvas> element or converted to an ImageBitmap, avoiding the slow route of the MediaElement.

However there is a catch, apart from its current low support, this API needs that the input has been demuxed already.

There are some demuxers online, for instance for MP4 videos GPAC's mp4box.js will help a lot.

A full example can be found on the proposal's repo.

The key part consists of

const decoder = new VideoDecoder({
  output: onFrame, // the callback to handle all the VideoFrame objects
  error: e => console.error(e),
});
decoder.configure(config); // depends on the input file, your demuxer should provide it
demuxer.start((chunk) => { // depends on the demuxer, but you need it to return chunks of video data
  decoder.decode(chunk); // will trigger our onFrame callback  
})

Note that we can even grab the frames of a MediaStream, thanks to MediaCapture Transform's MediaStreamTrackProcessor.
This means that we should be able to combine HTMLMediaElement.captureStream() and this API in order to get our VideoFrames, without the need for a demuxer. However this is true only for a few codecs, and it means that we will extract frames at reading speed...

Anyway, here is an example working on latest Chromium based browsers, with chrome://flags/#enable-experimental-web-platform-features switched on:

const frames = [];
const button = document.querySelector("button");
const select = document.querySelector("select");
const canvas = document.querySelector("canvas");
const ctx = canvas.getContext("2d");

button.onclick = async(evt) => {
  if (window.MediaStreamTrackProcessor) {
    let stopped = false;
    const track = await getVideoTrack();
    const processor = new MediaStreamTrackProcessor(track);
    const reader = processor.readable.getReader();
    readChunk();

    function readChunk() {
      reader.read().then(async({ done, value }) => {
        if (value) {
          const bitmap = await createImageBitmap(value);
          const index = frames.length;
          frames.push(bitmap);
          select.append(new Option("Frame #" + (index + 1), index));
          value.close();
        }
        if (!done && !stopped) {
          readChunk();
        } else {
          select.disabled = false;
        }
      });
    }
    button.onclick = (evt) => stopped = true;
    button.textContent = "stop";
  } else {
    console.error("your browser doesn't support this API yet");
  }
};

select.onchange = (evt) => {
  const frame = frames[select.value];
  canvas.width = frame.width;
  canvas.height = frame.height;
  ctx.drawImage(frame, 0, 0);
};

async function getVideoTrack() {
  const video = document.createElement("video");
  video.crossOrigin = "anonymous";
  video.src = "https://upload.wikimedia.org/wikipedia/commons/a/a4/BBH_gravitational_lensing_of_gw150914.webm";
  document.body.append(video);
  await video.play();
  const [track] = video.captureStream().getVideoTracks();
  video.onended = (evt) => track.stop();
  return track;
}

video,canvas {
  max-width: 100%
}

<button>start</button>
<select disabled>
</select>
<canvas></canvas>

How to capture thumbnails (frame) from video with HTML5 and Jquery?

Check the following:
https://jsfiddle.net/psbqn9d0/1/

It won't create the view you shown in the picture, but will change the thumbnail every 5 second. I have also added few logging to help you understand the code.

Now, if you want to achieve what is shown in figure then this is possible:-

1. if you have loaded the video completely. Then create all the thumbnails and add them to the container.
2. you create the thumbnails on the server and send them to user (like done by most video sites (so we can see what is in the video without loading!))
3. OR you can create all the thumbnails without loading the whole video. I am not sure if this is the best way, but this will definitely work.

You should create another video tag with the same source link, and hide it using css. Then you will create a javascript function which will seek it to every 5th second programmatically and keep saving the thumbnails and adding them in your container. This way, your thumbnails will be generated for the whole video without loading the whole video.

I hope it clarifies your doubts!

How to get frames from a component video/video

Take a look at this codepen: Demo

var videoId = 'video';    var scaleFactor = 0.25;    var snapshots = [];
    /**     * Captures a image frame from the provided video element.     *     * @param {Video} video HTML5 video element from where the image frame will be captured.     * @param {Number} scaleFactor Factor to scale the canvas element that will be return. This is an optional parameter.     *     * @return {Canvas}     */    function capture(video, scaleFactor) {        if (scaleFactor == null) {            scaleFactor = 1;        }        var w = video.videoWidth * scaleFactor;        var h = video.videoHeight * scaleFactor;        var canvas = document.createElement('canvas');        canvas.width = w;        canvas.height = h;        var ctx = canvas.getContext('2d');        ctx.drawImage(video, 0, 0, w, h);        return canvas;    }
    /**     * Invokes the <code>capture</code> function and attaches the canvas element to the DOM.     */    function shoot() {        var video = document.getElementById(videoId);        var output = document.getElementById('output');        var canvas = capture(video, scaleFactor);        canvas.onclick = function() {            window.open(this.toDataURL(image/jpg));        };        snapshots.unshift(canvas);        output.innerHTML = '';        for (var i = 0; i < 4; i++) {            output.appendChild(snapshots[i]);        }    }
(function() {  var captureit = document.getElementById('cit');  captureit.click();})();

---

Related Topics