Javascript: Extract Video Frames Reliably

JavaScript: Extract video frames reliably

[2021 update]: Since this question (and answer) has first been posted, things have evolved in this area, and it is finally time to make an update; the method that was exposed here went out-of-date, but luckily a few new or incoming APIs can help us better in extracting video frames:

The most promising and powerful one, but still under development, with a lot of restrictions: WebCodecs

This new API unleashes access to the media decoders and encoders, enabling us to access raw data from video frames (YUV planes), which may be a lot more useful for many applications than rendered frames; and for the ones who need rendered frames, the VideoFrame interface that this API exposes can be drawn directly to a <canvas> element or converted to an ImageBitmap, avoiding the slow route of the MediaElement.

However there is a catch, apart from its current low support, this API needs that the input has been demuxed already.

There are some demuxers online, for instance for MP4 videos GPAC's mp4box.js will help a lot.

A full example can be found on the proposal's repo.

The key part consists of

const decoder = new VideoDecoder({
  output: onFrame, // the callback to handle all the VideoFrame objects
  error: e => console.error(e),
});
decoder.configure(config); // depends on the input file, your demuxer should provide it
demuxer.start((chunk) => { // depends on the demuxer, but you need it to return chunks of video data
  decoder.decode(chunk); // will trigger our onFrame callback  
})

Note that we can even grab the frames of a MediaStream, thanks to MediaCapture Transform's MediaStreamTrackProcessor.
This means that we should be able to combine HTMLMediaElement.captureStream() and this API in order to get our VideoFrames, without the need for a demuxer. However this is true only for a few codecs, and it means that we will extract frames at reading speed...

Anyway, here is an example working on latest Chromium based browsers, with chrome://flags/#enable-experimental-web-platform-features switched on:

const frames = [];
const button = document.querySelector("button");
const select = document.querySelector("select");
const canvas = document.querySelector("canvas");
const ctx = canvas.getContext("2d");

button.onclick = async(evt) => {
  if (window.MediaStreamTrackProcessor) {
    let stopped = false;
    const track = await getVideoTrack();
    const processor = new MediaStreamTrackProcessor(track);
    const reader = processor.readable.getReader();
    readChunk();

    function readChunk() {
      reader.read().then(async({ done, value }) => {
        if (value) {
          const bitmap = await createImageBitmap(value);
          const index = frames.length;
          frames.push(bitmap);
          select.append(new Option("Frame #" + (index + 1), index));
          value.close();
        }
        if (!done && !stopped) {
          readChunk();
        } else {
          select.disabled = false;
        }
      });
    }
    button.onclick = (evt) => stopped = true;
    button.textContent = "stop";
  } else {
    console.error("your browser doesn't support this API yet");
  }
};

select.onchange = (evt) => {
  const frame = frames[select.value];
  canvas.width = frame.width;
  canvas.height = frame.height;
  ctx.drawImage(frame, 0, 0);
};

async function getVideoTrack() {
  const video = document.createElement("video");
  video.crossOrigin = "anonymous";
  video.src = "https://upload.wikimedia.org/wikipedia/commons/a/a4/BBH_gravitational_lensing_of_gw150914.webm";
  document.body.append(video);
  await video.play();
  const [track] = video.captureStream().getVideoTracks();
  video.onended = (evt) => track.stop();
  return track;
}

video,canvas {
  max-width: 100%
}

<button>start</button>
<select disabled>
</select>
<canvas></canvas>

Extract frames from video angular

The average HTML frame rate is 24 FPS, so if the video is 30 seconds, the total frame number is 30 x 24
Get the time of the video

const videoDOM=document.getElementById("video"); 
videoDom.duration

this will give u NaN, so to avoid that, we need to wait for the duration to change then call videoDom.duration

videoDom.ondurationchange = () => {
   this.vidDuration = vid.duration;
};

that way we can get an actual duration

Javascript: frame precise video stop

The video has frame rate of 25fps, and not 24fps:

After putting the correct value it works ok: demo

The VideoFrame api heavily relies on FPS provided by you. You can find FPS of your videos offline and send as metadata along with stop frames from server.

---

The site videoplayer.handmadeproductions.de uses window.requestAnimationFrame() to get the callback.

---

There is a new better alternative to requestAnimationFrame. The requestVideoFrameCallback(), allows us to do per-video-frame operations on video.

The same functionality, you domed in OP, can be achieved like this:

   const callback = (now, metadata) => {
      if (startTime == 0) {
        startTime = now;
      }
      elapsed = metadata.mediaTime;
      currentFrame = metadata.presentedFrames - doneCount;

      fps = (currentFrame / elapsed).toFixed(3);
      fps = !isFinite(fps) ? 0 : fps;

      updateStats();
      if (stopFrames.includes(currentFrame)) {
        pauseMyVideo();
      } else {
        video.requestVideoFrameCallback(callback);
      }
   };
   video.requestVideoFrameCallback(callback);

And here is how demo looks like.

The API works on chromium based browsers like Chrome, Edge, Brave etc.

---

There is a JS library, which finds frame rate from video binary file, named mediainfo.js.

---

Related Topics