Process Video Stream from Memory Buffer

process video stream from memory buffer

I had a similar need recently. I was looking for a way in OpenCV to play a video that was already in memory, but without ever having to write the video file to disk. I found out that the FFMPEG interface already supports this through av_open_input_stream. There is just a little more prep work required compared to the av_open_input_file call used in OpenCV to open a file.

Between the following two websites I was able to piece together a working solution using the ffmpeg calls. Please refer to the information on these websites for more details:

http://ffmpeg.arrozcru.org/forum/viewtopic.php?f=8&t=1170

http://cdry.wordpress.com/2009/09/09/using-custom-io-callbacks-with-ffmpeg/

To get it working in OpenCV, I ended up adding a new function to the CvCapture_FFMPEG class:

virtual bool openBuffer( unsigned char* pBuffer, unsigned int bufLen );

I provided access to it through a new API call in the highgui DLL, similar to cvCreateFileCapture. The new openBuffer function is basically the same as the open( const char* _filename ) function with the following difference:

err = av_open_input_file(&ic, _filename, NULL, 0, NULL);

is replaced by:

ic = avformat_alloc_context();
ic->pb = avio_alloc_context(pBuffer, bufLen, 0, pBuffer, read_buffer, NULL, NULL);

if(!ic->pb) {
    // handle error
}

// Need to probe buffer for input format unless you already know it
AVProbeData probe_data;
probe_data.buf_size = (bufLen < 4096) ? bufLen : 4096;
probe_data.filename = "stream";
probe_data.buf = (unsigned char *) malloc(probe_data.buf_size);
memcpy(probe_data.buf, pBuffer, probe_data.buf_size);

AVInputFormat *pAVInputFormat = av_probe_input_format(&probe_data, 1);

if(!pAVInputFormat)
    pAVInputFormat = av_probe_input_format(&probe_data, 0);

// cleanup
free(probe_data.buf);
probe_data.buf = NULL;

if(!pAVInputFormat) {
    // handle error
}

pAVInputFormat->flags |= AVFMT_NOFILE;

err = av_open_input_stream(&ic , ic->pb, "stream", pAVInputFormat, NULL);

Also, make sure to call av_close_input_stream in the CvCapture_FFMPEG::close() function instead of av_close_input_file in this situation.

Now the read_buffer callback function that is passed in to avio_alloc_context I defined as:

static int read_buffer(void *opaque, uint8_t *buf, int buf_size)
{
    // This function must fill the buffer with data and return number of bytes copied.
    // opaque is the pointer to private_data in the call to avio_alloc_context (4th param)
    
    memcpy(buf, opaque, buf_size);
    return buf_size;
}

This solution assumes the entire video is contained in a memory buffer and would probably have to be tweaked to work with streaming data.

So that's it! Btw, I'm using OpenCV version 2.1 so YMMV.

How to stream frames from OpenCV C++ code to Video4Linux or ffmpeg?

We may use the same technique as in my following Python code sample.

• Execute FFmpeg as sub-process, open stdin pipe for writing

   FILE *pipeout = popen(ffmpeg_cmd.c_str(), "w")
  

• Write frame.data to stdin pipe of FFmpeg sub-process (in a loop)

   fwrite(frame.data, 1, width*height*3, pipeout);
  

• Close the pipe at the end (it is going to close the sub-process)

   pclose(pipeout);
  

The following sample is a generic example - building numbered frames, and writing the encoded video to an MKV output file.

The example uses the following equivalent command line:

ffmpeg -y -f rawvideo -r 10 -video_size 320x240 -pixel_format bgr24 -i pipe: -vcodec libx264 -crf 24 -pix_fmt yuv420p output.mkv

You may adjust the arguments for your specific requirements (replace output.mkv with udp://@127.0.0.1:25000).

Replace the numbered frame with capture >> frame, and adjust the size and framerate.

Code sample:

#include <stdio.h>
#include <chrono>
#include <thread>
#include "opencv2/opencv.hpp"

int main()
{
    int width = 320;
    int height = 240;
    int n_frames = 100;
    int fps = 10;

    //Use a "generic" example (write the output video in output.mkv video file).
    //ffmpeg -y -f rawvideo -r 10 -video_size 320x240 -pixel_format bgr24 -i pipe: -vcodec libx264 -crf 24 -pix_fmt yuv420p output.mkv
    std::string ffmpeg_cmd = std::string("ffmpeg -y -f rawvideo -r ") + std::to_string(fps) +
                             " -video_size " + std::to_string(width) + "x" + std::to_string(height) +
                             " -pixel_format bgr24 -i pipe: -vcodec libx264 -crf 24 -pix_fmt yuv420p output.mkv";

    //Execute FFmpeg as sub-process, open stdin pipe (of FFmpeg sub-process) for writing.
    //In Windows we need to use _popen and in Linux popen
#ifdef _MSC_VER
    FILE *pipeout = _popen(ffmpeg_cmd.c_str(), "wb");   //Windows (ffmpeg.exe must be in the execution path)
#else
    //https://batchloaf.wordpress.com/2017/02/12/a-simple-way-to-read-and-write-audio-and-video-files-in-c-using-ffmpeg-part-2-video/
    FILE *pipeout = popen(ffmpeg_cmd.c_str(), "w");     //Linux (assume ffmpeg exist in /usr/bin/ffmpeg (and in path).
#endif

    for (int i = 0; i < n_frames; i++)
    {
        cv::Mat frame = cv::Mat(height, width, CV_8UC3);
        frame = cv::Scalar(60, 60, 60); //Fill background with dark gray
        cv::putText(frame, std::to_string(i+1), cv::Point(width/2-50*(int)(std::to_string(i+1).length()), height/2+50), cv::FONT_HERSHEY_DUPLEX, 5, cv::Scalar(30, 255, 30), 10);  // Draw a green number

        cv::imshow("frame", frame);cv::waitKey(1); //Show the frame for testing

        //Write width*height*3 bytes to stdin pipe of FFmpeg sub-process (assume frame data is continuous in the RAM).
        fwrite(frame.data, 1, width*height*3, pipeout);
    }

    // Flush and close input and output pipes
    fflush(pipeout);

#ifdef _MSC_VER
    _pclose(pipeout);   //Windows
#else
    pclose(pipeout);    //Linux
#endif

    //It looks like we need to wait one more second at the end. //https://stackoverflow.com/a/62804585/4926757
    std::this_thread::sleep_for(std::chrono::milliseconds(1000)); // sleep for 1 second

    return 0;
}

Related Topics