Displaying Webcam Video with Qt

open webcamera with OpenCV and show it with QLabel - white window

I don't have much experience, but I can see what can go wrong here:

 for (;;) {

    cap >> image;
        //conversion from Mat to QImage
    Mat dest;
    cvtColor(image, dest,CV_BGR2RGB);
    QImage image1= QImage((uchar*) dest.data, dest.cols, dest.rows, dest.step, QImage::Format_RGB888);

        //show Qimage using QLabel
    myLabel.setPixmap(QPixmap::fromImage(image1));
    myLabel.show();
    //imshow("camera",image);
    //if (waitKey(30)>= 0)  break;
 }

You are doing this in dead loop - it will cause your QLabel to update itself infinitely, so you may not see anything. Also, if uncommenting waitKey is helping you, that pretty much mean that you are converting data to QImage well, but something other is broken.

Note that a.exec() will never execute, as you will be stuck in the loop, but I guess this was enough for hitting the concept.

In order not to stuck event loop, you need to create QTimer and every x milliseconds to update your widget:

 class VideoWindow: public QWidget
 {
    Q_OBJECT
    public:
        VideoWindow(QWidget* parent = 0): QWidget(parent), cap(0)
        {
            timer = new QTimer(this);
            connect(timer, SIGNAL(timeout()), this, SLOT(updatePicture()));
            timer->start(20);
        }

    public slots:
        void updatePicture()
        {
            cap >> image;
            //conversion from Mat to QImage
            Mat dest;
            cvtColor(image, dest,CV_BGR2RGB);
            QImage image1 = QImage((uchar*) dest.data, dest.cols, dest.rows, dest.step, QImage::Format_RGB888);

            //show Qimage using QLabel
            setPixmap(QPixmap::fromImage(image1));
        }

    private:
        QTimer * timer;
        VideoCapture cap;
};

int main(int argc, char** argv)
{
    QApplication app(argc, argv);
    VideoWindow w;
    w.show();

    app.exec();
    return 0;   
}

Windows + Qt and how to capture webcam feed without OpenCV

Ok, I finally did it, so I will post here my solution so we have something clear about this.

I used a library called 'ESCAPI': http://sol.gfxile.net/escapi/index.html

This provide a extremely easy way to capture frames from the device. With this raw data, I just create a QImage which later show in a QLabel.

I created a simple object to handle this.

#include <QDebug>
#include "camera.h"

Camera::Camera(int width, int height, QObject *parent) :
    QObject(parent),
    width_(width),
    height_(height)
{
    capture_.mWidth = width;
    capture_.mHeight = height;
    capture_.mTargetBuf = new int[width * height];

    int devices = setupESCAPI();
    if (devices == 0)
    {
        qDebug() << "[Camera] ESCAPI initialization failure or no devices found";
    }
}

Camera::~Camera()
{
    deinitCapture(0);
}

int Camera::initialize()
{
    if (initCapture(0, &capture_) == 0)
    {
        qDebug() << "[Camera] Capture failed - device may already be in use";
        return -2;
    }
    return 0;
}

void Camera::deinitialize()
{
    deinitCapture(0);
}

int Camera::capture()
{
    doCapture(0);
    while(isCaptureDone(0) == 0);

    image_ = QImage(width_, height_, QImage::Format_ARGB32);
    for(int y(0); y < height_; ++y)
    {
        for(int x(0); x < width_; ++x)
        {
            int index(y * width_ + x);
            image_.setPixel(x, y, capture_.mTargetBuf[index]);
         }
    }
    return 1;
}

And the header file:

#ifndef CAMERA_H
#define CAMERA_H

#include <QObject>
#include <QImage>
#include "escapi.h"

class Camera : public QObject
{
    Q_OBJECT

public:
    explicit Camera(int width, int height, QObject *parent = 0);
    ~Camera();
    int initialize();
    void deinitialize();
    int capture();
    const QImage& getImage() const { return image_; }
    const int* getImageRaw() const { return capture_.mTargetBuf; }

private:
    int width_;
    int height_;
    struct SimpleCapParams capture_;
    QImage image_;
};

#endif // CAMERA_H

It's so simple, but just for example purposes.
The use should be something like:

Camera cam(320, 240);
cam.initialize();
cam.capture();
QImage img(cam.getImage());
ui->label->setPixmap(QPixmap::fromImage(img));

Of course, you can use a QTimer and update the frame in QLabel and you will have video there...

Hope it help! and thanks Nicholas for your help!

How to efficiently display OpenCV video in Qt?

Using QImage::scanLine forces a deep copy, so at the minimum, you should use constScanLine, or, better yet, change the slot's signature to:

void widget::set_image(const QImage & image);

Of course, your problem then becomes something else: the QImage instance points to the data of a frame that lives in another thread, and can (and will) change at any moment.

There is a solution for that: one needs to use fresh frames allocated on the heap, and the frame needs to be captured within QImage. QScopedPointer is used to prevent memory leaks until the QImage takes ownership of the frame.

static void matDeleter(void* mat) { delete static_cast<cv::Mat*>(mat); }

class capture {
   Q_OBJECT
   bool m_enable;
   ...
public:
   Q_SIGNAL void image_ready(const QImage &);
   ...
};

void capture::start_process() {
  m_enable = true;
  while(m_enable) {
    QScopedPointer<cv::Mat> frame(new cv::Mat);
    if (!m_video_handle->read(*frame)) {
      break;
    }
    cv::cvtColor(*frame, *frame, CV_BGR2RGB);

    // Here the image instance takes ownership of the frame.
    const QImage image(frame->data, frame->cols, frame->rows, frame->step,
                       QImage::Format_RGB888, matDeleter, frame.take());       
    emit image_ready(image);
    cv::waitKey(30);
  }
}

Of course, since Qt provides native message dispatch and a Qt event loop by default in a QThread, it's a simple matter to use QObject for the capture process. Below is a complete, tested example.

The capture, conversion and viewer all run in their own threads. Since cv::Mat is an implicitly shared class with atomic, thread-safe access, it's used as such.

The converter has an option of not processing stale frames - useful if conversion is only done for display purposes.

The viewer runs in the gui thread and correctly drops stale frames. There's never a reason for the viewer to deal with stale frames.

If you were to collect data to save to disk, you should run the capture thread at high priority. You should also inspect OpenCV apis to see if there's a way of dumping the native camera data to disk.

To speed up conversion, you could use the gpu-accelerated classes in OpenCV.

The example below makes sure that in none of the memory is reallocated unless necessary for a copy: the Capture class maintains its own frame buffer that is reused for each subsequent frame, so does the Converter, and so does the ImageViewer.

There are two deep copies of image data made (besides whatever happens internally in cv::VideoCatprure::read):

1. The copy to the Converter's QImage.

2. The copy to ImageViewer's QImage.

Both copies are needed to assure decoupling between the threads and prevent data reallocation due to the need to detach a cv::Mat or QImage that has the reference count higher than 1. On modern architectures, memory copies are very fast.

Since all image buffers stay in the same memory locations, their performance is optimal - they stay paged in and cached.

The AddressTracker is used to track memory reallocations for debugging purposes.

// https://github.com/KubaO/stackoverflown/tree/master/questions/opencv-21246766
#include <QtWidgets>
#include <algorithm>
#include <opencv2/opencv.hpp>

Q_DECLARE_METATYPE(cv::Mat)

struct AddressTracker {
   const void *address = {};
   int reallocs = 0;
   void track(const cv::Mat &m) { track(m.data); }
   void track(const QImage &img) { track(img.bits()); }
   void track(const void *data) {
      if (data && data != address) {
         address = data;
         reallocs ++;
      }
   }
};

The Capture class fills the internal frame buffer with the captured frame. It notifies of a frame change. The frame is the user property of the class.

class Capture : public QObject {
   Q_OBJECT
   Q_PROPERTY(cv::Mat frame READ frame NOTIFY frameReady USER true)
   cv::Mat m_frame;
   QBasicTimer m_timer;
   QScopedPointer<cv::VideoCapture> m_videoCapture;
   AddressTracker m_track;
public:
   Capture(QObject *parent = {}) : QObject(parent) {}
   ~Capture() { qDebug() << __FUNCTION__ << "reallocations" << m_track.reallocs; }
   Q_SIGNAL void started();
   Q_SLOT void start(int cam = {}) {
      if (!m_videoCapture)
         m_videoCapture.reset(new cv::VideoCapture(cam));
      if (m_videoCapture->isOpened()) {
         m_timer.start(0, this);
         emit started();
      }
   }
   Q_SLOT void stop() { m_timer.stop(); }
   Q_SIGNAL void frameReady(const cv::Mat &);
   cv::Mat frame() const { return m_frame; }
private:
   void timerEvent(QTimerEvent * ev) {
      if (ev->timerId() != m_timer.timerId()) return;
      if (!m_videoCapture->read(m_frame)) { // Blocks until a new frame is ready
         m_timer.stop();
         return;
      }
      m_track.track(m_frame);
      emit frameReady(m_frame);
   }
};

The Converter class converts the incoming frame to a scaled-down QImage user property. It notifies of the image update. The image is retained to prevent memory reallocations. The processAll property selects whether all frames will be converted, or only the most recent one should more than one get queued up.

class Converter : public QObject {
   Q_OBJECT
   Q_PROPERTY(QImage image READ image NOTIFY imageReady USER true)
   Q_PROPERTY(bool processAll READ processAll WRITE setProcessAll)
   QBasicTimer m_timer;
   cv::Mat m_frame;
   QImage m_image;
   bool m_processAll = true;
   AddressTracker m_track;
   void queue(const cv::Mat &frame) {
      if (!m_frame.empty()) qDebug() << "Converter dropped frame!";
      m_frame = frame;
      if (! m_timer.isActive()) m_timer.start(0, this);
   }
   void process(const cv::Mat &frame) {
      Q_ASSERT(frame.type() == CV_8UC3);
      int w = frame.cols / 3.0, h = frame.rows / 3.0;
      if (m_image.size() != QSize{w,h})
         m_image = QImage(w, h, QImage::Format_RGB888);
      cv::Mat mat(h, w, CV_8UC3, m_image.bits(), m_image.bytesPerLine());
      cv::resize(frame, mat, mat.size(), 0, 0, cv::INTER_AREA);
      cv::cvtColor(mat, mat, CV_BGR2RGB);
      emit imageReady(m_image);
   }
   void timerEvent(QTimerEvent *ev) {
      if (ev->timerId() != m_timer.timerId()) return;
      process(m_frame);
      m_frame.release();
      m_track.track(m_frame);
      m_timer.stop();
   }
public:
   explicit Converter(QObject * parent = nullptr) : QObject(parent) {}
   ~Converter() { qDebug() << __FUNCTION__ << "reallocations" << m_track.reallocs; }
   bool processAll() const { return m_processAll; }
   void setProcessAll(bool all) { m_processAll = all; }
   Q_SIGNAL void imageReady(const QImage &);
   QImage image() const { return m_image; }
   Q_SLOT void processFrame(const cv::Mat &frame) {
      if (m_processAll) process(frame); else queue(frame);
   }
};

The ImageViewer widget is the equivalent of a QLabel storing a pixmap. The image is the user property of the viewer. The incoming image is deep-copied into the user property, to prevent memory reallocations.

class ImageViewer : public QWidget {
   Q_OBJECT
   Q_PROPERTY(QImage image READ image WRITE setImage USER true)
   bool painted = true;
   QImage m_img;
   AddressTracker m_track;
   void paintEvent(QPaintEvent *) {
      QPainter p(this);
      if (!m_img.isNull()) {
         setAttribute(Qt::WA_OpaquePaintEvent);
         p.drawImage(0, 0, m_img);
         painted = true;
      }
   }
public:
   ImageViewer(QWidget * parent = nullptr) : QWidget(parent) {}
   ~ImageViewer() { qDebug() << __FUNCTION__ << "reallocations" << m_track.reallocs; }
   Q_SLOT void setImage(const QImage &img) {
      if (!painted) qDebug() << "Viewer dropped frame!";
      if (m_img.size() == img.size() && m_img.format() == img.format()
          && m_img.bytesPerLine() == img.bytesPerLine())
         std::copy_n(img.bits(), img.sizeInBytes(), m_img.bits());
      else
         m_img = img.copy();
      painted = false;
      if (m_img.size() != size()) setFixedSize(m_img.size());
      m_track.track(m_img);
      update();
   }
   QImage image() const { return m_img; }
};

The demonstration instantiates the classes described above and runs the capture and conversion in dedicated threads.

class Thread final : public QThread { public: ~Thread() { quit(); wait(); } };

int main(int argc, char *argv[])
{
   qRegisterMetaType<cv::Mat>();
   QApplication app(argc, argv);
   ImageViewer view;
   Capture capture;
   Converter converter;
   Thread captureThread, converterThread;
   // Everything runs at the same priority as the gui, so it won't supply useless frames.
   converter.setProcessAll(false);
   captureThread.start();
   converterThread.start();
   capture.moveToThread(&captureThread);
   converter.moveToThread(&converterThread);
   QObject::connect(&capture, &Capture::frameReady, &converter, &Converter::processFrame);
   QObject::connect(&converter, &Converter::imageReady, &view, &ImageViewer::setImage);
   view.show();
   QObject::connect(&capture, &Capture::started, [](){ qDebug() << "Capture started."; });
   QMetaObject::invokeMethod(&capture, "start");
   return app.exec();
}

#include "main.moc"

This concludes the complete example. Note: The previous revision of this answer unnecessarily reallocated the image buffers.

---

Related Topics