Opencv Cv::Mat and Eigen::Matrix

OpenCV CV::Mat and Eigen::Matrix

You should consider using Eigen::Map to wrap OpenCV matrices in order to be used directly by the Eigen SDK.
This allows you to apply almost all functionalities implemented in Eigen on matrix allocated by OpenCV

In particular you simply instantiate an Eigen::Map providing the pointer to the cv::Mat buffer:

//allocate memory for a 4x4 float matrix
cv::Mat cvT(4,4,CV_32FC1); 

//directly use the buffer allocated by OpenCV
Eigen::Map<Matrix4f> eigenT( cvT.data() ); 

for more information on Eigen::Map take a look at
Eigen Tutorial: Map Class

How to use cv::Mat and Eigen::Matrix correctly? (OpenCV + Eigen)

The previous answer only works on GrayScale, this one works on Color.

The key is to reshape the cv::Mat.

cv::Mat::reshape(int new_channel, int new_rows);

---

The result:

---

The code:

#include <iostream>
#include <opencv2/opencv.hpp>
#include <Eigen/Dense>
#include <opencv2/core/eigen.hpp>

int main(int argc, char **argv) {
    cv::Mat image = cv::imread( "Knight.jpg");
    if ( !image.data )
    {
        printf("No image data \n");
        return -1;
    }

    cv::imshow("Source", image);

    int chs = image.channels();

    // (1) display multiplied by (1.0/255)
    {
        cv::Mat img = image.reshape(1, 0);
        std::cout << img.size() << ", " << img.channels() << std::endl;
        typedef Eigen::Matrix<double, Eigen::Dynamic, Eigen::Dynamic> MatrixXd;
        MatrixXd mat;

        cv::cv2eigen(img, mat);
        cv::Mat img2;
        cv::eigen2cv(mat, img2);
        img2 = img2.reshape(chs, 0);
        cv::imshow("doube color image ", img2*(1.0/255));
        cv::imwrite("dst_double.png", img2);

    }

    // (2) change Eigen Matrix type
    {
        cv::Mat img = image.reshape(1, 0);
        std::cout << img.size() << ", " << img.channels() << std::endl;
        typedef Eigen::Matrix<unsigned char, Eigen::Dynamic, Eigen::Dynamic> MatrixXuc;
        MatrixXuc mat;

        cv::cv2eigen(img, mat);
        cv::Mat img2;
        cv::eigen2cv(mat, img2);
        img2 = img2.reshape(chs, 0);
        std::cout << img2.size() << ", " << img2.channels() << std::endl;
        cv::imshow("uchar color image", img2);
        cv::imwrite("dst_uchar.png", img2);
    }

    cv::waitKey(0);
    return 0;
}

how to initialize a Eigen Matrix from an opencv cv::Mat, or from an array which is row-major?

Sample from Conversion between OpenCV and Eigen:

cv::Mat_<float> a = Mat_<float>::ones(2,2);
Eigen::Matrix<float,Dynamic,Dynamic> b;
cv2eigen(a,b);

It is already answered on SO:

//allocate memory for a 4x4 float matrix
cv::Mat cvT(4,4,CV_32FC1); 

//directly use the buffer allocated by OpenCV
Eigen::Map<Matrix4f> eigenT( cvT.data() ); 

and in one more SO post

Map BGR OpenCV Mat to Eigen Tensor

The answer might be disappointing for you.

After going through 12 pages, My conclusion is you have to separate the RGB to individual channel MAT and then convert to eigenmatrix. Or create your own Eigen type and opencv convert function

In OpenCV it is tested like this. It only allows a single channel greyscale image

https://github.com/daviddoria/Examples/blob/master/c%2B%2B/OpenCV/ConvertToEigen/ConvertToEigen.cxx

And in OpenCV it is implemented like this. Which dont give you much option for custom type aka cv::scalar to eigen std::vector

https://github.com/stonier/opencv2/blob/master/modules/core/include/opencv2/core/eigen.hpp

And according to this post,

https://stackoverflow.com/questions/32277887/using-eigen-array-of-arrays-for-rgb-images

I think Eigen was not meant to be used in this way (with vectors as
"scalar" types).

they also have the difficulting in dealing with RGB image in eigen.

Take note that Opencv Scalar and eigen Scalar has a different meaning

It is possible to do so if and only if you use your own datatype aka matrix

So you either choose to store the 3 channel info in 3 eigen matrix and you can use default eigen and opencv routing.

Mat src = imread("img.png",CV_LOAD_IMAGE_COLOR); //load  image
Mat bgr[3];   //destination array
split(src,bgr);//split source    
//Note: OpenCV uses BGR color order
imshow("blue.png",bgr[0]); //blue channel
imshow("green.png",bgr[1]); //green channel
imshow("red.png",bgr[2]); //red channel
Eigen::MatrixXd bm,gm,rm;
cv::cv2eigen(bgr[0], bm); 
cv::cv2eigen(bgr[1], gm); 
cv::cv2eigen(bgr[2], rm);

Or you can define your own type and write you own version of the opencv cv2eigen function

custom eigen type follow this. and it wont be pretty

https://eigen.tuxfamily.org/dox/TopicCustomizing_CustomScalar.html
https://eigen.tuxfamily.org/dox/TopicNewExpressionType.html

Rewrite your own cv2eigen_custom function similar to this

https://github.com/stonier/opencv2/blob/master/modules/core/include/opencv2/core/eigen.hpp

So good luck.

Edit

Since you need tensor. forget about cv function

Mat image;
image = imread(argv[1], CV_LOAD_IMAGE_COLOR); 
Tensor<float, 3> t_3d(image.rows, image.cols, 3);

// t_3d(i, j, k) where i is row j is column and k is channel. 
for (int i = 0; i < image.rows; i++) 
  for (int j = 0; j < image.cols; j++) 
  {
       t_3d(i, j, 0) = (float)image.at<cv::Vec3b>(i,j)[0]; 
       t_3d(i, j, 1) = (float)image.at<cv::Vec3b>(i,j)[1];
       t_3d(i, j, 2) = (float)image.at<cv::Vec3b>(i,j)[2];
       //cv ref Mat.at<data_Type>(row_num, col_num)
  }

watch out for i,j as em not sure about the order. I only write the code based on reference. didnt compile for it.

Also watch out for image type to tensor type cast problem. Some times you might not get what you wanted.

this code should in principle solve your problem

Edit number 2

following the example of this

int storage[128];  // 2 x 4 x 2 x 8 = 128
TensorMap<Tensor<int, 4>> t_4d(storage, 2, 4, 2, 8);

Applied to your case is

cv::Mat frame=imread('myimg.ppm');
TensorMap<Tensor<float, 3>> t_3d(frame.data, image.rows, image.cols, 3);

problem is I'm not sure this will work or not. Even it works, you still have to figure out how the inside data is being organized so that you can get the shape correctly. Good luck

Convert the cv::Mat of OpenCV to Eigen

Instead of using A.cols, use A.step (or A.step[0] or A.step[1]?). If the OpenCV matrix is not continuous, then m.step != m.cols*m.elemSize(). You'll just have to ignore any additional columns, e.g. with A_Eigen.block<0,0>(A.rows, A.cols).

copying elements from an opencv matrix to a eigen matrix

eigenMatrix.resize(i + 1, 9); destroys the content of eigenMatrix.
Since you already know the final dimension at the beginning, just write

Eigen::Matrix<double, Eigen::Dynamic, 9> eigenMatrix;
eigenMatrix.resize(curvatures.size(), 9);

or even just

Eigen::Matrix<double, Eigen::Dynamic, 9> eigenMatrix(curvatures.size(), 9);

before starting the for-loop.

If you need to resize a matrix, but keep the content, you can use conservativeResize() -- but that should be avoided since it requires a full copy for each resizing.

---

Related Topics