Obtaining Weights in Cvsvm, the Svm Implementation of Opencv

Obtaining weights in CvSVM, the SVM implementation of OpenCV

After dealing with it I have been able to obtain the weights. For obtaining the weights one has to obtain first the support vectors and then add them multiplied by the alpha values.

// get the svm weights by multiplying the support vectors by the alpha values
int numSupportVectors = SVM.get_support_vector_count();
const float *supportVector;
const CvSVMDecisionFunc *dec = SVM.decision_func;
svmWeights = (float *) calloc((numOfFeatures+1),sizeof(float));
for (int i = 0; i < numSupportVectors; ++i)
{
    float alpha = *(dec[0].alpha + i);
    supportVector = SVM.get_support_vector(i);
    for(int j=0;j<numOfFeatures;j++)
        *(svmWeights + j) += alpha * *(supportVector+j);
}
*(svmWeights + numOfFeatures) = - dec[0].rho; //Be careful with the sign of the bias!

The only trick here is that the instance variable float *decision_function is protected on the opencv framework, so I had to change it in order to access it.

How to calculate confidence score from CvSVM

just call CvSVM::predict with the returnDFVal param set to true, and it will return distances instead of class-labels

getting primal form from CvSVM trained file

Answer with a test harness. I put in new answer as it would add allot of clutter to the original answer, possibly making it a bit confusing.

//dummy features
std:: vector<float> 
dummyDerReaderForOneDer(const vector<float> &pattern)
{
    int i = std::rand() % pattern.size();
    int j = std::rand() % pattern.size();
    vector<float> patternPulNoise(pattern);
    std::random_shuffle(patternPulNoise.begin()+std::min(i,j),patternPulNoise.begin()+std::max(i,j));
    return patternPulNoise;
};

//extend CvSVM to get access to weights
class mySVM : public CvSVM
{
public:
    vector<float>
    getWeightVector(const int descriptorSize);
};

//get the weights
vector<float>
mySVM::getWeightVector(const int descriptorSize)
{
    vector<float> svmWeightsVec(descriptorSize+1);
    int numSupportVectors = get_support_vector_count();

    //this is protected, but can access due to inheritance rules 
    const CvSVMDecisionFunc *dec = CvSVM::decision_func;

    const float *supportVector;
    float* svmWeight = &svmWeightsVec[0];

    for (int i = 0; i < numSupportVectors; ++i)
    {
        float alpha = *(dec[0].alpha + i);
        supportVector = get_support_vector(i);
        for(int j=0;j<descriptorSize;j++)
        {
            *(svmWeight + j) += alpha * *(supportVector+j);
        }
    }
    *(svmWeight + descriptorSize) = - dec[0].rho;

    return svmWeightsVec;
}

// main harness entry point for detector test
int main (int argc, const char * argv[])
{

    //dummy variables for example
    int posFiles = 10;
    int negFiles = 10;
    int dims = 1000;
    int randomFactor = 4;

    //setup some dummy data
    vector<float> dummyPosPattern;
    dummyPosPattern.assign(int(dims/randomFactor),1.f);
    dummyPosPattern.resize(dims );
    random_shuffle(dummyPosPattern.begin(),dummyPosPattern.end());

    vector<float> dummyNegPattern;
    dummyNegPattern.assign(int(dims/randomFactor),1.f);
    dummyNegPattern.resize(dims );
    random_shuffle(dummyNegPattern.begin(),dummyNegPattern.end());

    // the labels and lables mat
    float posLabel = 1.f;
    float negLabel = 2.f;
    cv::Mat cSvmLabels;

    //the data mat
    cv::Mat cSvmTrainingData;

    //dummy linear svm parmas
    SVMParams cSvmParams;
    cSvmParams.svm_type = cv::SVM::C_SVC;
    cSvmParams.C = 0.0100;
    cSvmParams.kernel_type = cv::SVM::LINEAR;
    cSvmParams.term_crit =  cv::TermCriteria(CV_TERMCRIT_ITER+CV_TERMCRIT_EPS, 1000000, FLT_EPSILON);

    cout << "creating training data. please wait" << endl;
    int i;
    for(i=0;i<posFiles;i++)
    {
        //your feature for one box from file
        vector<float> d = dummyDerReaderForOneDer(dummyPosPattern);

        //push back a new mat made from the vectors data, with copy  data flag on
        //this shows the format of the mat for a single example, (1 (row) X dims(col) ), as  training mat has each **row** as an example;
        //the push_back works like vector add adds each example to the bottom of the matrix
        cSvmTrainingData.push_back(cv::Mat(1,dims,CV_32FC1,d.data(),true));

        //push back a pos label to the labels mat
        cSvmLabels.push_back(posLabel);
    }

    //do same with neg files;
    for(i=0;i<negFiles;i++)
    {
        float a =  rand(); 
        vector<float> d = dummyDerReaderForOneDer(dummyNegPattern);
        cSvmTrainingData.push_back(cv::Mat(1,dims,CV_32FC1,d.data(),true));
        cSvmLabels.push_back(negLabel);
    }

    //have a look
    cv::Mat viz;
    cSvmTrainingData.convertTo(viz,CV_8UC3);
    viz = viz*255;
    cv::imshow("svmData", viz);
    cv::waitKey(10);
    cout << "press any key to continue" << endl;
    getchar();

    viz.release();

    //create the svm;
    cout << "training, please wait" << endl;

    mySVM svm;
    svm.train(cSvmTrainingData,cSvmLabels,cv::Mat(),cv::Mat(),cSvmParams);

    cout << "get weights" << endl; 
    vector<float> svmWeights = svm.getWeightVector(dims);
    for(i=0; i<dims+1; i++)
    {
        cout << svmWeights[i] << ", ";
        if(i==dims)
        {
            cout << endl << "bias: "  << svmWeights[i]  << endl;
        }
    }

    cout << "press any key to continue" << endl;
    getchar();

    cout << "testing, please wait" << endl;
    //test the svm with a large amount of new unseen fake one at a time
    int totExamples = 10;
    int k;
    for(i=0;i<totExamples; i++)
    {
        cout << endl << endl;
        vector<float> dPos = dummyDerReaderForOneDer(dummyPosPattern);
        cv::Mat dMatPos(1,dims,CV_32FC1,dPos.data(),true);
        float predScoreFromDual = svm.predict(dMatPos,true);
        float predScoreBFromPrimal = svmWeights[dims];

        for( k = 0; k <= dims - 4; k += 4 )
                predScoreBFromPrimal += dPos[k]*svmWeights[k] + dPos[k+1]*svmWeights[k+1] +
                    dPos[k+2]*svmWeights[k+2] + dPos[k+3]*svmWeights[k+3];
            for( ; k < dims; k++ )
                predScoreBFromPrimal += dPos[k]*svmWeights[k];

         cout << "Dual Score:\t" << predScoreFromDual << "\tPrimal Score:\t" <<  predScoreBFromPrimal << endl; 

    }

    cout << "press any key to continue" << endl;
    getchar();
    return(0);
} 

Can SVM solution change after shuffling the inputs?

I am not familiar with the OpenCV implementation. But do this: run several trials on exactly the same data set -- no shuffling, same order, same data points. See if the SVM changes. Obviously, in theory, it shouldn't. But it could be that there is some small randomization step somewhere in the implementation that produces different outputs for the same input.

Edit: As Chris A. asks, do the feature vectors correspond to their proper labels after shuffling? If not, that would obviously destroy your results.

How to find Mat file consisting of hog features

this sample should contain the cases that we have discussed above in the comments.

//dummy readers
std:: vector<float> 
dummyDerReaderForOneDer(const vector<float> &pattern)
{
    int i = std::rand() % pattern.size();
    int j = std::rand() % pattern.size();
    vector<float> patternPulNoise(pattern);
    std::random_shuffle(patternPulNoise.begin()+std::min(i,j),patternPulNoise.begin()+std::max(i,j));
    return patternPulNoise;
};

std:: vector<float> 
dummyDerReaderForManyDers(const vector<float> &posPattern,
                                const vector<float> &negPattern, 
                                    vector<float> labels,
                                        float posLabel, float negLabel)
{

    int i,j;
    vector<float> allPatternsInOneVector;

    for(i=0;i< labels.size(); i++)
    {
        vector<float> patternPulNoise;
        if(labels[i]==posLabel)
        {
            patternPulNoise  = dummyDerReaderForOneDer(posPattern);
        }
        if(labels[i]==negLabel)
        {
            patternPulNoise  = dummyDerReaderForOneDer(negPattern);
        }

        for(j=0;j< patternPulNoise.size(); j++)
        {
            allPatternsInOneVector.push_back(patternPulNoise[j]);
        }
    }
    return allPatternsInOneVector;
}

// main harness entry point for detector test
int main (int argc, const char * argv[])
{

    //dummy variables for example
    int posFiles = 128;
    int negFiles = 128;
    int dims = 3780;

    //setup some dummy data
    vector<float> dummyPosPattern;
    dummyPosPattern.assign(1000,1.f);
    dummyPosPattern.resize(dims );
    random_shuffle(dummyPosPattern.begin(),dummyPosPattern.end());

    vector<float> dummyNegPattern;
    dummyNegPattern.assign(1000,1.f);
    dummyNegPattern.resize(dims );
    random_shuffle(dummyNegPattern.begin(),dummyNegPattern.end());

    // the labels and lables mat
    float posLabel = 1.f;
    float negLabel = 2.f;
    cv::Mat cSvmLabels;

    //the data mat
    cv::Mat cSvmTrainingData;

    //dummy linear svm parmas
    SVMParams cSvmParams;
    cSvmParams.svm_type = cv::SVM::C_SVC;
    cSvmParams.C = 0.0100;
    cSvmParams.kernel_type = cv::SVM::LINEAR;
    cSvmParams.term_crit =  cv::TermCriteria(CV_TERMCRIT_ITER+CV_TERMCRIT_EPS, 1000000, FLT_EPSILON);

    cout << "creating training data. please wait" << endl;
    int i;
    for(i=0;i<posFiles;i++)
    {
        //your feature for one box from file
        vector<float> d = dummyDerReaderForOneDer(dummyPosPattern);

        //push back a new mat made from the vectors data, with copy  data flag on
        //this shows the format of the mat for a single example, (1 (row) X dims(col) ), as  training mat has each **row** as an example;
        //the push_back works like vector add adds each example to the bottom of the matrix
        cSvmTrainingData.push_back(cv::Mat(1,dims,CV_32FC1,d.data(),true));

        //push back a pos label to the labels mat
        cSvmLabels.push_back(posLabel);
    }

    //do same with neg files;
    for(i=0;i<negFiles;i++)
    {
        float a =  rand(); 
        vector<float> d = dummyDerReaderForOneDer(dummyNegPattern);
        cSvmTrainingData.push_back(cv::Mat(1,dims,CV_32FC1,d.data(),true));
        cSvmLabels.push_back(negLabel);
    }

    //have a look
    cv::Mat viz;
    cSvmTrainingData.convertTo(viz,CV_8UC3);
    viz = viz*255;
    cv::imshow("svmData", viz);
    cv::waitKey(10);
    cout << "press any key to continue" << endl;
    getchar();

    viz.release();

    //create the svm;
    cout << "training, please wait" << endl;

    CvSVM svm;
    svm.train(cSvmTrainingData,cSvmLabels,cv::Mat(),cv::Mat(),cSvmParams);

    cout << "testing, please wait" << endl;
    //test the svm with a large amount of new unseen fake one at a time
    int totExamples = 10000;
    float TP(0.f), FP(0.f), FN(0.f), TN(0.f);
    for(i=0;i<totExamples; i++)
    {
        vector<float> dPos = dummyDerReaderForOneDer(dummyPosPattern);
        cv::Mat dMatPos(1,dims,CV_32FC1,dPos.data(),true);
        float predLabelPos = svm.predict(dMatPos);

        if(predLabelPos == posLabel) TP++;
        else(FN++);

        vector<float> dNeg = dummyDerReaderForOneDer(dummyNegPattern);
        cv::Mat dMatNeg(1,dims,CV_32FC1,dNeg.data(),true);
        float predLabelNeg = svm.predict(dMatNeg);

        if(predLabelNeg == negLabel) TN++;
        else(FP++);

        if(i%1000==0)
            cout << "testing " << i << "of " <<  totExamples << endl; 
    }

    //http://en.wikipedia.org/wiki/Precision_and_recall
    float sensitivity  = TP / (TP + FN);
    float specificity = TN / (TN + FP);
    float precision = TP / (TP + FP);

    cout << "sensitivity: " <<  sensitivity << endl;
    cout << "specificity: " <<  specificity << endl;
    cout << "precision: " <<  precision << endl;

    cout << "press any key to continue" << endl;
    getchar();

    cout << "creating test data, please wait" << endl;
    //test the svm with a large amount of new unseen fake data all at one time
    TP=FP=FN=TN = 0.f;
    vector<float> testLabels;
    for(i=0;i<int(totExamples/2);i++)
    {
        testLabels.push_back(posLabel);
    }
    for(i;i<totExamples;i++)
    {
        testLabels.push_back(negLabel);
    }

    vector<float> allExamples = dummyDerReaderForManyDers(dummyPosPattern,dummyNegPattern,testLabels,posLabel,negLabel);

    //on big mat
    cv::Mat allExamplesMat(1,allExamples.size(),CV_32FC1,allExamples.data(),true);

    //need reshape to one example per row

    allExamplesMat = allExamplesMat.reshape(0,totExamples);

    //have a look
    allExamplesMat.convertTo(viz,CV_8UC3);
    viz = viz*255;
    cv::imshow("testData", viz);
    cv::waitKey(10);
    cout << "press any key to continue" << endl;
    getchar();
    viz.release();

    //test them all at once ( uses intel tbb :)
    cout << "predict all at once, please wait" << endl;  
    cv::Mat predLabels_mat;
    svm.predict(allExamplesMat,predLabels_mat);

    //evaluate
    for(i=0;i<testLabels.size();i++)
    {
        float testLabel = testLabels.at(i);
        float predLabel = predLabels_mat.at<float>(i);
        if(testLabel==predLabel)
        {
            if(testLabel==posLabel) TP++;
            else TN++;

        }
        else
        {
            if(testLabel==posLabel) FP++;
            else FN++;

        }

    }

    //http://en.wikipedia.org/wiki/Precision_and_recall
    sensitivity  = TP / (TP + FN);
    specificity = TN / (TN + FP);
    precision = TP / (TP + FP);

    cout << "sensitivity: " <<  sensitivity << endl;
    cout << "specificity: " <<  specificity << endl;
    cout << "precision: " <<  precision << endl;

    cout << "press any key to continue" << endl;
    getchar();
    return(0);
}

Feeding HOG into SVM: the HOG has 9 bins, but the SVM takes in a 1D matrix

The SVM always takes in a single row of data per feature vector. The dimensionality of the feature vector is thus the length of the row. If you're dealing with 2D data, then there are 2 items per feature vector. Example of 2D data is on this webpage:

http://www.csie.ntu.edu.tw/~cjlin/libsvm/

code of an equivalent demo in OpenCV http://sites.google.com/site/btabibian/labbook/svmusingopencv

The point is that even though you're thinking of the histogram as 2D with 9-bin cells, the feature vector is in fact the flattened version of this. So it's correct to flatten it out into a long feature vector. The result for me was a feature vector of length 2304 (16x16x9) and I get 100% prediction accuracy on a small test set (i.e. it's probably slightly less than 100% but it's working exceptionally well).

The reason this works is that the SVM is working on a system of weights per item of the feature vector. So it doesn't have anything to do with the problem's dimension, the hyperplane is always in the same dimension as the feature vector. Another way of looking at it is to forget about the hyperplane and just view it as a bunch of weights for each item in the feature vector. In this case, it needs one weighting for every item, then it multiplies each item by its weighting and outputs the result.

---

Related Topics