Find Corner of Papers

How do I find corners of a paper when there are printed corners/lines on paper itself?

Here is one way to do that in Python/OpenCV.

Note that the found corners are listed counter-clockwise from the top-most corner.

• Read the input
• Convert to gray
• Gaussian blur
• Otsu threshold
• Morphology open/close to clean up the threshold
• Get largest contour
• Approximate a polygon from the contour
• Get the corners
• Draw the polygon on the input
• Compute side lengths
• Compute output corresponding corners
• Get perspective transformation matrix from corresponding corner points
• Warp the input image according to the matrix
• Save the results

Input:

import cv2
import numpy as np

# read image
img = cv2.imread("efile.jpg")

# convert img to grayscale
gray = cv2.cvtColor(img, cv2.COLOR_BGR2GRAY)

# blur image
blur = cv2.GaussianBlur(gray, (3,3), 0)

# do otsu threshold on gray image
thresh = cv2.threshold(blur, 0, 255, cv2.THRESH_BINARY+cv2.THRESH_OTSU)[1]

# apply morphology
kernel = np.ones((7,7), np.uint8)
morph = cv2.morphologyEx(thresh, cv2.MORPH_CLOSE, kernel)
morph = cv2.morphologyEx(morph, cv2.MORPH_OPEN, kernel)

# get largest contour
contours = cv2.findContours(morph, cv2.RETR_EXTERNAL, cv2.CHAIN_APPROX_SIMPLE)
contours = contours[0] if len(contours) == 2 else contours[1]
area_thresh = 0
for c in contours:
    area = cv2.contourArea(c)
    if area > area_thresh:
        area_thresh = area
        big_contour = c

# draw white filled largest contour on black just as a check to see it got the correct region
page = np.zeros_like(img)
cv2.drawContours(page, [big_contour], 0, (255,255,255), -1)

# get perimeter and approximate a polygon
peri = cv2.arcLength(big_contour, True)
corners = cv2.approxPolyDP(big_contour, 0.04 * peri, True)

# draw polygon on input image from detected corners
polygon = img.copy()
cv2.polylines(polygon, [corners], True, (0,0,255), 1, cv2.LINE_AA)
# Alternate: cv2.drawContours(page,[corners],0,(0,0,255),1)

# print the number of found corners and the corner coordinates
# They seem to be listed counter-clockwise from the top most corner
print(len(corners))
print(corners)

# for simplicity get average of top/bottom side widths and average of left/right side heights
# note: probably better to get average of horizontal lengths and of vertical lengths
width = 0.5*( (corners[0][0][0] - corners[1][0][0]) + (corners[3][0][0] - corners[2][0][0]) )
height = 0.5*( (corners[2][0][1] - corners[1][0][1]) + (corners[3][0][1] - corners[0][0][1]) )
width = np.int0(width)
height = np.int0(height)

# reformat input corners to x,y list
icorners = []
for corner in corners:
    pt = [ corner[0][0],corner[0][1] ]
    icorners.append(pt)
icorners = np.float32(icorners)

# get corresponding output corners from width and height
ocorners = [ [width,0], [0,0], [0,height], [width,height] ]
ocorners = np.float32(ocorners)

# get perspective tranformation matrix
M = cv2.getPerspectiveTransform(icorners, ocorners)

# do perspective 
warped = cv2.warpPerspective(img, M, (width, height))

# write results
cv2.imwrite("efile_thresh.jpg", thresh)
cv2.imwrite("efile_morph.jpg", morph)
cv2.imwrite("efile_polygon.jpg", polygon)
cv2.imwrite("efile_warped.jpg", warped)

# display it
cv2.imshow("efile_thresh", thresh)
cv2.imshow("efile_morph", morph)
cv2.imshow("efile_page", page)
cv2.imshow("efile_polygon", polygon)
cv2.imshow("efile_warped", warped)
cv2.waitKey(0)

Thresholded image:

Morphology cleaned image:

Polygon drawn on input:

Extracted Corners (counterclockwise from top right corner)

4

[[[693  67]]
 [[ 23  85]]
 [[ 62 924]]
 [[698 918]]]

Warped Result:

Find corner of papers

Take for reference my original code, which simply detects squares on an image.

That means that in the main method of the application you would write something like the following pseudo-code to call find_squares():

Mat image = imread("test.jpg", 1);

// Detect all regions in the image that are similar to a rectangle
vector<vector<Point> > squares;
find_squares(image, squares);

// The largest of them probably represents the paper
vector<Point> largest_square;
find_largest_square(squares, largest_square);

// Print the x,y coordinates of the square
cout << "Point 1: " << largest_square[0] << endl;
cout << "Point 2: " << largest_square[1] << endl;
cout << "Point 3: " << largest_square[2] << endl;
cout << "Point 4: " << largest_square[3] << endl;

The trick relies on find_largest_square() presented below:

void find_largest_square(const vector<vector<Point> >& squares, vector<Point>& biggest_square)
{
    if (!squares.size())
    {
            // no squares detected
            return;
    }

    int max_width = 0;
    int max_height = 0;
    int max_square_idx = 0;
    const int n_points = 4;

    for (size_t i = 0; i < squares.size(); i++)
    {
            // Convert a set of 4 unordered Points into a meaningful cv::Rect structure.
            Rect rectangle = boundingRect(Mat(squares[i]));

    //        cout << "find_largest_square: #" << i << " rectangle x:" << rectangle.x << " y:" << rectangle.y << " " << rectangle.width << "x" << rectangle.height << endl;

            // Store the index position of the biggest square found
            if ((rectangle.width >= max_width) && (rectangle.height >= max_height))
            {
                    max_width = rectangle.width;
                    max_height = rectangle.height;
                    max_square_idx = i;
            }
    }

    biggest_square = squares[max_square_idx];
}

Algorithm to detect corners of paper sheet in photo

I'm Martin's friend who was working on this earlier this year. This was my first ever coding project, and kinda ended in a bit of a rush, so the code needs some errr...decoding...
I'll give a few tips from what I've seen you doing already, and then sort my code on my day off tomorrow.

First tip, OpenCV and python are awesome, move to them as soon as possible. :D

Instead of removing small objects and or noise, lower the canny restraints, so it accepts more edges, and then find the largest closed contour (in OpenCV use findcontour() with some simple parameters, I think I used CV_RETR_LIST). might still struggle when it's on a white piece of paper, but was definitely providing best results.

For the Houghline2() Transform, try with the CV_HOUGH_STANDARD as opposed to the CV_HOUGH_PROBABILISTIC, it'll give rho and theta values, defining the line in polar coordinates, and then you can group the lines within a certain tolerance to those.

My grouping worked as a look up table, for each line outputted from the hough transform it would give a rho and theta pair. If these values were within, say 5% of a pair of values in the table, they were discarded, if they were outside that 5%, a new entry was added to the table.

You can then do analysis of parallel lines or distance between lines much more easily.

Hope this helps.

Find corner of papers

Take for reference my original code, which simply detects squares on an image.

That means that in the main method of the application you would write something like the following pseudo-code to call find_squares():

Mat image = imread("test.jpg", 1);

// Detect all regions in the image that are similar to a rectangle
vector<vector<Point> > squares;
find_squares(image, squares);

// The largest of them probably represents the paper
vector<Point> largest_square;
find_largest_square(squares, largest_square);

// Print the x,y coordinates of the square
cout << "Point 1: " << largest_square[0] << endl;
cout << "Point 2: " << largest_square[1] << endl;
cout << "Point 3: " << largest_square[2] << endl;
cout << "Point 4: " << largest_square[3] << endl;

The trick relies on find_largest_square() presented below:

void find_largest_square(const vector<vector<Point> >& squares, vector<Point>& biggest_square)
{
    if (!squares.size())
    {
            // no squares detected
            return;
    }

    int max_width = 0;
    int max_height = 0;
    int max_square_idx = 0;
    const int n_points = 4;

    for (size_t i = 0; i < squares.size(); i++)
    {
            // Convert a set of 4 unordered Points into a meaningful cv::Rect structure.
            Rect rectangle = boundingRect(Mat(squares[i]));

    //        cout << "find_largest_square: #" << i << " rectangle x:" << rectangle.x << " y:" << rectangle.y << " " << rectangle.width << "x" << rectangle.height << endl;

            // Store the index position of the biggest square found
            if ((rectangle.width >= max_width) && (rectangle.height >= max_height))
            {
                    max_width = rectangle.width;
                    max_height = rectangle.height;
                    max_square_idx = i;
            }
    }

    biggest_square = squares[max_square_idx];
}

Recognizing corner's page with openCV partialy fails

As fmw42 suggested, you need to restrict the problem more. There are way too many variables to build a "works under all circumstances" solution. A possible, very basic, solution would be to try and get the convex hull of the page.

Another, more robust approach, would be to search for the four vertices of the corners and extrapolate lines to approximate the paper edges. That way you don't need perfect, clean edges, because you would reconstruct them using the four (maybe even three) corners.

To find the vertices you can run Hough Line detector or a Corner Detector on the edges and get at least four discernible clusters of end/starting points. From that you can average the four clusters to get a pair of (x, y) points per corner and extrapolate lines using those points.

That solution would be hypothetical and pretty laborious for a Stack Overflow question, so let me try the first proposal - detection via convex hull. Here are the steps:

1. Threshold the input image
2. Get edges from the input
3. Get the external contours of the edges using a minimum area filter
4. Get the convex hull of the filtered image
5. Get the corners of the convex hull

Let's see the code:

# imports:
import cv2
import numpy as np

# image path
path = "D://opencvImages//"
fileName = "img2.jpg"

# Reading an image in default mode:
inputImage = cv2.imread(path + fileName)

# Deep copy for results:
inputImageCopy = inputImage.copy()

# Convert BGR to grayscale:
grayInput = cv2.cvtColor(inputImageCopy, cv2.COLOR_BGR2GRAY)

# Threshold via Otsu:
_, binaryImage = cv2.threshold(grayInput, 0, 255, cv2.THRESH_BINARY + cv2.THRESH_OTSU)

The first step is to get a binary image, very straightforward. This is the result if you threshold via Otsu:

It is never a good idea to try and segment an object from a textured (or high frequency) background, however, in this case the paper it is discernible in the image histogram and the binary image is reasonably good. Let's try and detect edges on this image, I'm applying Canny with the same parameters as your code:

# Get edges:
cannyImage = cv2.Canny(binaryImage, threshold1=120, threshold2=255, edges=1)

Which produces this:

Seems good enough, the target edges are mostly present. Let's detect contours. The idea is to set an area filter, because the target contour is the biggest amongst the rest. I (heuristically) set a minimum area of 100000 pixels. Once the target contour is found I get its convex hull, like this:

# Find the EXTERNAL contours on the binary image:
contours, hierarchy = cv2.findContours(cannyImage, cv2.RETR_EXTERNAL, cv2.CHAIN_APPROX_SIMPLE)

# Store the corners:
cornerList = []

# Look for the outer bounding boxes (no children):
for i, c in enumerate(contours):

    # Approximate the contour to a polygon:
    contoursPoly = cv2.approxPolyDP(c, 3, True)

    # Convert the polygon to a bounding rectangle:
    boundRect = cv2.boundingRect(contoursPoly)

    # Get the bounding rect's data:
    rectX = boundRect[0]
    rectY = boundRect[1]
    rectWidth = boundRect[2]
    rectHeight = boundRect[3]

    # Estimate the bounding rect area:
    rectArea = rectWidth * rectHeight

    # Set a min area threshold
    minArea = 100000

    # Filter blobs by area:
    if rectArea > minArea:

        # Get the convex hull for the target contour:
        hull = cv2.convexHull(c)
        # (Optional) Draw the hull:
        color = (0, 0, 255)
        cv2.polylines(inputImageCopy, [hull], True, color, 2)

You'll notice I've prepared beforehand a list (cornerList) in which I'll store (hopefully) all the corners. The last two lines of the previous snippet are optional, they draw the convex hull via cv2.polylines, this would be the resulting image:

Still inside the loop, after we compute the convex hull, we will get the corners via cv2.goodFeaturesToTrack, which implements a Corner Detector. The function receives a binary image, so we need to prepare a black image with the convex hull points drawn in white:

        # Create image for good features to track:
        (height, width) = cannyImage.shape[:2]
        # Black image same size as original input:
        hullImg = np.zeros((height, width), dtype =np.uint8)

        # Draw the points:
        cv2.drawContours(hullImg, [hull], 0, 255, 2)
        cv2.imshow("hullImg", hullImg)
        cv2.waitKey(0)

This is the image:

Now, we must set the corner detector. It needs the number of corners you are looking for, a minimum "quality" parameter that discards poor points detected as "corners" and a minimum distance between the corners. Check out the documentation for more parameters. Let's set the detector, it will return an array of points where it detected a corner. After we get this array, we will store each point in our cornerList, like this:

        # Set the corner detection:
        maxCorners = 4
        qualityLevel = 0.01
        minDistance = int(max(height, width) / maxCorners)

        # Get the corners:
        corners = cv2.goodFeaturesToTrack(hullImg, maxCorners, qualityLevel, minDistance)
        corners = np.int0(corners)

        # Loop through the corner array and store/draw the corners:
        for c in corners:

            # Flat the array of corner points:
            (x, y) = c.ravel()
            # Store the corner point in the list:
            cornerList.append((x,y))

            # (Optional) Draw the corner points:
            cv2.circle(inputImageCopy, (x, y), 5, 255, 5)
            cv2.imshow("Corners", inputImageCopy)
            cv2.waitKey(0)
        

Additionally you can draw the corners as circles, it will yield this image:

This is the same algorithm tested on your third image:

Getting the coordinates of vertices of an A4 sheet with coins on it, for its further projective transformation and coin detection

You can try using detectHarrisFeatures on the S color channel of HSV color space:

I was looking for a color space that gets maximum contrast of the paper.

It looks like the saturation color channel of HSV makes a good contrast between the paper and the background.

Image is resized the image by a factor of 0.25, for removing noise.

detectHarrisFeatures finds the 4 corners of the paper, but it might not be robust enough.

You may need to find more features, and find the 4 correct features, using some logic.

Here is a code sample:

%Read input image
I = imread('im.png');

%Remove the margins, and replace them using padding (just because the image is a MATLAB figure)
I = padarray(I(11:end-10, 18:end-17, :), [10, 17], 'both', 'replicate');

HSV = rgb2hsv(I);

%H = HSV(:, :, 1);%figure;imshow(H);title('H');
S = HSV(:, :, 2);%figure;imshow(S);title('S');
%V = HSV(:, :, 3);%figure;imshow(V);title('V');

%Reduce image size by a factor of 0.25 in each axis
S = imresize(S, 0.25);

%S = imclose(S, ones(3)); %May be requiered

%Detect corners
corners = detectHarrisFeatures(S);

imshow(S); hold on;
plot(corners.selectStrongest(4));

Result:

---

Different approach you may try:

• Take a photo without the coins.
• Mark the corners manually, and extract features of the 4 corners.
• Use image matching techniques to match the image with the coins with the image without the coins (mach basted on the 4 corners).

---

Related Topics