Wrong Image Orientation After Displaytransform Call

CapturedImage in ARFrame to portrait/vertical

There's an API for that.

ARFrame.displayTransform(for:viewportSize:) gives you a transform matrix that rotates from the image's sensor-native orientation to whatever UI orientation you specify, and provides a scale factor to fit a viewport whose aspect ratio doesn't necessarily match that of the camera.

You can then use that transform in whatever display pipeline you're working with to show the rotated image. For example:

• check the Xcode project template for a Metal-based ARKit app to see how they feed that transform matrix to a vertex shader.

• create a Core Image CIImage from the pixel buffer, then use transformed(by:) to get a rotated image, which you can then convert to UIImage and display in UIImageView.

CapturedImage in ARFrame to portrait/vertical

There's an API for that.

ARFrame.displayTransform(for:viewportSize:) gives you a transform matrix that rotates from the image's sensor-native orientation to whatever UI orientation you specify, and provides a scale factor to fit a viewport whose aspect ratio doesn't necessarily match that of the camera.

You can then use that transform in whatever display pipeline you're working with to show the rotated image. For example:

• check the Xcode project template for a Metal-based ARKit app to see how they feed that transform matrix to a vertex shader.

• create a Core Image CIImage from the pixel buffer, then use transformed(by:) to get a rotated image, which you can then convert to UIImage and display in UIImageView.

iOS how to correctly handle orientation when capturing video using AVAssetWriter

Video orientation is handled by the AVAssetWriterInput.transform, looks like the getVideoTransform() implementation is not correct - CGAffineTransform expects the rotation angle to be in radians, so need to change to something like this:

private func getVideoTransform() -> CGAffineTransform {
    switch UIDevice.current.orientation {
        case .portrait:
            return .identity
        case .portraitUpsideDown:
            return CGAffineTransform(rotationAngle: .pi)
        case .landscapeLeft:
            return CGAffineTransform(rotationAngle: .pi/2)
        case .landscapeRight:
            return CGAffineTransform(rotationAngle: -.pi/2)
        default:
            return .identity
        }
    }

---

From Apple Technical Q&A:
https://developer.apple.com/library/archive/qa/qa1744/_index.html

If you are using an AVAssetWriter object to write a movie file, you
can use the transform property of the associated AVAssetWriterInput to
specify the output file orientation. This will write a display
transform property into the output file as the preferred
transformation of the visual media data for display purposes. See the
AVAssetWriterInput.h interface file for the details.

iOS ARKit how to save ARFrame .capturedImage to file?

There are a couple of issues I encountered when trying to pull a screenshot from an ARFrame:

1. The CVPixelBuffer underlined in the ARFrame is rotated to the left, which means that you will need to rotate the image.
2. The frame itself is much wider than what you see in the camera. This is done because AR needs a wider range of image to be able to see beyond what you see for it's scene.

E.G:
This is an image taken from a raw ARFrame

The following below may assist you to handle those two steps easily:
https://github.com/Rightpoint/ARKit-CoreML/blob/master/Library/UIImage%2BUtilities.swift

The problem with these are that this code uses UIGraphicsBeginImageContextWithOptions, which is heavy util.

But if performance are not your main issue(Let's say for an AR Experience) these might be suffice.

If performance is an issue, you can do the mathematical calculation to understand the conversion you will need to do, and simply cut the image needed from the original picture and rotate that.

Hope this will assist you!

Convert ARFrame's captured image to UIImage orientation issue

1. Is there another way to convert the image with the correct orientation?

You may try to use snapshot() of ARSCNView (inherited from SCNView), which:

Draws the contents of the view and returns them as a new image object

so if you have an object like:

@IBOutlet var arkitSceneView:ARSCNView!

you only need to do so:

let imageFromArkitScene:UIImage? = arkitSceneView.snapshot()

---

2. Why does this happen?

It's because the CVPixelBuffer comes from ARFrame, which is :

captured (continuously) from the device camera, by the running AR session.

Well, since the camera orientation does not change with the rotation of the device (they are separate), to be able to adjust the orientation of your frame to the current view, you should re-orient the image captured from your camera applying the affine transform extracted with displayTransform(for:viewportSize:):

Returns an affine transform for converting between normalized image coordinates and a coordinate space appropriate for rendering the camera image onscreen.

you may find good documentation here, usage example:

let orient = UIApplication.shared.statusBarOrientation
let viewportSize = yourSceneView.bounds.size
let transform = frame.displayTransform(for: orient, viewportSize: viewportSize).inverted()
var finalImage = CIImage(cvPixelBuffer: pixelBuffer).transformed(by: transform)

Stream a display, transform it, and display it in a window?

It looks like you can do most of the work with Core Image:

1. The handler of the CGDisplayStream gives you an IOSurface. You can wrap that inside a CIImage with CIImage(ioSurface:).
2. Apply any (or multiple) CGAffineTransform to the CIImage using image.transformed(by:).
3. Use a CIContext to render the resulting image into an MTKView. Maybe you can get some inspiration from my example here.

Note: I'm not 100% sure if Core Image would properly retain the IOSurface for you when you wrap it in a CIImage. You should double-check if you still need to perform any of the actions described in the handler documentation.

How to rotate JPEG images based on the orientation metadata?

If you want to rotate your images, I would suggest to use the metadata extractor library http://code.google.com/p/metadata-extractor/. You can get the image information with the following code:

// Inner class containing image information
public static class ImageInformation {
    public final int orientation;
    public final int width;
    public final int height;

    public ImageInformation(int orientation, int width, int height) {
        this.orientation = orientation;
        this.width = width;
        this.height = height;
    }

    public String toString() {
        return String.format("%dx%d,%d", this.width, this.height, this.orientation);
    }
}

public static ImageInformation readImageInformation(File imageFile)  throws IOException, MetadataException, ImageProcessingException {
    Metadata metadata = ImageMetadataReader.readMetadata(imageFile);
    Directory directory = metadata.getFirstDirectoryOfType(ExifIFD0Directory.class);
    JpegDirectory jpegDirectory = metadata.getFirstDirectoryOfType(JpegDirectory.class);

    int orientation = 1;
    try {
        orientation = directory.getInt(ExifIFD0Directory.TAG_ORIENTATION);
    } catch (MetadataException me) {
        logger.warn("Could not get orientation");
    }
    int width = jpegDirectory.getImageWidth();
    int height = jpegDirectory.getImageHeight();

    return new ImageInformation(orientation, width, height);
}

Then given the orientation you retrieve, you can rotate and/or flip the image to the right orientation. The Affine transform for the EXIF orientation is given by the following method:

// Look at http://chunter.tistory.com/143 for information
public static AffineTransform getExifTransformation(ImageInformation info) {

    AffineTransform t = new AffineTransform();

    switch (info.orientation) {
    case 1:
        break;
    case 2: // Flip X
        t.scale(-1.0, 1.0);
        t.translate(-info.width, 0);
        break;
    case 3: // PI rotation 
        t.translate(info.width, info.height);
        t.rotate(Math.PI);
        break;
    case 4: // Flip Y
        t.scale(1.0, -1.0);
        t.translate(0, -info.height);
        break;
    case 5: // - PI/2 and Flip X
        t.rotate(-Math.PI / 2);
        t.scale(-1.0, 1.0);
        break;
    case 6: // -PI/2 and -width
        t.translate(info.height, 0);
        t.rotate(Math.PI / 2);
        break;
    case 7: // PI/2 and Flip
        t.scale(-1.0, 1.0);
        t.translate(-info.height, 0);
        t.translate(0, info.width);
        t.rotate(  3 * Math.PI / 2);
        break;
    case 8: // PI / 2
        t.translate(0, info.width);
        t.rotate(  3 * Math.PI / 2);
        break;
    }

    return t;
}

The rotation of the image would be done by the following method:

public static BufferedImage transformImage(BufferedImage image, AffineTransform transform) throws Exception {

    AffineTransformOp op = new AffineTransformOp(transform, AffineTransformOp.TYPE_BICUBIC);

    BufferedImage destinationImage = op.createCompatibleDestImage(image, (image.getType() == BufferedImage.TYPE_BYTE_GRAY) ? image.getColorModel() : null );
    Graphics2D g = destinationImage.createGraphics();
    g.setBackground(Color.WHITE);
    g.clearRect(0, 0, destinationImage.getWidth(), destinationImage.getHeight());
    destinationImage = op.filter(image, destinationImage);
    return destinationImage;
}

In a server environment, don't forget to run with -Djava.awt.headless=true

---

Related Topics