Arkit: How to Tell If User's Face Is Parallel to Camera

ARKit: How to tell if user's face is parallel to camera

Assuming you are using ARFaceTrackingConfiguration in your app, you can actually retrieve the transforms of both the ARFaceAnchor and the camera to determine their orientations. You can get a simd_float4x4 matrix of the head's orientation in world space by using ARFaceAnchor.transform property. Similarly, you can get the transform of the SCNCamera or ARCamera of your scene.

To compare the camera's and face's orientations relative to each other in a SceneKit app (though there are similar functions on the ARKit side of things), I get the world transform for the node that is attached to each of them, let's call them faceNode attached to the ARFaceAnchor and cameraNode representing the ARSCNView.pointOfView. To find the angle between the camera and your face, for example, you could do something like this:

let faceOrientation: simd_quatf = faceNode.simdWorldTransform
let cameraOrientation: simd_quatf = cameraNode.simdWorldTransform
let deltaOrientation: simd_quatf = faceOrientation.inverse * cameraOrientation

By looking at deltaOrientation.angle and deltaOrientation.axis you can determine the relative angles on each axis between the face and the camera. If you do something like deltaOrientation.axis * deltaOrientation.angles, you have a simd_float3 vector giving you a sense of the pitch, yaw and roll (in radians) of the head relative to the camera.

There are a number of ways you can do this using the face anchor and camera transforms, but this simd quaternion method works quite well for me. Hope this helps!

Head tilt position ARKit - iOS

The documentation states

The inherited transform property describes the face’s current position
and orientation in world coordinates; that is, in a coordinate space
relative to that specified by the worldAlignment property of the
session configuration.

And so if you're using ARWorldAlignmentGravity or ARWorldAlignmentGravityAndHeading (« The coordinate system's y-axis is parallel to gravity ») you can determine whether the head is tilted or not.

How to detect vertical planes in ARKit?

Edit: This is now supported as of ARKit 1.5 (iOS 11.3). Simply use .vertical. I have kept the previous post below for historical purposes.


TL;DR

Vertical plane detection is not (yet) a feature that exists in ARKit. The .horizontal suggests that this feature could be being worked on and might be added in the future. If it was just a Boolean value, this would suggest that it is final.

Confirmation

This suspicion was confirmed by a conversation that I had with an Apple engineer at WWDC17.

Explanation

You could argue that creating an implementation for this would be difficult as there are infinitely many more orientations for a vertical plane rather than a horizontal one, but as rodamn said, this is probably not the case.

From rodamn’s comment:
At its simplest, a plane is defined to be three coplanar points. You have a surface candidate once there are sufficient detected coplanar features detected along a surface (vertical, horizontal, or at any arbitrary angle). It's just that the normal for horizontals will be along the up/down axis, while vertical's normals will be parallel to the ground plane. The challenge is that unadorned drywall tends to generate few visual features, and plain walls may often go undetected. I strongly suspect that this is why the .vertical feature is not yet released.

However, there is a counter argument to this. See comments from rickster for more information.

What is the orientation of ARKit's camera space?

Okay, so my initial diagrams were incorrect, as @Mart10 correctly pointed out. Landscape right in the ARKit Camera context means the phone is in landscape with the home button on the right side. This post explains this in greater detail.

This means the camera coordinate system convention is, when in landscape with the home button on the right side: +x: right, +y: up, +z: backward.

This is different from the "sensor coordinate system", which is the coordinate system that the x, y, and z geomagnetic data values from Core Location come in. The "sensor coordinate system" is, when in landscape with the home button on the right side: +x: up, +y: left, +z: backward. You can use a transform to go between them.



Related Topics



Leave a reply



Submit