Implement a Crosshair Kind Behaviour in Realitykit

Implement a crosshair kind behaviour in RealityKit

Try the following solution:

import ARKit
import RealityKit

class ViewController: UIViewController {
    
    @IBOutlet var arView: ARView!
    var sphere: ModelEntity?
    
    override func touchesBegan(_ touches: Set<UITouch>, with event: UIEvent?) {
        let touch = arView.center
        let results: [CollisionCastHit] = arView.hitTest(touch)
        
        if let result: CollisionCastHit = results.first {
            if result.entity.name == "Cube" && sphere?.isAnchored == true {
                print("BOOM!")
            }
        }
    }
    
    override func viewDidLoad() {
        super.viewDidLoad()
        // Crosshair
        let mesh01 = MeshResource.generateSphere(radius: 0.01)
        sphere = ModelEntity(mesh: mesh01)
        sphere?.transform.translation.z = -0.15
        let cameraAnchor = AnchorEntity(.camera)
        sphere?.setParent(cameraAnchor)
        arView.scene.addAnchor(cameraAnchor)
        
        // Model for collision
        let mesh02 = MeshResource.generateBox(size: 0.3)
        let box = ModelEntity(mesh: mesh02, materials: [SimpleMaterial()])
        box.generateCollisionShapes(recursive: true)
        box.name = "Cube"
        let planeAnchor = AnchorEntity(.plane(.any, 
                              classification: .any, 
                               minimumBounds: [0.2, 0.2]))
        box.setParent(planeAnchor)
        arView.scene.addAnchor(planeAnchor)
    }
}

Applying downward force to an object using RealityKit

Ollie trick

In real life, if you shift your entire body's weight to the nose of the skateboard's deck (like doing the Ollie Maneuver), the skateboard's center of mass shifts from the middle towards the point where the force is being applied. In RealityKit, if you need to tear the rear (front) wheels of the skateboard off the floor, move the model's center of mass towards the slope.

The repositioning of the center of mass occurs in a local coordinate system.

import SwiftUI
import RealityKit

struct ContentView : View {
    var body: some View {
        ARViewContainer().ignoresSafeArea()
    }
}

struct ARViewContainer: UIViewRepresentable {
    
    func makeUIView(context: Context) -> ARView {
        
        let arView = ARView(frame: .zero)
        arView.debugOptions = .showPhysics         // shape visualization
        
        let scene = try! Experience.loadScene()
        let name = "skateboard_01_base_stylized_lod0"
        typealias ModelPack = ModelEntity & HasPhysicsBody & HasCollision
        
        let model = scene.findEntity(named: name) as! ModelPack
        model.physicsBody = .init()
        model.generateCollisionShapes(recursive: true)
        model.physicsBody?.massProperties.centerOfMass.position = [0, 0,-27]

        arView.scene.anchors.append(scene)
        return arView
    }
    
    func updateUIView(_ uiView: ARView, context: Context) { }
}

Physics shape

The second problem that you need to solve is to replace the model's box shape of the physical body (RealityKit and Reality Composer generate this type of shape by default). Its shape cannot be in the form of a monolithic box, it's quite obvious, because the box-shaped form does not allow the force to be applied appropriately. You need a shape similar to the outline of the model.

So, you can use the following code to create a custom shape:

(four spheres for wheels and box for deck)

let shapes: [ShapeResource] = [ 
                                .generateBox(size: [ 20, 4, 78])
                                    .offsetBy(translation: [ 0.0, 11, 0.0]),
                                .generateSphere(radius: 3.1)
                                    .offsetBy(translation: [ 7.5, 3, 21.4]),
                                .generateSphere(radius: 3.1)
                                    .offsetBy(translation: [ 7.5, 3,-21.4]),
                                .generateSphere(radius: 3.1)
                                    .offsetBy(translation: [-7.5, 3, 21.4]),
                                .generateSphere(radius: 3.1)
                                    .offsetBy(translation: [-7.5, 3,-21.4])
                              ]

// model.physicsBody = PhysicsBodyComponent(shapes: shapes, mass: 4.5)

model.collision = CollisionComponent(shapes: shapes)

P.S.

Reality Composer model's settings (I used Xcode 14.0 RC 1).

I want to addChild the Entity created by the RealityComposer

I solved by myself.

The reason is because I added the scene as child.

This code is to load the scene created by RealityComposer.
TwitterCard was scene name.

let twitterCard = try! QRScene.loadTwitterCard()

Scene name is presented in here.

So, correctly ModelEntity's name is here.
I needed to use this.

This load correctly.

addChild(twitterCard.cardObject!)

Only write this, it could add child Entity created by RealityComposer.

I have problem yet.

I can’t post notification for moving motion created by RealityComposer.

Because of twitterCard is not added to ARView’s scene. If can’t do this, we must write a lot of animation codes.

When needed, I create another post, thanks.

Is this RealityKit stuff poorly conceived or am I missing something?

Let's assume that we've created three scenes in Reality Composer called BoxScene for horizontal plane detection (world tracking), StarScene for vertical plane detection (world tracking) and PrismScene for image detection (image tracking) respectively. In each scene we gave names to our models – there are automatic variables generated from these names – goldenBox, plasticStar and paintedPrism.

To switch from World Tracking config to Image Tracking config in RealityKit we must use definite AnchorEntity's initializers written inside buttons' @IBActions – .image and .plane.

Look at the following code to find out how to do what you want.

import RealityKit
import UIKit

class ViewController: UIViewController {

    @IBOutlet var arView: ARView!
    
    let cubeScene = try! Experience.loadBoxScene()
    let starScene = try! Experience.loadStarScene()
    let prismScene = try! Experience.loadPrismScene()

    // IMAGE TRACKING
    @IBAction func image(_ button: UIButton) {
        
        arView.scene.anchors.removeAll()
        
        let anchor = AnchorEntity(.image(group: "AR Resources", 
                                          name: "image"))
        
        let prism = prismScene.paintedPrism!
        anchor.addChild(prism)
        arView.scene.anchors.append(anchor)
    }

    
    // WORLD TRACKING
    @IBAction func verticalAndHorizontal(_ button: UIButton) {
        
        arView.scene.anchors.removeAll()
        
        let trackingAnchor = AnchorEntity(.plane([.vertical, .horizontal],
                                  classification: .any,
                                   minimumBounds: [0.1, 0.1]))
         
        let cube = cubeScene.goldenBox!
        let star = starScene.plasticStar!
        
        if trackingAnchor.anchor?.anchoring.target == .some(.plane([.vertical, 
                                                                    .horizontal], 
                                                    classification: .any, 
                                                     minimumBounds: [0.1, 0.1])) {
            
            let anchor1 = AnchorEntity(.plane(.horizontal,
                              classification: .any,
                               minimumBounds: [0.1, 0.1]))
            
            anchor1.addChild(cube)
            arView.scene.anchors.append(anchor1)
        }
        
        if trackingAnchor.anchor?.anchoring.target == .some(.plane([.vertical, 
                                                                    .horizontal], 
                                                    classification: .any, 
                                                     minimumBounds: [0.1, 0.1])) {
            
            let anchor2 = AnchorEntity(.plane(.vertical,
                              classification: .any,
                               minimumBounds: [0.1, 0.1]))

            anchor2.addChild(star)
            arView.scene.anchors.append(anchor2)
        }           
    }
}

P. S.

At the moment I have no computer with me, I've written it on iPhone. So I don't know if there are any errors in this code...

How to Add Material to ModelEntity programatically in RealityKit?

Updated: June 14, 2022

RealityKit materials

There are 6 types of materials in RealityKit 2.0 and RealityFoundation at the moment:

• SimpleMaterial
• UnlitMaterial
• OcclusionMaterial (read this post to find out how to setup SceneKit occlusion shader)
• VideoMaterial (look at this post to find out how to setup it)
• PhysicallyBasedMaterial
• CustomMaterial

To apply these materials use the following logic:

import Cocoa
import RealityKit

class ViewController: NSViewController {        
    @IBOutlet var arView: ARView!
    
    override func awakeFromNib() {
        let box = try! Experience.loadBox()
        
        var simpleMat = SimpleMaterial()
        simpleMat.color = .init(tint: .blue, texture: nil)
        simpleMat.metallic = .init(floatLiteral: 0.7)
        simpleMat.roughness = .init(floatLiteral: 0.2)
        
        var pbr = PhysicallyBasedMaterial()
        pbr.baseColor = .init(tint: .green, texture: nil) 

        let mesh: MeshResource = .generateBox(width: 0.5, 
                                             height: 0.5, 
                                              depth: 0.5, 
                                       cornerRadius: 0.02, 
                                         splitFaces: true)

        let boxComponent = ModelComponent(mesh: mesh,
                                     materials: [simpleMat, pbr])

        box.steelBox?.children[0].components.set(boxComponent)
        box.steelBox?.orientation = Transform(pitch: .pi/4, 
                                                yaw: .pi/4, 
                                               roll: 0).rotation
        arView.scene.anchors.append(box)
    }
}

Read this post to find out how to load a texture for RealityKit's shaders.

How to create RealityKit's shaders similar to SceneKit's shaders

We know that in SceneKit there are 5 different shading models, so we can use RealityKit's SimpleMaterial, PhysicallyBasedMaterial and UnlitMaterial to generate all these five shaders that we've been accustomed to.

Let's see how it looks like:

SCNMaterial.LightingModel.blinn           – SimpleMaterial(color: . gray,
                                                        roughness: .float(0.5),
                                                       isMetallic: false)

SCNMaterial.LightingModel.lambert         – SimpleMaterial(color: . gray,
                                                        roughness: .float(1.0),
                                                       isMetallic: false)
  
SCNMaterial.LightingModel.phong           – SimpleMaterial(color: . gray,
                                                        roughness: .float(0.0),
                                                       isMetallic: false)

SCNMaterial.LightingModel.physicallyBased – PhysicallyBasedMaterial()

// all three shaders (`.constant`, `UnlitMaterial` and `VideoMaterial `) 
// don't depend on lighting
SCNMaterial.LightingModel.constant        – UnlitMaterial(color: .gray)
                                          – VideoMaterial(avPlayer: avPlayer)

RealityKit – What is `steelBox` instance loading?

A generated content in Experience.swift file is directly linked to scenes coming from Reality Composer. steelBox property is a "key" to model entity inside RC scene. It's made for developer's convenience to get a quick access to a desired model.

The string "Steel Box" is not compulsory, it's rather optional. You can name your model whatever you want. However, you can even keep this string field empty (in that case Xcode assigns to it a long string of letters and digits or simply an empty string instead of a modified name "Steel Box" – and there will be no steelBox property for you).

"Steel Box" name was automatically used to create `steelBox` property

So, for instance, if you named a model in Reality Composer scene "Plastic Sphere", it automatically creates a variable called plasticSphere:

public var plasticSphere: Entity? { get }

In RealityKit you can access your cube entity this way:

 let boxAnchor = try! Experience.loadBox()
 arView.scene.anchors.append(boxAnchor)

 print(boxAnchor.steelBox!)

or using a subscript for its hierarchy:

 let boxAnchor = try! Experience.loadBox()
 arView.scene.anchors.append(boxAnchor)

 print(boxAnchor.children[0].children[0].children[0])

This results in printing Entity's hierarchy:

As we can see from Console "Steel Box" is just a steelBox's entity name.

print(boxAnchor.steelBox!.name as Any)

// "Steel Box"

However, as I said before, when you leave this field for model's name empty in Reality Composer...

...you can't retrieve your entity by its property name from scene.

You can retrieve it only using children's hierarchy:

let boxAnchor = try! Experience.loadBox()
arView.scene.anchors.append(boxAnchor)

print(boxAnchor.children[0].children[0].children[0])

So, as we can see there's no entity's name (string is empty) in Console now.

But you can assign it at any time:

boxAnchor.steelBox!.name = "My favorite box"

---

Related Topics