Realitykit - How to Add a Video Material to a Modelentity

How to Add Material to ModelEntity programatically in RealityKit?

Updated: June 14, 2022

RealityKit materials

There are 6 types of materials in RealityKit 2.0 and RealityFoundation at the moment:

SimpleMaterial
UnlitMaterial
OcclusionMaterial (read this post to find out how to setup SceneKit occlusion shader)
VideoMaterial (look at this post to find out how to setup it)
PhysicallyBasedMaterial
CustomMaterial

To apply these materials use the following logic:

import Cocoa
import RealityKit

class ViewController: NSViewController {        
    @IBOutlet var arView: ARView!
    
    override func awakeFromNib() {
        let box = try! Experience.loadBox()
        
        var simpleMat = SimpleMaterial()
        simpleMat.color = .init(tint: .blue, texture: nil)
        simpleMat.metallic = .init(floatLiteral: 0.7)
        simpleMat.roughness = .init(floatLiteral: 0.2)
        
        var pbr = PhysicallyBasedMaterial()
        pbr.baseColor = .init(tint: .green, texture: nil) 

        let mesh: MeshResource = .generateBox(width: 0.5, 
                                             height: 0.5, 
                                              depth: 0.5, 
                                       cornerRadius: 0.02, 
                                         splitFaces: true)

        let boxComponent = ModelComponent(mesh: mesh,
                                     materials: [simpleMat, pbr])

        box.steelBox?.children[0].components.set(boxComponent)
        box.steelBox?.orientation = Transform(pitch: .pi/4, 
                                                yaw: .pi/4, 
                                               roll: 0).rotation
        arView.scene.anchors.append(box)
    }
}

Sample Image

Read this post to find out how to load a texture for RealityKit's shaders.

How to create RealityKit's shaders similar to SceneKit's shaders

We know that in SceneKit there are 5 different shading models, so we can use RealityKit's SimpleMaterial, PhysicallyBasedMaterial and UnlitMaterial to generate all these five shaders that we've been accustomed to.

Let's see how it looks like:

SCNMaterial.LightingModel.blinn           – SimpleMaterial(color: . gray,
                                                        roughness: .float(0.5),
                                                       isMetallic: false)

SCNMaterial.LightingModel.lambert         – SimpleMaterial(color: . gray,
                                                        roughness: .float(1.0),
                                                       isMetallic: false)
  
SCNMaterial.LightingModel.phong           – SimpleMaterial(color: . gray,
                                                        roughness: .float(0.0),
                                                       isMetallic: false)

SCNMaterial.LightingModel.physicallyBased – PhysicallyBasedMaterial()

// all three shaders (`.constant`, `UnlitMaterial` and `VideoMaterial `) 
// don't depend on lighting
SCNMaterial.LightingModel.constant        – UnlitMaterial(color: .gray)
                                          – VideoMaterial(avPlayer: avPlayer)

RealityKit – Why ModelEntity doesn't change dynamically?

Fix the following errors and it will do the trick:

Use $ for timeAccumulate property wrapper in ContentView struct to get a binding struct.
```
ARViewContainer(timeAccumulate: $timeAccumulate)
```
Then use @Binding attribute for timeAccumulate property in ARViewContainer struct.
```
@Binding var timeAccumulate: Int
```

baseColor is still working in iOS 15 but it'll be irrelevant on iOS 16. So, use color instead:

var material = SimpleMaterial()

material.color = .init(tint: .white,
                    texture: .init(try! .load(named: "texture.png")))

Add UIImage as texture to a Plane in RealityKit

Currently, you cannot use UIImage or CIImage as a shader's texture in RealityKit 2.0. In both versions of RealityKit, the texture must be loaded via the String type parameter of the load() method.

RealityKit 2.0

To assign a texture to a shader in RealityKit 2.0 use the following approach:

let mesh: MeshResource = .generatePlane(width: 0.45, depth: 0.45)
        
var material = SimpleMaterial()
material.color = .init(tint: .white.withAlphaComponent(0.999),
                    texture: .init(try! .load(named: "texture.png")))
material.metallic = .float(1.0)
material.roughness = .float(0.0)

let model = ModelEntity(mesh: mesh, materials: [material])

RealityKit 1.0

To assign a texture to a shader in RealityKit 1.0 use this approach:

let scene = try! Experience.loadMyScene()

var material = SimpleMaterial()
material.baseColor = try! .texture(.load(named: "texture.png"))
material.metallic = MaterialScalarParameter(floatLiteral: 1.0)
material.roughness = MaterialScalarParameter(floatLiteral: 0.0)
material.tintColor = UIColor.white

let mesh: MeshResource = .generatePlane(width: 0.45, depth: 0.45)
let component = ModelComponent(mesh: mesh, materials: [material])

scene.myFavoriteScene?.children[0].components.set(component)
arView.scene.anchors.append(scene)

CGImage

Nonetheless, you can create a texture resource from an in-memory Core Graphics image:

static func generate(from: CGImage, 
                 withName: String?, 
                  options: TextureResource.CreateOptions) -> TextureResource

Also, you can use a URL parameter:

material.color.texture = .init(try! .load(contentsOf: url))  // RealityKit 2.0