Capture Metal Mtkview as Movie in Realtime

Capture Metal MTKView as Movie in realtime?

Here's a small class that performs the essential functions of writing out a movie file that captures the contents of a Metal view:

class MetalVideoRecorder {
    var isRecording = false
    var recordingStartTime = TimeInterval(0)

    private var assetWriter: AVAssetWriter
    private var assetWriterVideoInput: AVAssetWriterInput
    private var assetWriterPixelBufferInput: AVAssetWriterInputPixelBufferAdaptor

    init?(outputURL url: URL, size: CGSize) {
        do {
            assetWriter = try AVAssetWriter(outputURL: url, fileType: .m4v)
        } catch {
            return nil
        }

        let outputSettings: [String: Any] = [ AVVideoCodecKey : AVVideoCodecType.h264,
            AVVideoWidthKey : size.width,
            AVVideoHeightKey : size.height ]

        assetWriterVideoInput = AVAssetWriterInput(mediaType: .video, outputSettings: outputSettings)
        assetWriterVideoInput.expectsMediaDataInRealTime = true

        let sourcePixelBufferAttributes: [String: Any] = [
            kCVPixelBufferPixelFormatTypeKey as String : kCVPixelFormatType_32BGRA,
            kCVPixelBufferWidthKey as String : size.width,
            kCVPixelBufferHeightKey as String : size.height ]

        assetWriterPixelBufferInput = AVAssetWriterInputPixelBufferAdaptor(assetWriterInput: assetWriterVideoInput,
                                                                           sourcePixelBufferAttributes: sourcePixelBufferAttributes)

        assetWriter.add(assetWriterVideoInput)
    }

    func startRecording() {
        assetWriter.startWriting()
        assetWriter.startSession(atSourceTime: .zero)

        recordingStartTime = CACurrentMediaTime()
        isRecording = true
    }

    func endRecording(_ completionHandler: @escaping () -> ()) {
        isRecording = false

        assetWriterVideoInput.markAsFinished()
        assetWriter.finishWriting(completionHandler: completionHandler)
    }

    func writeFrame(forTexture texture: MTLTexture) {
        if !isRecording {
            return
        }

        while !assetWriterVideoInput.isReadyForMoreMediaData {}

        guard let pixelBufferPool = assetWriterPixelBufferInput.pixelBufferPool else {
            print("Pixel buffer asset writer input did not have a pixel buffer pool available; cannot retrieve frame")
            return
        }

        var maybePixelBuffer: CVPixelBuffer? = nil
        let status  = CVPixelBufferPoolCreatePixelBuffer(nil, pixelBufferPool, &maybePixelBuffer)
        if status != kCVReturnSuccess {
            print("Could not get pixel buffer from asset writer input; dropping frame...")
            return
        }

        guard let pixelBuffer = maybePixelBuffer else { return }

        CVPixelBufferLockBaseAddress(pixelBuffer, [])
        let pixelBufferBytes = CVPixelBufferGetBaseAddress(pixelBuffer)!

        // Use the bytes per row value from the pixel buffer since its stride may be rounded up to be 16-byte aligned
        let bytesPerRow = CVPixelBufferGetBytesPerRow(pixelBuffer)
        let region = MTLRegionMake2D(0, 0, texture.width, texture.height)

        texture.getBytes(pixelBufferBytes, bytesPerRow: bytesPerRow, from: region, mipmapLevel: 0)

        let frameTime = CACurrentMediaTime() - recordingStartTime
        let presentationTime = CMTimeMakeWithSeconds(frameTime, preferredTimescale: 240)
        assetWriterPixelBufferInput.append(pixelBuffer, withPresentationTime: presentationTime)

        CVPixelBufferUnlockBaseAddress(pixelBuffer, [])
    }
}

After initializing one of these and calling startRecording(), you can add a scheduled handler to the command buffer containing your rendering commands and call writeFrame (after you end encoding, but before presenting the drawable or committing the buffer):

let texture = currentDrawable.texture
commandBuffer.addCompletedHandler { commandBuffer in
    self.recorder.writeFrame(forTexture: texture)
}

When you're done recording, just call endRecording, and the video file will be finalized and closed.

Caveats:

This class assumes the source texture to be of the default format, .bgra8Unorm. If it isn't, you'll get crashes or corruption. If necessary, convert the texture with a compute or fragment shader, or use Accelerate.

This class also assumes that the texture is the same size as the video frame. If this isn't the case (if the drawable size changes, or your screen autorotates), the output will be corrupted and you may see crashes. Mitigate this by scaling or cropping the source texture as your application requires.

Displaying decoded video stream with MTKView results in undesirable blurry output

It looks like you have the most serious issue of view scale addressed, the other issues are proper YCbCr rendering (which it sounds like you are going to avoid by outputting BGRA pixels when decoding) and then there is scaling the original movie to match the dimensions of the view. When you request BGRA pixel data the data is encoded as sRGB, so you should treat the data in the texture as sRGB. Metal will automatically do the non-linear to linear conversion for you when reading from a sRGB texture, but you have to tell Metal that it is sRGB pixel data (using MTLPixelFormatBGRA8Unorm_sRGB). To implement scaling, you just need to render from the BGRA data into the view with linear resampling. See the SO question I linked above if you want to have a look at the source code for MetalBT709Decoder which is my own project that implements proper rendering of BT.709.

All black frames when trying to write Metal frames to Quicktime file with AVFoundation AVAssetWriter

I believe you are writing your frames too early -- by calling writeFrame from within your render loop, you are essentially capturing the drawable at a time when it is still empty (the GPU just hasn't rendered it yet).

Remember that before you call commmandBuffer.commit(), the GPU hasn't even begun rendering your frame. You need to wait for the GPU to finish rendering before trying to grab the resulting frame. The sequence is a bit confusing because you're also calling present() before calling commit(), but that isn't the actual order of operations in run-time. That present call is merely telling Metal to schedule a call to present your frame to the screen once the GPU has finished rendering.

You should call writeFrame from within a completion handler (using commandBuffer.addCompletedHandler()). That should take care of this.

UPDATE: While the answer above is correct, it is only partial. Since the OP was using a discrete GPU with private VRAM, the CPU wasn't able to see the render target pixels. The solution to that problem is to add an MTLBlitCommandEncoder, and use the synchronize() method to ensure the rendered pixels are copied back to RAM from the GPU's VRAM.

How to render/export frames offline in Metal?

I adapted my answer here and the Apple Metal game template to create this sample, which demonstrates how to record a video file directly from a sequence of frames rendered by Metal.

Since all rendering in Metal draws to a texture, it's not too hard to adapt normal Metal code so that it's suitable for rendering offline into a movie file. To recap the core recording process:

Create an AVAssetWriter that targets your URL of choice
Create an AVAssetWriterInput of type .video so you can write video frames
Wrap an AVAssetWriterInputPixelBufferAdaptor around the input so you can append CVPixelBuffers as frames to the video
After you start recording, each frame, copy the pixels from your rendered frame texture into a pixel buffer obtained from the adapter's pixel buffer pool.
When you're done, mark the input as finished and finish writing to the asset writer.

As for driving the recording, since you aren't getting delegate callbacks from an MTKView or CADisplayLink, you need to do it yourself. The basic pattern looks like this:

for t in stride(from: 0, through: duration, by: frameDelta) {
    draw(in: renderBuffer, depthTexture: depthBuffer, time: t) { (texture) in
        recorder.writeFrame(forTexture: texture, time: t)
    }
}

If your rendering and recording code is asynchronous and thread-safe, you can throw this on a background queue to keep your interface responsive. You could also throw in a progress callback to update your UI if your rendering takes a long time.

Note that since you're not running in real-time, you'll need to ensure that any animation takes into account the current frame time (or the timestep between frames) so things run at the proper rate when played back. In my sample, I do this by just having the rotation of the cube depend directly on the frame's presentation time.

Capture Metal Mtkview as Movie in Realtime