iOS Screen Sharing (Using Replaykit) Using Webrtc in Swift

ReplayKit SampleHandler pass data to the app module from upload broadcast extension

Finally I get to know that it's not possible. We can share data like strings using UserDefaults by creating app groups but that's not what I want to do here. I need to share CMSampleBuffer so this is out of the question. The Extension and the host app run on separate sandbox environments and sharing data between two wont be feasible.

What we can do is writing the complete video call logic inside the extension itself. All the relevant code that will be responsible to send these frames to the server to a particular room (Room name and other details can be shared using UserDefaults) and stream from the extension itself, which I believe is a long road. Or write a common code for streaming that'll be utilised by the host app and the extension (Again, data between these two wont be possible, both will create a different instance for the file).

Sometimes enforcing security can be painful for devs!

How to send CMSampleBuffer to WebRTC?

Hi Sam WebRTC have one function which can process CMSampleBuffer frames to get Video Frames. But it is working with CVPixelBuffer. So you have to firstly convert your CMSampleBuffer to CVPixelBuffer. And than add this frames into your localVideoSource with RTCVideoCapturer. i have solved similar problem on AVCaptureVideoDataOutputSampleBufferDelegate. This delegate produces CMSampleBuffer as ReplayKit. i hope that below code lines could be help to you. You can try at the below code lines to solve your problem.

private var videoCapturer: RTCVideoCapturer?
private var localVideoSource = RTCClient.factory.videoSource()
private var localVideoTrack: RTCVideoTrack?
private var remoteVideoTrack: RTCVideoTrack?
private var  peerConnection: RTCPeerConnection? = nil
public static let factory: RTCPeerConnectionFactory = {
    RTCInitializeSSL()
    let videoEncoderFactory = RTCDefaultVideoEncoderFactory()
    let videoDecoderFactory = RTCDefaultVideoDecoderFactory()
    return RTCPeerConnectionFactory(encoderFactory: videoEncoderFactory, decoderFactory: videoDecoderFactory)
}()

extension RTCClient : AVCaptureVideoDataOutputSampleBufferDelegate {
func captureOutput(_ output: AVCaptureOutput, didOutput sampleBuffer: CMSampleBuffer, from connection: AVCaptureConnection) {
    print("didOutPut: \(sampleBuffer)")
    
    guard let imageBuffer: CVImageBuffer = CMSampleBufferGetImageBuffer(sampleBuffer) else { return }
    let timeStampNs: Int64 = Int64(CMTimeGetSeconds(CMSampleBufferGetPresentationTimeStamp(sampleBuffer)) * 1000000000) 
    
    let rtcPixlBuffer = RTCCVPixelBuffer(pixelBuffer: imageBuffer)
    let rtcVideoFrame = RTCVideoFrame(buffer: rtcPixlBuffer, rotation: ._90, timeStampNs: timeStampNs)
    self.localVideoSource.capturer(videoCapturer!, didCapture: rtcVideoFrame)
    
  }  
}

Also you need configuration like that for mediaSender,

func createMediaSenders() {
    let streamId = "stream"
    
    let videoTrack = self.createVideoTrack()
    self.localVideoTrack = videoTrack
    self.peerConnection!.add(videoTrack, streamIds: [streamId])
    self.remoteVideoTrack = self.peerConnection!.transceivers.first { $0.mediaType == .video }?.receiver.track as? RTCVideoTrack
    
}
private func createVideoTrack() -> RTCVideoTrack {
let videoTrack = RTCClient.factory.videoTrack(with: self.videoSource, trackId: "video0")
    return videoTrack
}

Related Topics