How to Merge Audio and Video Using Avmutablecompositiontrack

AVMutableComposition -How to Merge Multiple Audio Recordings with 1 Video Recording

Your approach is correct, but you've mixed up the two parameters that you're using for insertTimeRange, and you're adding the video and audio from your video track multiple times.

The first parameter in insertTimeRange refers to the timeRange within the original audio asset, not the composition; so assuming that for each audio clip you are looking to add the entire clip, the time range should always start at .zero, not at startTime. The at: parameter should no be .zero, but rather "startTime" - the time within the composition where you want to add the audio.

Regarding your video track and your audioFromVideoTrack, I would not add these as part of the loop, but rather just add them before the loop. Otherwise you are adding them multiple times (once for each audio item), rather than just once, and this can lead to unwanted behavior or the export sessions failing altogether.

I edited your code but wasn't able to actually test it so take it with a grain of salt.

guard let videoCompositionTrack = mixComposition.addMutableTrack(withMediaType: .video, preferredTrackID: Int32(kCMPersistentTrackID_Invalid)) else { return }
guard let audioFromVideoCompositionTrack = mixComposition.addMutableTrack(withMediaType: .audio, preferredTrackID: Int32(kCMPersistentTrackID_Invalid)) else { return }
guard let audioModelCompositionTrack = mixComposition.addMutableTrack(withMediaType: .audio, preferredTrackID: Int32(kCMPersistentTrackID_Invalid)) else { return }

let videoAsset = AVURLAsset(url: videoURL)
guard let videoTrack = videoAsset.tracks(withMediaType: .video).first else { return }

do {
    try videoCompositionTrack.insertTimeRange(CMTimeRangeMake(start: .zero, duration: videoAsset.duration), of: videoTrack, at: .zero)
    if let audioFromVideoTrack = videoAsset.tracks(withMediaType: .audio).first {
        try audioFromVideoCompositionTrack.insertTimeRange(CMTimeRangeMake(start: CMTime.zero, duration: videoAsset.duration), of: audioFromVideoTrack, at: .zero)
    }
} catch {
}

for audioModel in audioModels {
    let audioAsset = AVURLAsset(url: audioModel.url!)
    let startTime = CMTime(seconds: audioModel.startTime!, preferredTimescale: 1000)
    do {
        if let audioTrackFromAudioModel = audioAsset.tracks(withMediaType: .audio).first {
            try audioModelCompositionTrack.insertTimeRange(CMTimeRangeMake(start: .zero, duration: audioAsset.duration), of: audioTrackFromAudioModel, at: startTime)
        }
    } catch {
    }
}

let exporter = AVAssetExportSession(asset: mixComposition, presetName: AVAssetExportPresetHighestQuality)
// ... I know what to do from here

Merge two videos with audio and video together in iOS

The problem is that you are adding a second video track to the composition. You need to insert both videos into the same video track. Just delete your let videoTrack2 and go from there.

How to merge 1 video and 2 or more audio files with AVFoundation

Ok, I just found what the problem was; basically, there is one golden rule that must be followed when using AVMutableComposition (at least to merge multiple audios), that is:

1 audio = 1 video + 1 instruction

In other words, for every audio, there must be 1 video and 1 instruction. Following this rule my previous code results in the following:

public void mergeAudios()
        {
            //This funtion merges the final video with the new audio

            #region HoldVideoTrack
            AVAsset video_asset = AVAsset.FromUrl(NSUrl.FromFilename(FinalVideo));

            //This range applies to the video, not to the mixcomposition
            CMTimeRange range = new CMTimeRange()
            {
                Start = new CMTime(0, 0),
                Duration = video_asset.Duration
            };
            #endregion

            AVMutableComposition mixComposition = new AVMutableComposition();

            #region AddsVideo
            AVMutableCompositionTrack videoTrack = mixComposition.AddMutableTrack(AVMediaType.Video, 0);
            AVAssetTrack assetVideoTrack = video_asset.TracksWithMediaType(AVMediaType.Video)[0];
            videoTrack.InsertTimeRange(range, assetVideoTrack, CMTime.Zero, out NSError error1);
            #endregion

            #region AddsVideo'sAudio
            //If the device can't use the microphone then the original video's audio will not exist
            AVCaptureDevice microphone = AVCaptureDevice.DefaultDeviceWithMediaType(AVMediaType.Audio);
            if (microphone != null)
            {
                AVMutableCompositionTrack audio_video_Track = mixComposition.AddMutableTrack(AVMediaType.Audio, 0);
                AVAssetTrack assetAudioVideoTrack = video_asset.TracksWithMediaType(AVMediaType.Audio)[0];
                audio_video_Track.InsertTimeRange(range, assetAudioVideoTrack, mixComposition.Duration, out NSError error2);
            }
            #endregion

            //[TTS_list.Count + 1]; +1 = original Video
            AVMutableVideoCompositionLayerInstruction[] Instruction_Array = new AVMutableVideoCompositionLayerInstruction[TTS_list.Count + 1];
            //This instruction is for "FinalVideo"
            Instruction_Array[0] = SetInstruction(video_asset, mixComposition.Duration, videoTrack);

            #region TestingEnviroment
            //We will use counter to specify the position in Instruction_Array, we start with 1 because we have already added 1 instruction for "FinalVideo"
            int counter = 1;
            foreach(Audio _audioo in TTS_list)
            {
                #region Video
                AVMutableCompositionTrack videoTrack_forAudio = mixComposition.AddMutableTrack(AVMediaType.Video, 0);
                AVAssetTrack assetVideoTrack_forAudio = video_asset.TracksWithMediaType(AVMediaType.Video)[0];

                //This range applies to the video, not to the mixcomposition, making its duration 0 and having no overall effect on the final video.
                //We have to declare 1 video for each audio in order to merge multiple audios. Doing it this way the videos have no effect, but the audios do
                CMTimeRange range0 = new CMTimeRange()
                {
                    Start = new CMTime(0, 0),
                    Duration = CMTime.FromSeconds(0, 600)
                };
                videoTrack_forAudio.InsertTimeRange(range0, assetVideoTrack_forAudio, mixComposition.Duration, out NSError error4);
                #endregion

                #region Audio
                AVAsset audio_asset = AVAsset.FromUrl(NSUrl.FromFilename(_audioo.Path));

                //This range applies to the video, not to the mixcomposition
                //We use _audio.Duration instead of audio_asset.Duration.Seconds because the audio's duration might be trimmed
                CMTimeRange audio_CMTime = new CMTimeRange()
                {
                    Start = new CMTime(0, 0),
                    Duration = CMTime.FromSeconds(_audioo.Duration, 600)
                };

                //This range applies to mixcomposition, not to the video
                var starting_CMTime = CMTime.FromSeconds(_audioo.Starting_Point, 600);

                AVMutableCompositionTrack audioTrack = mixComposition.AddMutableTrack(AVMediaType.Audio, 0);
                AVAssetTrack assetAudioTrack = audio_asset.TracksWithMediaType(AVMediaType.Audio)[0];
                audioTrack.InsertTimeRange(audio_CMTime, assetAudioTrack, starting_CMTime, out NSError error5);
                #endregion

                #region Instruction
                Instruction_Array[counter] = SetInstruction(video_asset, mixComposition.Duration, videoTrack);
                counter += 1;
                #endregion
            }
            #endregion

            #region Instructions
            var mainInstruction = new AVMutableVideoCompositionInstruction();

            CMTimeRange rangeIns = new CMTimeRange()
            {
                Start = new CMTime(0, 0),
                Duration = mixComposition.Duration
            };

            mainInstruction.BackgroundColor = UIColor.FromRGBA(0.63f, 0.84f, 0.82f, 1.000f).CGColor;
            mainInstruction.TimeRange = rangeIns;
            mainInstruction.LayerInstructions = Instruction_Array;
            #endregion

            var mainComposition = new AVMutableVideoComposition()
            {
                Instructions = new AVVideoCompositionInstruction[1] { mainInstruction },
                FrameDuration = new CMTime(1, 30),
                RenderSize = new CoreGraphics.CGSize(UIScreenWidth, UIScreenHeight)
            };

            finalVideo_path = NSUrl.FromFilename(Path.Combine(Path.GetTempPath(), "temporaryClip/FinalVideoEdit.mov"));
            if (File.Exists(Path.GetTempPath() + "temporaryClip/FinalVideoEdit.mov"))
            {
                File.Delete(Path.GetTempPath() + "temporaryClip/FinalVideoEdit.mov");
            }

            AVAssetExportSession exportSession = new AVAssetExportSession(mixComposition, AVAssetExportSessionPreset.MediumQuality)
            {
                OutputUrl = finalVideo_path,
                OutputFileType = AVFileType.QuickTimeMovie,
                ShouldOptimizeForNetworkUse = true,
                VideoComposition = mainComposition
            };
            exportSession.ExportAsynchronously(_OnExportDone);
        }

Merging Videos Together with AVMutableComposition Causes No Audio

The reason you're not getting audio is that you're not adding the audio track. You need to create an additional AVMutableCompositionTrack with a type of AVMediaTypeAudio:

AVMutableComposition *mixComposition = [[AVMutableComposition alloc] init];

AVMutableCompositionTrack *videoTrack = [mixComposition addMutableTrackWithMediaType:AVMediaTypeVideo
                                                           preferredTrackID:kCMPersistentTrackID_Invalid];

AVMutableCompositionTrack *audioTrack = [mixComposition addMutableTrackWithMediaType:AVMediaTypeAudio
                                                           preferredTrackID:kCMPersistentTrackID_Invalid];

And insert the time range for the source audio and video tracks into both composition tracks:

CMTime insertTime = kCMTimeZero;

for (id object in movieArray) {

    AVAsset *asset = [AVAsset assetWithURL:object];

    CMTimeRange timeRange = CMTimeRangeMake(kCMTimeZero, asset.duration);

    [videoTrack insertTimeRange:timeRange
                        ofTrack:[[asset tracksWithMediaType:AVMediaTypeVideo] objectAtIndex:0]
                         atTime:insertTime
                          error:nil];

    [audioTrack insertTimeRange:timeRange
                         ofTrack:[[asset tracksWithMediaType:AVMediaTypeAudio] objectAtIndex:0]
                          atTime:insertTime
                           error:nil];

    insertTime = CMTimeAdd(insertTime,asset.duration);
}

Merging audio tracks in a single track using AVMutableComposition

OK, finally I should have found the solution. Really using the AVMutableAudioMix resulting movie file has only one audio track instead of two.

EDIT
Answering to Justin comment, here is the trick:

        let audioMix = AVMutableAudioMix()
        let vip = AVMutableAudioMixInputParameters(track: self.videoAudioTrack!)
        vip.trackID = self.videoAudioTrack!.trackID
        vip.setVolume(self.videoAudioMixerVolume, at: .zero)
        let aip = AVMutableAudioMixInputParameters(track: self.audioTrack!)
        aip.trackID = self.audioTrack!.trackID
        aip.setVolume(self.audioMixerVolume, at: .zero)
        audioMix.inputParameters = [vip, aip]
        easset.audioMix = audioMix

Where videoAudioTrack is the audio track for the video clip, wherease audioTrack is another simple audio track. easset is the AVAssetExporterSession object.

why no sound when I merge Audio and video using AVMutableCompositionTrack?

Your code works. assure if your audio exist on the path. I have try to load sound and video from bundle and merge using your code. it works as i expected. my load from bundle looks like this.

NSString *pathAudio = [[NSBundle mainBundle] pathForResource:@"sound" ofType:@"mp3"];
AVURLAsset* audioAsset = [[AVURLAsset alloc]initWithURL:[NSURL fileURLWithPath:pathAudio] options:nil];

NSString *pathVideo = [[NSBundle mainBundle] pathForResource:@"Clip1" ofType:@"mp4"];
AVURLAsset* videoAsset = [[AVURLAsset alloc]initWithURL:[NSURL fileURLWithPath:pathVideo] options:nil];

UPDATED

In order to merge video sound and another sound same time add another AVMutableCompositionTrack to your AVMutableComposition.

// add another track for video sound
AVMutableCompositionTrack *videoSoundTrack = [mixComposition addMutableTrackWithMediaType:AVMediaTypeAudio
                                                                     preferredTrackID:kCMPersistentTrackID_Invalid];

//insert video sound to the track
[videoSoundTrack insertTimeRange:CMTimeRangeMake(kCMTimeZero, videoAsset.duration)
                         ofTrack:[[videoAsset tracksWithMediaType:AVMediaTypeAudio] objectAtIndex:0]
                          atTime:kCMTimeZero error:nil];

Merging clips with AVFoundation creates single video in black

Ok, so thanks to Shawn's help I have accomplished what I was trying to do.
There were 2 main mistakes in my code that generated this problem, the first one was how the property of the CMTime given to VideoTrack was set: Start = new CMTime(0,0), instead of Start = new CMTime.Zero,. I still don't know what difference does it make, but it prevented the code from displaying the video and the audio of each asset, leaving a video with the length of all the clips combined and the background of AVMutableVideoCompositionInstruction.
The second mistake was how I set the instructions, the configuration that worked for me can be found in the following code.

Here is the final function working as correctly:

public void MergeClips()
        {
            //microphone
            AVCaptureDevice microphone = AVCaptureDevice.DefaultDeviceWithMediaType(AVMediaType.Audio);

            AVMutableComposition mixComposition = AVMutableComposition.Create();
            AVVideoCompositionLayerInstruction[] Instruction_Array = new AVVideoCompositionLayerInstruction[Clips.Count];

            foreach (string clip in Clips)
            {
                var asset = AVUrlAsset.FromUrl(new NSUrl(clip, false)) as AVUrlAsset;
                #region HoldVideoTrack

                //This range applies to the video, not to the mixcomposition
                CMTimeRange range = new CMTimeRange()
                {
                    Start = CMTime.Zero,
                    Duration = asset.Duration
                };

                var duration = mixComposition.Duration;
                NSError error;

                AVMutableCompositionTrack videoTrack = mixComposition.AddMutableTrack(AVMediaType.Video, 0);
                AVAssetTrack assetVideoTrack = asset.TracksWithMediaType(AVMediaType.Video)[0];
                videoTrack.InsertTimeRange(range, assetVideoTrack, duration, out error);
                videoTrack.PreferredTransform = assetVideoTrack.PreferredTransform;

                if (microphone != null)
                {
                    AVMutableCompositionTrack audioTrack = mixComposition.AddMutableTrack(AVMediaType.Audio, 0);
                    AVAssetTrack assetAudioTrack = asset.TracksWithMediaType(AVMediaType.Audio)[0];
                    audioTrack.InsertTimeRange(range, assetAudioTrack, duration, out error);
                }
                #endregion

                #region Instructions
                int counter = Clips.IndexOf(clip);
                Instruction_Array[counter] = SetInstruction(asset, mixComposition.Duration, videoTrack);
                #endregion
            }

            // 6
            AVMutableVideoCompositionInstruction mainInstruction = AVMutableVideoCompositionInstruction.Create() as AVMutableVideoCompositionInstruction;

            CMTimeRange rangeIns = new CMTimeRange()
            {
                Start = new CMTime(0, 0),
                Duration = mixComposition.Duration
            };
            mainInstruction.TimeRange = rangeIns;
            mainInstruction.LayerInstructions = Instruction_Array;
            
            var mainComposition = AVMutableVideoComposition.Create();
            mainComposition.Instructions = new AVVideoCompositionInstruction[1] { mainInstruction };
            mainComposition.FrameDuration = new CMTime(1, 30);
            mainComposition.RenderSize = new CGSize(mixComposition.NaturalSize.Height, mixComposition.NaturalSize.Width);
            
            finalVideo_path = NSUrl.FromFilename(Path.Combine(Path.GetTempPath(), "Whole2.mov"));
            if (File.Exists(Path.GetTempPath() + "Whole2.mov"))
            {
                File.Delete(Path.GetTempPath() + "Whole2.mov");
            }

            //... export video ...
            AVAssetExportSession exportSession = new AVAssetExportSession(mixComposition, AVAssetExportSessionPreset.HighestQuality)
            {
                OutputUrl = NSUrl.FromFilename(Path.Combine(Path.GetTempPath(), "Whole2.mov")),
                OutputFileType = AVFileType.QuickTimeMovie,
                ShouldOptimizeForNetworkUse = true,
                VideoComposition = mainComposition
            };
            exportSession.ExportAsynchronously(_OnExportDone);
        }

        private AVMutableVideoCompositionLayerInstruction SetInstruction(AVAsset asset, CMTime currentTime, AVAssetTrack mixComposition_video_Track)
        {
            var instruction = AVMutableVideoCompositionLayerInstruction.FromAssetTrack(mixComposition_video_Track);

            var startTime = CMTime.Subtract(currentTime, asset.Duration);

            //NaturalSize.Height is passed as a width parameter because IOS stores the video recording horizontally 
            CGAffineTransform translateToCenter = CGAffineTransform.MakeTranslation(mixComposition_video_Track.NaturalSize.Height, 0);
            //Angle in radiants, not in degrees
            CGAffineTransform rotate = CGAffineTransform.Rotate(translateToCenter, (nfloat)(Math.PI / 2));

            instruction.SetTransform(rotate, (CMTime.Subtract(currentTime, asset.Duration)));

            instruction.SetOpacity(1, startTime);
            instruction.SetOpacity(0, currentTime);

            return instruction;
        }

As I said I solved my problem thanks to Shawn's help, and most of this code was translated to C# from his answers, so please, if you were planning on voting up this answer, vote up Shawn's one instead, or both.

---

Related Topics