How to Get Buffer from Avaudioengine's Installtap at High Frequency

how to get buffer from AVAudioEngine's installTap at high frequency

I think you misunderstand how to handle this. Instead of getting very small amounts of new data at a high frequency, you should factor your code to get a larger chunk of data on a longer time interval.

You can then set up your display code to start an animation to animate in a graph of the new data over the time interval you are given. 1/10th of a second is actually a pretty short time interval for an animation.

I would suggest averaging the amplitude of the data for that 1/10th of a second time interval and animating that new average data into place.

BTW, see the update I posted to the answer to your question about creating an animated graph of the data.

The animation I came up with looks like this:

My writing the installTap buffer to an AVAudioFile seems to fail data-wise

You need to let your AVAudioFile go out of scope (nil it at some point), that's how you call AVAudioFile's close() method, which presumably finishes writing out header information.

How To Increase Callback Frequency of AVAudioEngine.installTap()

Since there is is no way to increase the frequency, I resorted to interpolating the previous value with the current value to smoothen the transition.

On extracting the sound pressure level from AVAudioPCMBuffer

I think the first step is to get the envelope of the sound. You could use simple averaging to calculate an envelope, but you need to add a rectification step (usually means using abs() or square() to make all samples positive)

More commonly a simple iir-filter is used instead of averaging, with different constants for attack and decay, here is a lab. Note that these constants depend on the sampling frequency, you can use this formula to calculate the constants:

1 - exp(-timePerSample*2/smoothingTime)

Step 2

When you have the envelope, you can smooth it with an additional filter, and then compare the two envelopes to find a sound that is louder than the baselevel, here's a more complete lab.

Note that detecting audio "events" can be quite tricky, and hard to predict, make sure you have a lot of debbugging aid!

Connecting AVAudioMixerNode to AVAudioEngine

You can hear your microphone recording through your speakers because your microphone is connected to downMixer, which is connected to engine.outputNode. You could probably just mute the output for the downMixer if you aren't using it with other inputs:

downMixer.outputVolume = 0.0


Low Pass filter + sample rate conversion using Avaudioengine iOS

I think the main problem with this code was that the AVAudioConverter was being created before calling engine.prepare() which can and will change the mainMixerNode output format. Aside from that, there was a redundant connection of mainMixerNode to outputNode, along with a probably incorrect format - mainMixerNode is documented to be automatically created and connected to the output node "on demand". The tap also did not need a format.

let bus = 0
let inputNode = engine.inputNode

let equalizer = AVAudioUnitEQ(numberOfBands: 2)

equalizer.bands[0].filterType = .lowPass
equalizer.bands[0].frequency = 3000
equalizer.bands[0].bypass = false

equalizer.bands[1].filterType = .highPass
equalizer.bands[1].frequency = 1000
equalizer.bands[1].bypass = false
engine.attach(equalizer) //Attach equalizer

// Connect nodes
engine.connect(inputNode, to: equalizer, format: inputNode.inputFormat(forBus: 0))
engine.connect(equalizer, to: engine.mainMixerNode, format: inputNode.inputFormat(forBus: 0))

// call before creating converter because this changes the mainMixer's output format
engine.prepare()

let outputFormat = AVAudioFormat(commonFormat: .pcmFormatInt16,
                                 sampleRate: 5000,
                                 channels: 1,
                                 interleaved: false)!

// Downsampling converter
guard let converter: AVAudioConverter = AVAudioConverter(from: engine.mainMixerNode.outputFormat(forBus: 0), to: outputFormat) else {
    print("Can't convert in to this format")
    return
}

engine.mainMixerNode.installTap(onBus: bus, bufferSize: 2688, format: nil) { (buffer, time) in
    var newBufferAvailable = true
    
    let inputCallback: AVAudioConverterInputBlock = { inNumPackets, outStatus in
        if newBufferAvailable {
            outStatus.pointee = .haveData
            newBufferAvailable = false
            return buffer
        } else {
            outStatus.pointee = .noDataNow
            return nil
        }
    }
    
    
    let convertedBuffer = AVAudioPCMBuffer(pcmFormat: outputFormat, frameCapacity: AVAudioFrameCount(outputFormat.sampleRate) * buffer.frameLength / AVAudioFrameCount(buffer.format.sampleRate))!
    
    var error: NSError?
    let status = converter.convert(to: convertedBuffer, error: &error, withInputFrom: inputCallback)
    assert(status != .error)
    
    
    if status == .haveData {
        // Process with converted buffer
    }
}

do {
    try engine.start()
} catch {
    print("Can't start the engine: \(error)")
}

Animate CAShapelayer content when new item is added

Don't try to do your animation in draw(rect:). That puts all the work on the CPU, not the GPU, and does not take advantage of the hardware-accelerated animation in iOS.

I would suggest instead using a CAShapeLayer and a CABasicAnimation to animate your path.

Install the CGPath from your UIBezierPath into the CAShapeLayer, and then create a CABasicAnimation that changes the path property of the shape layer.

The trick to getting smooth shape animations is to have the same number of control points for every step in the animation. Thus you should not add more and more points to your path, but rather create a new path that contains a graph of the last n points of your waveform.

I would suggest keeping a ring buffer of the n points you want to graph, and building a GCPath/UIBezierPath out of that ring buffer. As you add more points, the older points would "age out" of the ring buffer and you'd always graph the same number of points.

Edit:

Ok, you need something simpler than a ring buffer: Let's call it a lastNElementsBuffer. It should let you add items, discarding the oldest element, and then always return the most recent elements added.

Here is a simple implementation:

public struct LastNItemsBuffer<T> {
    fileprivate var array: [T?]
    fileprivate var index = 0

    public init(count: Int) {
        array = [T?](repeating: nil, count: count)
    }

    public mutating func clear() {
        forceToValue(value: nil)
    }

    public mutating func forceToValue(value: T?) {
        let count = array.count
        array = [T?](repeating: value, count: count)
    }

    public mutating func write(_ element: T) {
        array[index % array.count] = element
        index += 1
    }
    public func lastNItems() -> [T] {
        var result = [T?]()
        for loop in 0..<array.count {
            result.append(array[(loop+index) % array.count])
        }
        return result.compactMap { $0 }
    }
}

If you create such a buffer of CGFloat values, and populate it with all zeros, you could then start saving new waveform values to it as they are read.

Then you'd create an animation that would create a path using the buffer of values, plus a new value, and then create an animation that shifts the new path to the left, revealing the new point.

I created a demo project on Github that shows the technique. You can download it here: https://github.com/DuncanMC/LastNItemsBufferGraph.git

Here is a sample animation:

Edit #2:

It sounds like you need a slightly different style of animation that what I did in my sample app. You should modify the method buildPath(plusValue:) in GraphView.swift to draw the style of graph you desire from the array of sample values (plus an optional new value). The rest of the sample app should work as written.

I updated the app to also offer a bar graph style similar to Apple's Voice memo app:

Edit #3:

In another thread you said you wanted to be able to allow the user to scroll back and forth through the graph, and proposed using a scroll view to manage that process.

The problem there is that your audio sample could be over a long time interval, and so the image of the whole waveform graph could be too large to hold in memory. (Imagine a graph of a 3 minute recording with a data-point for every 1/20th of a second, and each data-point is graphed 10 points wide by 200 points tall. Thats 3 * 60 * 20 = 3600 data points. If you use 10 points horizontally, on a 3X Retina display, that's 30 pixels wide per data point or 108,000 pixels wide, • 200 points • 3X = 600 pixels tall, or 64.8 MILLION pixels. At 3 bytes/pixel, (8 bits/color with no alpha) that's 194.4 million bytes of data, just for a 3 minute recording. Now let's say it's a 2 hour long recording. Time to run out of memory and crash.)

Instead I would say you should save a buffer of data points for your entire recording, scaled down to the smallest data type that would give you one point precision. You could probably use a single byte per data point. Save those points in a struct that also includes the samples/second.

Write a function that takes the graph data struct and time offset as input, and generates a CGPath for that time offset, plus or minus enough data points to make the graph wider than your display window on either side. Then you could animate the graph in either direction for forward or reverse playback. You could implement a tap gesture recognizer for letting the user drag the graph back and forth, or a slider, or whatever you needed. When you get to the end of the current graph, you'd just call your function to generate a new portion of the graph and display that new portion offset to the right screen location.

---

Related Topics