Nslayoutmanager Hides New Line Characters No Matter What I Do

NSLayoutManager hides new line characters no matter what I do

As I figured out, the default implementation of NSTypesetter's setNotShownAttribute: of the class doesn't change already generated glyphs in its glyph storage. So, call of super doesn't produce any effect. I just have to replace glyphs manually before calling super.

So, the most efficient implementation of showing invisible characters (you will see the difference while zooming the view) is this:

Limitations of this approach: if your app has to have multiple fonts in text view, then this approach might not be such a good idea, because the font of those displayed invisible characters will be different as well. And that's not what you might want to achieve.

Subclass NSLayoutManager and override setGlyphs to show space chars:

public override func setGlyphs(_ glyphs: UnsafePointer<CGGlyph>, properties props: UnsafePointer<NSGlyphProperty>, characterIndexes charIndexes: UnsafePointer<Int>, font aFont: Font, forGlyphRange glyphRange: NSRange) {
    var substring = (self.currentTextStorage.string as NSString).substring(with: glyphRange)

    // replace invisible characters with visible
    if PreferencesManager.shared.shouldShowInvisibles == true {
        substring = substring.replacingOccurrences(of: " ", with: "\u{00B7}")
    }

    // create a CFString
    let stringRef = substring as CFString
    let count = CFStringGetLength(stringRef)

    // convert processed string to the C-pointer
    let cfRange = CFRangeMake(0, count)
    let fontRef = CTFontCreateWithName(aFont.fontName as CFString?, aFont.pointSize, nil)
    let characters = UnsafeMutablePointer<UniChar>.allocate(capacity: MemoryLayout<UniChar>.size * count)
    CFStringGetCharacters(stringRef, cfRange, characters)

    // get glyphs for the pointer of characters
    let glyphsRef = UnsafeMutablePointer<CGGlyph>.allocate(capacity: MemoryLayout<CGGlyph>.size * count)
    CTFontGetGlyphsForCharacters(fontRef, characters, glyphsRef, count)

    // set those glyphs
    super.setGlyphs(glyphsRef, properties:props, characterIndexes: charIndexes, font: aFont, forGlyphRange: glyphRange)
}

Subclass NSATSTypesetter and assign it to your NSLayoutManager subclas. The subclass will display the new line characters and make sure that every invisible character will be drawn with a different color:

class CustomTypesetter: NSATSTypesetter {

    override func setNotShownAttribute(_ flag: Bool, forGlyphRange glyphRange: NSRange) {
        var theFlag = flag

        if PreferencesManager.shared.shouldShowInvisibles == true   {
            theFlag = false

            // add new line glyphs into the glyph storage
            var newLineGlyph = yourFont.glyph(withName: "paragraph")
            self.substituteGlyphs(in: glyphRange, withGlyphs: &newLineGlyph)

            // draw new line char with different color
            self.layoutManager?.addTemporaryAttribute(NSForegroundColorAttributeName, value: NSColor.invisibleTextColor, forCharacterRange: glyphRange)
        }

        super.setNotShownAttribute(theFlag, forGlyphRange: glyphRange)
    }

    /// Currently hadn't found any faster way to draw space glyphs with different color
    override func setParagraphGlyphRange(_ paragraphRange: NSRange, separatorGlyphRange paragraphSeparatorRange: NSRange) {
        super.setParagraphGlyphRange(paragraphRange, separatorGlyphRange: paragraphSeparatorRange)

        guard PreferencesManager.shared.shouldShowInvisibles == true else { return }

        if let substring = (self.layoutManager?.textStorage?.string as NSString?)?.substring(with: paragraphRange) {
            let expression = try? NSRegularExpression.init(pattern: "\\s", options: NSRegularExpression.Options.useUnicodeWordBoundaries)
            let sunstringRange = NSRange(location: 0, length: substring.characters.count)

            if let matches = expression?.matches(in: substring, options: NSRegularExpression.MatchingOptions.withoutAnchoringBounds, range: sunstringRange) {
                for match in matches {
                    let globalSubRange = NSRange(location: paragraphRange.location + match.range.location, length: 1)
                    self.layoutManager?.addTemporaryAttribute(NSForegroundColorAttributeName, value: Color.invisibleText, forCharacterRange: globalSubRange)
                }
            }
        }
    }
}

To show/hide invisible characters just call:

let storageRange = NSRange(location: 0, length: currentTextStorage.length)
layoutManager.invalidateGlyphs(forCharacterRange: storageRange, changeInLength: 0, actualCharacterRange: nil)
layoutManager.ensureGlyphs(forGlyphRange: storageRange)

NSLayoutManager Text new line seems to start from bottom of rectangle and go up

Override isFlipped to return YES in your subclass of NSView.

drawGlyphsForGlyphRange in Swift (Displaying Invisible characters). replaceGlyphAtIndex deprecated

I found another way to display Glyphs. (Posting my code below as it may be useful to others)

override func drawGlyphsForGlyphRange(glyphsToShow: NSRange, atPoint origin: NSPoint) {
    if let storage = self.textStorage {
        let s = storage.string
        let startIndex = s.startIndex

        var padding:CGFloat = 0
        for glyphIndex in glyphsToShow.location ..< glyphsToShow.location + glyphsToShow.length {
            let characterIndex = self.characterIndexForGlyphAtIndex(glyphIndex)
            if characterIndex < s.characters.count
            {
                var glyphStr = ""
                let ch = s[startIndex.advancedBy(characterIndex)]
                switch ch {
                case " ": //Blank Space
                    glyphStr = periodCenteredUniCodeStr
                case "\n": //New Line
                    glyphStr = lineBreakUniCodeStr
                case newLineUniCodeStr:
                    glyphStr = lineBreakUniCodeStr
                    padding += 5
                default:
                    break
                }
                var glyphPoint = self.locationForGlyphAtIndex(glyphIndex)
                let glyphRect = self.lineFragmentRectForGlyphAtIndex(glyphIndex, effectiveRange: nil)
                if glyphStr.characters.count > 0{
                    glyphPoint.x = glyphPoint.x + glyphRect.origin.x
                    glyphPoint.y = glyphRect.origin.y
                    NSString(string: glyphStr).drawInRect(NSMakeRect(glyphPoint.x, glyphPoint.y, 10, 10), withAttributes: nil)
                }
            }else{
                print("Wrong count here")
            }
        }

    }
    super.drawGlyphsForGlyphRange(glyphsToShow, atPoint: origin)
}

NSLayoutManager glyph generation and caret position

NSLayoutManager is surprisingly poorly documented, and understanding its inner workings required looking through implementations in random public repositories.

To change glyphs in a range and have the caret position correctly, you first need to invalidate glyphs for both of the changed ranges, and then invalidate the layout for the range the caret is about to be positioned in.

Then, after making sure that there is a current graphic context, glyphs can be redrawn synchronously. After this, updating insertion point works normally.

In the example below, I have two objects, line and prevLine which contain ranges for both the line on which caret is positioned now, and the one caret moved away from.

-(void)hideAndShowMarkup {
    [self.layoutManager invalidateGlyphsForCharacterRange:line.range changeInLength:0 actualCharacterRange:nil];
    [self.layoutManager invalidateGlyphsForCharacterRange:prevLine.range changeInLength:0 actualCharacterRange:nil];
    [self.layoutManager invalidateLayoutForCharacterRange:line.range actualCharacterRange:nil];
        
    if (NSGraphicsContext.currentContext) {
        [self.layoutManager drawGlyphsForGlyphRange:line.range atPoint:self.frame.origin];
        [self.layoutManager drawGlyphsForGlyphRange:prevLine.range atPoint:self.frame.origin];
    }
    
    [self updateInsertionPointStateAndRestartTimer:NO];
}

Hide Markdown Characters with NSLayoutManager in Swift

2022 Disclaimer

While I had some good results running this piece of code when I originally submitted this answer, another SO user (Tim S.) warned me that in some cases applying the .null glyph properties to some glyphs make cause the app the hang or crash.

From what I could gather this only happens with the .null property, and around glyph 8192 (2^13)... I have no idea why, and honestly it looks like a TextKit bug (or at least not something the TextKit engineers did expect the framework to be used for).

For modern apps, I'd advise you to take a look a TextKit 2, which is supposed to abstract away glyphs handling and simplify all that stuff (disclaimer in the disclaimer : I haven't tried it yet).

Foreword

I implemented this method to achieve something similar in my app. Keep in mind that this API is very poorly documented, so my solution is based on trial and error instead of a deep understanding of all the moving parts here.

In short: it should work but use at you own risk :)

Note also that I went into a lot of details in this answer in the hope to make it accessible to any Swift developer, even one without a background in Objective-C or C. You probably already know some of the things detailed hereafter.

On TextKit and Glyphs

One of the things that is important to understand is that a glyph is the visual representation of one or more characters, as explained in WWDC 2018 Session 221 "TextKit Best Practices" :

slide of session 221 explaining the difference between characters and glyphs

I'd recommend watching the whole talk. It's not super helpful in the particular case of understanding how layoutManager(_:shouldGenerateGlyphs:properties:characterIndexes:font:forGlyphRange:) works, but it gives a good amount of info on how TextKit works in general.

Understanding `shouldGenerateGlyphs`

So. From what I understand, each time NSLayoutManager is about to generate a new glyph before rendering them, it will give you a chance to modify this glyph by calling layoutManager(_:shouldGenerateGlyphs:properties:characterIndexes:font:forGlyphRange:).

Modifying Glyphs

As per the doc, if you want to modify the glyphs you should do so in this method by calling setGlyphs(_:properties:characterIndexes:font:forGlyphRange:).

Lucky for us, setGlyphs expects the exact same arguments as passed to us in shouldGenerateGlyphs. This means that in theory you could implement shouldGenerateGlyphs with just a call to setGlyphs and all would be well (but that wouldn't be super useful).

Return Value

The doc also says that the return value of shouldGenerateGlyphs should be "The actual glyph range stored in this method". It doesn't make much sense, as the expected return type is Int and not NSRange as one might expect. From trial and error, I think the framework expects us here to return the number of modified glyphs in the passed glyphRange, starting at index 0 (more on that later).

Also, "glyph range stored in this method" refers the call to setGlyphs, which will store the newly generated glyphs internally (imo this is very poorly worded).

A Not So Useful Implementation

So here's a correct implementation of shouldGenerateGlyphs (which... does nothing):

func layoutManager(_ layoutManager: NSLayoutManager, shouldGenerateGlyphs glyphs: UnsafePointer<CGGlyph>, properties: UnsafePointer<NSLayoutManager.GlyphProperty>, characterIndexes: UnsafePointer<Int>, font: UIFont, forGlyphRange glyphRange: NSRange) -> Int {
    layoutManager.setGlyphs(glyphs, properties: fixedPropertiesPointer, characterIndexes: characterIndexes, font: font, forGlyphRange: glyphRange)

    return glyphRange.length
}

it should also be equivalent to just returning 0 from the method:

By returning 0, it can indicate for the layout manager to do the default processing.

Doing Something Useful

So now, how can we edit our glyphs properties to make this method do something useful (like hiding glyphs)?

Accessing the Arguments Values

Most of the arguments of shouldGenerateGlyphs are UnsafePointer. That's the TextKit C API leaking in the Swift layer, and one of the things that make implementing this method a hassle in the first place.

A key point is that all the arguments of type UnsafePointer here are arrays (in C, SomeType * — or its Swift equivalent UnsafePointer<SomeType> — is the how we represent an array), and those arrays are all of length glyphRange.length. That's indirectly documented in the setGlyphs method:

Each array has glyphRange.length items

What this means is that with the nice UnsafePointer API Apple has given us, we can iterate on the elements of these array with a loop like this:

for i in 0 ..< glyphRange.length {
    print(properties[i])
}

Under the hood, UnsafePointer will do pointer arithmetic to access memory at the right address given any index passed to the subscript. I'd recommend reading the UnsafePointer documentation, this is really cool stuff.

Passing Something Useful to setGlyphs

We're now able to print the content of our arguments, and inspect what properties the framework's given us for each glyph. Now, how do we modify those and pass the result to setGlyphs?

First, it's important to note that while we could modify the properties argument directly, it's probably a bad idea, because that chunk of memory isn't owned by us and we have no idea what the framework will do with this memory once we exit the method.

So the right way to go about this is to create our own array of glyph properties, and then pass that to setGlyphs:

var modifiedGlyphProperties = [NSLayoutManager.GlyphProperty]()
for i in 0 ..< glyphRange.length {
    // This contains the default properties for the glyph at index i set by the framework.
    var glyphProperties = properties[i]
    // We add the property we want to the mix. GlyphProperty is an OptionSet, we can use `.insert()` to do that.
    glyphProperties.insert(.null)
    // Append this glyph properties to our properties array.
    modifiedGlyphProperties.append(glyphProperties)
}

// Convert our Swift array to the UnsafePointer `setGlyphs` expects.
modifiedGlyphProperties.withUnsafeBufferPointer { modifiedGlyphPropertiesBufferPointer in
    guard let modifiedGlyphPropertiesPointer = modifiedGlyphPropertiesBufferPointer.baseAddress else {
        fatalError("Could not get base address of modifiedGlyphProperties")
    }

    // Call setGlyphs with the modified array.
    layoutManager.setGlyphs(glyphs, properties: modifiedGlyphPropertiesPointer, characterIndexes: characterIndexes, font: font, forGlyphRange: glyphRange)
}

return glyphRange.length

It's important to read the original glyph properties from the properties array and adding your custom ones to this base value (with the .insert() method). Otherwise you'd overwrite the default properties of your glyphs and weird things would happen (I've seen \n characters not inserting a visual line break anymore for example).

Deciding Which Glyphs to Hide

The previous implementation should work fine, but right now we're unconditionally hiding all generated glyphs, and it would be much more useful if we could hide only some of them (in your case when the glyph is *).

Hiding Based on Characters Values

To do that, you'll probably need to access the characters used to generate the final glyph. However, the framework doesn't give you the characters but their index in the string for each generated glyph. You'll need to iterate over these indexes and look into your NSTextStorage to find the corresponding characters.

Unfortunately, this is not a trivial task: Foundation uses UTF-16 code units to represent strings internally (that's what NSString and NSAttributedString use under the hood). So what the framework gives us with characterIndexes is not the indexes of "characters" in the usual sense of the word, but the indexes of UTF-16 code units^†.

Most of the time, each UTF-16 code unit will be used to generate a unique glyph, but in some cases multiple code units will be used to generate a unique glyph (this is called a UTF-16 surrogate pair, and is common when handling string with emojis). I'd recommend testing your code with some more "exotic" strings like for example:

textView.text = "Officiellement nous (‍‍‍) vivons dans un cha\u{0302}teau 海"

So, to be able to compare our characters, we first need to convert them to a simple representation of what we usually mean by "character":

/// Returns the extended grapheme cluster at `index` in an UTF16View, merging a UTF-16 surrogate pair if needed.
private func characterFromUTF16CodeUnits(_ utf16CodeUnits: String.UTF16View, at index: Int) -> Character {
    let codeUnitIndex = utf16CodeUnits.index(utf16CodeUnits.startIndex, offsetBy: index)
    let codeUnit = utf16CodeUnits[codeUnitIndex]

    if UTF16.isLeadSurrogate(codeUnit) {
        let nextCodeUnit = utf16CodeUnits[utf16CodeUnits.index(after: codeUnitIndex)]
        let codeUnits = [codeUnit, nextCodeUnit]
        let str = String(utf16CodeUnits: codeUnits, count: 2)
        return Character(str)
    } else if UTF16.isTrailSurrogate(codeUnit) {
        let previousCodeUnit = utf16CodeUnits[utf16CodeUnits.index(before: codeUnitIndex)]
        let codeUnits = [previousCodeUnit, codeUnit]
        let str = String(utf16CodeUnits: codeUnits, count: 2)
        return Character(str)
    } else {
        let unicodeScalar = UnicodeScalar(codeUnit)!
        return Character(unicodeScalar)
    }
}

Then we can use this function to extract the characters from our textStorage, and test them:

// First, make sure we'll be able to access the NSTextStorage.
guard let textStorage = layoutManager.textStorage else {
    fatalError("No textStorage was associated to this layoutManager")
}

// Access the characters.
let utf16CodeUnits = textStorage.string.utf16
var modifiedGlyphProperties = [NSLayoutManager.GlyphProperty]()
for i in 0 ..< glyphRange.length {
    var glyphProperties = properties[i]
    let character = characterFromUTF16CodeUnits(utf16CodeUnits, at: characterIndex)

    // Do something with `character`, e.g.:
    if character == "*" {
        glyphProperties.insert(.null)
    }
    
    modifiedGlyphProperties.append(glyphProperties)
}
    
// Convert our Swift array to the UnsafePointer `setGlyphs` expects.
modifiedGlyphProperties.withUnsafeBufferPointer { modifiedGlyphPropertiesBufferPointer in
    guard let modifiedGlyphPropertiesPointer = modifiedGlyphPropertiesBufferPointer.baseAddress else {
        fatalError("Could not get base address of modifiedGlyphProperties")
    }

    // Call setGlyphs with the modified array.
    layoutManager.setGlyphs(glyphs, properties: modifiedGlyphPropertiesPointer, characterIndexes: characterIndexes, font: font, forGlyphRange: glyphRange)
}

return glyphRange.length

Note that in the case of surrogate pairs, the loop will be executed twice (once on the lead surrogate, and once on the trail surrogate), and you'll end up comparing the same resulting character twice. This is fine though as you need to apply the same modification you want on both "parts" of the generated glyph.

Hiding Based on the TextStorage String Attributes

That's not what you've asked for in your question, but for completion's sake (and because it's what I do in my app), here how you can access your textStorage string attributes to hide some glyphs (in this example I'll hide all the parts of the text with an hypertext link):

// First, make sure we'll be able to access the NSTextStorage.
guard let textStorage = layoutManager.textStorage else {
    fatalError("No textStorage was associated to this layoutManager")
}

// Get the first and last characters indexes for this glyph range,
// and from that create the characters indexes range.
let firstCharIndex = characterIndexes[0]
let lastCharIndex = characterIndexes[glyphRange.length - 1]
let charactersRange = NSRange(location: firstCharIndex, length: lastCharIndex - firstCharIndex + 1)

var hiddenRanges = [NSRange]()
textStorage.enumerateAttributes(in: charactersRange, options: []) { attributes, range, _ in
    for attribute in attributes where attribute.key == .link {
        hiddenRanges.append(range)
    }
}

var modifiedGlyphProperties = [NSLayoutManager.GlyphProperty]()
for i in 0 ..< glyphRange.length {
    let characterIndex = characterIndexes[i]
    var glyphProperties = properties[i]

    let matchingHiddenRanges = hiddenRanges.filter { NSLocationInRange(characterIndex, $0) }
    if !matchingHiddenRanges.isEmpty {
        glyphProperties.insert(.null)
    }

    modifiedGlyphProperties.append(glyphProperties)
}

// Convert our Swift array to the UnsafePointer `setGlyphs` expects.
modifiedGlyphProperties.withUnsafeBufferPointer { modifiedGlyphPropertiesBufferPointer in
    guard let modifiedGlyphPropertiesPointer = modifiedGlyphPropertiesBufferPointer.baseAddress else {
        fatalError("Could not get base address of modifiedGlyphProperties")
    }

    // Call setGlyphs with the modified array.
    layoutManager.setGlyphs(glyphs, properties: modifiedGlyphPropertiesPointer, characterIndexes: characterIndexes, font: font, forGlyphRange: glyphRange)
}

return glyphRange.length

^† To understand the differences between those, I'd recommend reading the Swift Documentation on "Strings and Characters". Note also that what the framework calls "character" here is not the same as what Swift calls a Character (or "Extended Grapheme Clusters"). Again, "character" for the TextKit framework is an UTF-16 code unit (represented in Swift by Unicode.UTF16.CodeUnit).

Update 2020-04-16: Make use of .withUnsafeBufferPointer to convert the modifiedGlyphProperties array to an UnsafePointer. It removes the need to have an instance variable of the array to keep it alive in memory.

UITextView Draw Invisible/Whitespace Characters

I found a better solution than setting the showsInvisibleCharacters property of NSLayoutManager to true, by subclassing NSLayoutManager and overriding the method drawBackgroundForGlyphRange(NSRange, CGPoint), which allows for custom drawings for each whitespace character, for example:

class LayoutManager : NSLayoutManager {

    var text: String? { return textStorage?.string }

    var font: UIFont = UIFont.systemFontOfSize(UIFont.systemFontSize()) { 
        didSet { 
            guard let text = self.text else { return }
            let textRange = NSMakeRange(0, (text as NSString).length)                
            invalidateGlyphsForCharactersInRange(textRange, actualCharacterRange: nil)
            invalidateCharacterAttributesForCharactersInRange(textRange, actualCharacterRange: nil)
        }
    }

    override func drawBackgroundForGlyphRange(glyphsToShow: NSRange, atPoint origin: CGPoint) {

        super.drawBackgroundForGlyphRange(glyphsToShow, atPoint:origin)

        guard let text = self.text else { return }

        enumerateLineFragmentsForGlyphRange(glyphsToShow)
        { (rect: CGRect, usedRect: CGRect, textContainer: NSTextContainer, glyphRange: NSRange, stop: UnsafeMutablePointer<ObjCBool>) -> Void in

            let characterRange = self.characterRangeForGlyphRange(glyphRange, actualGlyphRange: nil)

            // Draw invisible tab space characters

            let line = (self.text as NSString).substringWithRange(characterRange)

            do {

                let expr = try NSRegularExpression(pattern: "\t", options: [])

                expr.enumerateMatchesInString(line, options: [.ReportProgress], range: line.range) 
                { (result: NSTextCheckingResult?, flags: NSMatchingFlags, stop: UnsafeMutablePointer<ObjCBool>) in

                    if let result = result {

                        let range = NSMakeRange(result.range.location + characterRange.location, result.range.length)
                        let characterRect = self.boundingRectForGlyphRange(range, inTextContainer: textContainer)

                        let symbol = "\u{21E5}"
                        let attrs = [NSFontAttributeName : Font]
                        let height = (symbol as NSString).sizeWithAttributes(attrs).height
                        symbol.drawInRect(CGRectOffset(characterRect, 1.0, height * 0.5, withAttributes: attrs)

                    }

                }

            } catch let error as NSError {
                print(error.localizedDescription)
            }

        }

    }

}

Invalid glyph index when setting viewController's layoutManager for NSTextStorage subclass

It looks like your layout manager hasn't been properly removed from its original NSTextStorage object. Call [self.readerTextView.textStorage removeLayoutManager:self.readerTextView.layoutManager] before you add the layout manager to your custom text storage sub class.

In general, I've found TextKit to not play well when mixing nib/xib/storyboard created UITextViews with custom subclasses of the TextKit objects. You can get around some of this by overriding - (id)awakeAfterUsingCoder:(NSCoder *)aDecoder on UITextView and constructing the text system using the classes you want, but normally I'd recommend sticking to in-code custom text system creation.

Nslayoutmanager Hides New Line Characters No Matter What I Do