How Can a Label Control Display Japanese Characters Properly When the Font Used Doesn't Support This Language

How can a Label control display Japanese characters properly when the Font used doesn't support this language?

About Font fallback

System feature, tied to the International Font Management. It automates the selection of a font other than the font selected by the user in an application, to represent glyphs that the font in use can't handle.

See also: International Font Enumeration and Selection

A selection of predefined font substitutes can be found in the System Registry:

HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows NT\CurrentVersion\FontLink\SystemLink

An example and some notes, in relation to the RichTextBox/RichEdit control, are reported here:

Some keys change my RichTextBox font

The control is created with EM_SETLANGOPTIONS set to the IMF_AUTOFONT option.

The predefined behaviour, when an Unicode subset cannot be displayed by the current font, it's an automatic font fallback. The text selection reports the font substitute reference.

A font substitute can be selected using the Unicode Subset Bitfields when the default mapping doesn't return a direct match.

Any font subsitution is in relation with the current system language and the available installed fonts. A number of fonts is specifically installed by the System to support this feature. The fallback font are usually marked as "Hidden" in the System Font repository (\Windows\Fonts in Windows). These can be marked as usable/enumerable, using the Show command in the toolbar button or contextual menu.

(Arial Unicode is among these).

.NET Localization: Japanese Characters Display as Squares

Any .NET code, including Windows Forms, uses Unicode encoded as UTF16. Your problem isn't likely to be an encoding issue, that produces question marks instead of squares. Getting a square indicates you are using a font that is missing the required glyph to display the Japanese character.

You can use the charmap.exe applet to find out what glyphs are supported by the font you use. If they are missing then the operating system is forced to fall back to a substitute font and fail at it when it cannot find one. Displaying squares is then all it can do. An old operating system version is a very likely cause for this mishap, particularly so for XP without the optional East Asian fonts installed.

Do note that this font problem is very unlikely to be an issue on a machine that boots the Japanese version of Windows. It will have to proper fonts to display Japanese text of course. You can get specific language versions of Windows through an MSDN subscription. At least get one for your QA staff so they can verify that everything works correctly.

How to render extended unicode character in PrintPreviewDialog

By looping through all my installed fonts I was able to find a font that displays the correct character. In my case it was "Segoe UI Symbol".

Why does a Latin-characters-only Java font claim to support Asian characters, even though it does not?

I finally figured it out. There were a number of underlying causes, which was further hindered by an added dose of cross-platform variability.

JFreeChart Renders Text in the Wrong Location Because It Uses a Different Font Object

The layout problem occurred because JFreeChart was inadvertently calculating the metrics for the layout using a different Font object than the one AWT actually uses to render the font. (For reference, JFreeChart's calculation happens in org.jfree.text#getTextBounds.)

The reason for the different Font object is a result of the implicit "magic manipulation" mentioned in the question, which is performed inside of java.awt.font.TextLayout#singleFont.

Those three lines of magic manipulation can be condensed to just this:

font = Font.getFont(font.getAttributes())

In English, this asks the font manager to give us a new Font object based on the "attributes" (name, family, point size, etc) of the supplied font. Under certain circumstances, the Font it gives back to you will be different from the Font you originally started with.

To correct the metrics (and thus fix the layout), the fix is to run the one-liner above on your own Font object before setting the font in JFreeChart objects.

After doing this, the layout worked fine for me, as did the Japanese characters. It should fix the layout for you too, although it may not show the Japanese characters correctly for you. Read below about native fonts to understand why.

The Mac OS X Font Manager Prefers to Return Native Fonts Even If You Feed it a Physical TTF File

The layout of the text was fixed by the above change...but why does this happen? Under what circumstances would the FontManager actually give us back a different type of Font object than the one we provided?

There are many reasons, but at least on Mac OS X, the reason related to the problem is that the font manager seems to prefer to return native fonts whenever possible.

In other words, if you create a new font from a physical TTF font named "Foobar" using Font.createFont, and then call Font.getFont() with attributes derived from your "Foobar" physical font...so long as OS X already has a Foobar font installed, the font manager will give you back a CFont object rather than the TrueTypeFont object you were expecting. This seems to hold true even if you register the font through GraphicsEnvironment.getLocalGraphicsEnvironment().registerFont.

In my case, this threw a red herring into the investigation: I already had the "Source Sans" font installed on my Mac, which meant that I was getting different results from people who did not.

Mac OS X Native Fonts Always Support Asian Characters

The crux of the matter is that Mac OS X CFont objects always support Asian character sets. I am unclear of the exact mechanism that allows this, but I suspect that it's some sort of fallback font feature of OS X itself and not Java. In either case, a CFont always claims to (and is truly able to) render Asian characters with the correct glyphs.

This makes clear the mechanism that allowed the original problem to occur:

we created a physical Font from a physical TTF file, which does not itself support Japanese.
the same physical font as above was also installed in my Mac OS X Font Book
when calculating the layout of the chart, JFreeChart asked the physical Font object for the metrics of the Japanese text. The physical Font could not do this correctly because it does not support Asian character sets.
when actually drawing the chart, the magic manipulation in TextLayout#singleFont caused it to obtain a CFont object and draw the glyph using the same-named native font, versus the physical TrueTypeFont. Thus, the glyphs were correct but they were not positioned properly.

You Will Get Different Results Depending on Whether You Registered the Font and Whether You Have The Font Installed in Your OS

If you call Font.getFont() with the attributes from a created TTF font, you will get one of three different results, depending on whether the font is registered and whether you have the same font installed natively:

If you do have a native platform font installed with the same name as your TTF font (regardless of whether you registered the font or not), you will get an Asian-supporting CFont for the font you wanted.
If you registered the TTF Font in the GraphicsEnvironment but you do not have a native font of the same name, calling Font.getFont() will return a physical TrueTypeFont object back. This gives you the font you want, but you don't get Asian characters.
If you did not register the TTF Font and you also do not have a native font of the same name, calling Font.getFont() return an Asian-supporting CFont, but it will not be the font you requested.

In hindsight, none of this is entirely surprising. Leading to:

I Was Inadvertently Using the Wrong Font

In the production app, I was creating a font, but I forgot to initially register it with the GraphicsEnvironment. If you haven't registered a font when you perform the magic manipulation above, Font.getFont() doesn't know how to retrieve it and you get a backup font instead. Oops.

On Windows, Mac and Linux, this backup font generally seems to be Dialog, which is a logical (composite) font that supports Asian characters. At least in Java 7u72, the Dialog font defaults to the following fonts for Western alphabets:

Mac: Lucida Grande
Linux (CentOS): Lucida Sans
Windows: Arial

This mistake was actually a good thing for our Asian users, because it meant that their character sets rendered as expected with the logical font...although the Western users were not getting the character sets that we wanted.

Since it had been rendering in the wrong fonts and we needed to fix the Japanese layout anyway, I decided that I would be better off trying to standardize on one single common font for future releases (and thus coming closer to trashgod's suggestions).

Additionally, the app has font rendering quality requirements that may not always permit the use of certain fonts, so a reasonable decision seemed to be to try to configure the app to use Lucida Sans, which is the one physical font that is included by Oracle in all copies of Java. But...

Lucida Sans Doesn't Play Well with Asian Characters on All Platforms

The decision to try using Lucida Sans seemed reasonable...but I quickly found out that there are platform differences in how Lucida Sans is handled. On Linux and Windows, if you ask for a copy of the "Lucida Sans" font, you get a physical TrueTypeFont object. But that font doesn't support Asian characters.

The same problem holds true on Mac OS X if you request "Lucida Sans"...but if you ask for the slightly different name "LucidaSans" (note the lack of space), then you get a CFont object that supports Lucida Sans as well as Asian characters, so you can have your cake and eat it too.

On other platforms, requesting "LucidaSans" yields a copy of the standard Dialog font because there is no such font and Java is returning its default. On Linux, you are somewhat lucky here because Dialog actually defaults to Lucida Sans for Western text (and it also uses a decent fallback font for Asian characters).

This gives us a path to get (almost) the same physical font on all platforms, and which also supports Asian characters, by requesting fonts with these names:

Mac OS X: "LucidaSans" (yielding Lucida Sans + Asian backup fonts)
Linux: "Dialog" (yielding Lucida Sans + Asian backup fonts)
Windows: "Dialog" (yielding Arial + Asian backup fonts)

I've pored over the fonts.properties on Windows and I could not find a font sequence that defaulted to Lucida Sans, so it looks like our Windows users will need to get stuck with Arial...but at least it's not that visually different from Lucida Sans, and the Windows font rendering quality is reasonable.

Where Did Everything End Up?

In sum, we're now pretty much just using platform fonts. (I am sure that @trashgod is having a good chuckle right now!) Both Mac and Linux servers get Lucida Sans, Windows gets Arial, the rendering quality is good, and everyone is happy!

Unicode Font Support - WinForms Label

Thanks to @HansPassant, I realized what the issue here was.

The JSON file in question was an embedded resource, and was set to FileType.Binary.

This was unnecessary, so I changed this to FileType.Text, which allows the Encoding property to be set.

With Encoding as Unicode (UTF-8) - Codepage 65001, this is now displaying as intended.

Displaying Japanese fonts in source code using Visual Studio

Presumably VS is interpreting the file with the wrong encoding.

Reopen it using "File -> Open -> File... -> Open -> Open With... -> Source Code (Text) Editor With Encoding" and try various encodings.

How Can a Label Control Display Japanese Characters Properly When the Font Used Doesn't Support This Language