Remove diacritical marks (ń ǹ ň ñ ṅ ņ ṇ ṋ ṉ ̈ ɲ ƞ ᶇ ɳ ȵ) from Unicode chars
I have done this recently in Java:
public static final Pattern DIACRITICS_AND_FRIENDS
= Pattern.compile("[\\p{InCombiningDiacriticalMarks}\\p{IsLm}\\p{IsSk}]+");
private static String stripDiacritics(String str) {
str = Normalizer.normalize(str, Normalizer.Form.NFD);
str = DIACRITICS_AND_FRIENDS.matcher(str).replaceAll("");
return str;
}
This will do as you specified:
stripDiacritics("Björn") = Bjorn
but it will fail on for example Białystok, because the ł character is not diacritic.
If you want to have a full-blown string simplifier, you will need a second cleanup round, for some more special characters that are not diacritics. Is this map, I have included the most common special characters that appear in our customer names. It is not a complete list, but it will give you the idea how to do extend it. The immutableMap is just a simple class from google-collections.
public class StringSimplifier {
public static final char DEFAULT_REPLACE_CHAR = '-';
public static final String DEFAULT_REPLACE = String.valueOf(DEFAULT_REPLACE_CHAR);
private static final ImmutableMap<String, String> NONDIACRITICS = ImmutableMap.<String, String>builder()
//Remove crap strings with no sematics
.put(".", "")
.put("\"", "")
.put("'", "")
//Keep relevant characters as seperation
.put(" ", DEFAULT_REPLACE)
.put("]", DEFAULT_REPLACE)
.put("[", DEFAULT_REPLACE)
.put(")", DEFAULT_REPLACE)
.put("(", DEFAULT_REPLACE)
.put("=", DEFAULT_REPLACE)
.put("!", DEFAULT_REPLACE)
.put("/", DEFAULT_REPLACE)
.put("\\", DEFAULT_REPLACE)
.put("&", DEFAULT_REPLACE)
.put(",", DEFAULT_REPLACE)
.put("?", DEFAULT_REPLACE)
.put("°", DEFAULT_REPLACE) //Remove ?? is diacritic?
.put("|", DEFAULT_REPLACE)
.put("<", DEFAULT_REPLACE)
.put(">", DEFAULT_REPLACE)
.put(";", DEFAULT_REPLACE)
.put(":", DEFAULT_REPLACE)
.put("_", DEFAULT_REPLACE)
.put("#", DEFAULT_REPLACE)
.put("~", DEFAULT_REPLACE)
.put("+", DEFAULT_REPLACE)
.put("*", DEFAULT_REPLACE)
//Replace non-diacritics as their equivalent characters
.put("\u0141", "l") // BiaLystock
.put("\u0142", "l") // Bialystock
.put("ß", "ss")
.put("æ", "ae")
.put("ø", "o")
.put("©", "c")
.put("\u00D0", "d") // All Ð ð from http://de.wikipedia.org/wiki/%C3%90
.put("\u00F0", "d")
.put("\u0110", "d")
.put("\u0111", "d")
.put("\u0189", "d")
.put("\u0256", "d")
.put("\u00DE", "th") // thorn Þ
.put("\u00FE", "th") // thorn þ
.build();
public static String simplifiedString(String orig) {
String str = orig;
if (str == null) {
return null;
}
str = stripDiacritics(str);
str = stripNonDiacritics(str);
if (str.length() == 0) {
// Ugly special case to work around non-existing empty strings
// in Oracle. Store original crapstring as simplified.
// It would return an empty string if Oracle could store it.
return orig;
}
return str.toLowerCase();
}
private static String stripNonDiacritics(String orig) {
StringBuilder ret = new StringBuilder
String lastchar = null;
for (int i = 0; i < orig.length(); i++) {
String source = orig.substring(i, i + 1);
String replace = NONDIACRITICS.get(source);
String toReplace = replace == null ? String.valueOf(source) : replace;
if (DEFAULT_REPLACE.equals(lastchar) && DEFAULT_REPLACE.equals(toReplace)) {
toReplace = "";
} else {
lastchar = toReplace;
}
ret.append(toReplace);
}
if (ret.length() > 0 && DEFAULT_REPLACE_CHAR == ret.charAt(ret.length() - 1)) {
ret.deleteCharAt(ret.length() - 1);
}
return ret.toString();
}
/*
Special regular expression character ranges relevant for simplification -> see http://docstore.mik.ua/orelly/perl/prog3/ch05_04.htm
InCombiningDiacriticalMarks: special marks that are part of "normal" ä, ö, î etc..
IsSk: Symbol, Modifier see http://www.fileformat.info/info/unicode/category/Sk/list.htm
IsLm: Letter, Modifier see http://www.fileformat.info/info/unicode/category/Lm/list.htm
*/
public static final Pattern DIACRITICS_AND_FRIENDS
= Pattern.compile("[\\p{InCombiningDiacriticalMarks}\\p{IsLm}\\p{IsSk}]+");
private static String stripDiacritics(String str) {
str = Normalizer.normalize(str, Normalizer.Form.NFD);
str = DIACRITICS_AND_FRIENDS.matcher(str).replaceAll("");
return str;
}
}
Remove diacritics from string in Java
Using API level 9+ you can use the Normalizer class, e.g.
String normalized = Normalizer.normalize("âbĉdêéè", Form.NFD)
.replaceAll("\\p{InCombiningDiacriticalMarks}+", "");
(Keysers linked answer looks better, it cleans more crap)
This would return "abcdeee".
Removing accent marks (diacritics) from Latin characters for comparison
You can make use of java.text.Normalizer and a little regex to get rid of the diacritical marks.
public static String removeDiacriticalMarks(String string) {
return Normalizer.normalize(string, Form.NFD)
.replaceAll("\\p{InCombiningDiacriticalMarks}+", "");
}
Usage example:
String text = "mšk žilina";
String normalized = removeDiacriticalMarks(text);
System.out.println(normalized); // msk zilina
How to change diacritic characters to non-diacritic ones
Copying from my own answer to another question:
Instead of creating your own table, you could instead convert the text to normalization form D, where the characters are represented as a base character plus the diacritics (for instance, "á" will be replaced by "a" followed by a combining acute accent). You can then strip everything which is not an ASCII letter.
The tables still exist, but are now the ones from the Unicode standard.
You could also try NFKD instead of NFD, to catch even more cases.
References:
- http://unicode.org/reports/tr15/
- http://www.siao2.com/2005/02/19/376617.aspx
- http://www.siao2.com/2007/05/14/2629747.aspx
Remove diacritics from a string
There is a function that Wordpress uses and works nice. Here's the working code with output.
<?php
function seems_utf8($str)
{
$length = strlen($str);
for ($i=0; $i < $length; $i++) {
$c = ord($str[$i]);
if ($c < 0x80) $n = 0; # 0bbbbbbb
elseif (($c & 0xE0) == 0xC0) $n=1; # 110bbbbb
elseif (($c & 0xF0) == 0xE0) $n=2; # 1110bbbb
elseif (($c & 0xF8) == 0xF0) $n=3; # 11110bbb
elseif (($c & 0xFC) == 0xF8) $n=4; # 111110bb
elseif (($c & 0xFE) == 0xFC) $n=5; # 1111110b
else return false; # Does not match any model
for ($j=0; $j<$n; $j++) { # n bytes matching 10bbbbbb follow ?
if ((++$i == $length) || ((ord($str[$i]) & 0xC0) != 0x80))
return false;
}
}
return true;
}
/**
* Converts all accent characters to ASCII characters.
*
* If there are no accent characters, then the string given is just returned.
*
* @param string $string Text that might have accent characters
* @return string Filtered string with replaced "nice" characters.
*/
function remove_accents($string) {
if ( !preg_match('/[\x80-\xff]/', $string) )
return $string;
if (seems_utf8($string)) {
$chars = array(
// Decompositions for Latin-1 Supplement
chr(195).chr(128) => 'A', chr(195).chr(129) => 'A',
chr(195).chr(130) => 'A', chr(195).chr(131) => 'A',
chr(195).chr(132) => 'A', chr(195).chr(133) => 'A',
chr(195).chr(135) => 'C', chr(195).chr(136) => 'E',
chr(195).chr(137) => 'E', chr(195).chr(138) => 'E',
chr(195).chr(139) => 'E', chr(195).chr(140) => 'I',
chr(195).chr(141) => 'I', chr(195).chr(142) => 'I',
chr(195).chr(143) => 'I', chr(195).chr(145) => 'N',
chr(195).chr(146) => 'O', chr(195).chr(147) => 'O',
chr(195).chr(148) => 'O', chr(195).chr(149) => 'O',
chr(195).chr(150) => 'O', chr(195).chr(153) => 'U',
chr(195).chr(154) => 'U', chr(195).chr(155) => 'U',
chr(195).chr(156) => 'U', chr(195).chr(157) => 'Y',
chr(195).chr(159) => 's', chr(195).chr(160) => 'a',
chr(195).chr(161) => 'a', chr(195).chr(162) => 'a',
chr(195).chr(163) => 'a', chr(195).chr(164) => 'a',
chr(195).chr(165) => 'a', chr(195).chr(167) => 'c',
chr(195).chr(168) => 'e', chr(195).chr(169) => 'e',
chr(195).chr(170) => 'e', chr(195).chr(171) => 'e',
chr(195).chr(172) => 'i', chr(195).chr(173) => 'i',
chr(195).chr(174) => 'i', chr(195).chr(175) => 'i',
chr(195).chr(177) => 'n', chr(195).chr(178) => 'o',
chr(195).chr(179) => 'o', chr(195).chr(180) => 'o',
chr(195).chr(181) => 'o', chr(195).chr(182) => 'o',
chr(195).chr(182) => 'o', chr(195).chr(185) => 'u',
chr(195).chr(186) => 'u', chr(195).chr(187) => 'u',
chr(195).chr(188) => 'u', chr(195).chr(189) => 'y',
chr(195).chr(191) => 'y',
// Decompositions for Latin Extended-A
chr(196).chr(128) => 'A', chr(196).chr(129) => 'a',
chr(196).chr(130) => 'A', chr(196).chr(131) => 'a',
chr(196).chr(132) => 'A', chr(196).chr(133) => 'a',
chr(196).chr(134) => 'C', chr(196).chr(135) => 'c',
chr(196).chr(136) => 'C', chr(196).chr(137) => 'c',
chr(196).chr(138) => 'C', chr(196).chr(139) => 'c',
chr(196).chr(140) => 'C', chr(196).chr(141) => 'c',
chr(196).chr(142) => 'D', chr(196).chr(143) => 'd',
chr(196).chr(144) => 'D', chr(196).chr(145) => 'd',
chr(196).chr(146) => 'E', chr(196).chr(147) => 'e',
chr(196).chr(148) => 'E', chr(196).chr(149) => 'e',
chr(196).chr(150) => 'E', chr(196).chr(151) => 'e',
chr(196).chr(152) => 'E', chr(196).chr(153) => 'e',
chr(196).chr(154) => 'E', chr(196).chr(155) => 'e',
chr(196).chr(156) => 'G', chr(196).chr(157) => 'g',
chr(196).chr(158) => 'G', chr(196).chr(159) => 'g',
chr(196).chr(160) => 'G', chr(196).chr(161) => 'g',
chr(196).chr(162) => 'G', chr(196).chr(163) => 'g',
chr(196).chr(164) => 'H', chr(196).chr(165) => 'h',
chr(196).chr(166) => 'H', chr(196).chr(167) => 'h',
chr(196).chr(168) => 'I', chr(196).chr(169) => 'i',
chr(196).chr(170) => 'I', chr(196).chr(171) => 'i',
chr(196).chr(172) => 'I', chr(196).chr(173) => 'i',
chr(196).chr(174) => 'I', chr(196).chr(175) => 'i',
chr(196).chr(176) => 'I', chr(196).chr(177) => 'i',
chr(196).chr(178) => 'IJ',chr(196).chr(179) => 'ij',
chr(196).chr(180) => 'J', chr(196).chr(181) => 'j',
chr(196).chr(182) => 'K', chr(196).chr(183) => 'k',
chr(196).chr(184) => 'k', chr(196).chr(185) => 'L',
chr(196).chr(186) => 'l', chr(196).chr(187) => 'L',
chr(196).chr(188) => 'l', chr(196).chr(189) => 'L',
chr(196).chr(190) => 'l', chr(196).chr(191) => 'L',
chr(197).chr(128) => 'l', chr(197).chr(129) => 'L',
chr(197).chr(130) => 'l', chr(197).chr(131) => 'N',
chr(197).chr(132) => 'n', chr(197).chr(133) => 'N',
chr(197).chr(134) => 'n', chr(197).chr(135) => 'N',
chr(197).chr(136) => 'n', chr(197).chr(137) => 'N',
chr(197).chr(138) => 'n', chr(197).chr(139) => 'N',
chr(197).chr(140) => 'O', chr(197).chr(141) => 'o',
chr(197).chr(142) => 'O', chr(197).chr(143) => 'o',
chr(197).chr(144) => 'O', chr(197).chr(145) => 'o',
chr(197).chr(146) => 'OE',chr(197).chr(147) => 'oe',
chr(197).chr(148) => 'R',chr(197).chr(149) => 'r',
chr(197).chr(150) => 'R',chr(197).chr(151) => 'r',
chr(197).chr(152) => 'R',chr(197).chr(153) => 'r',
chr(197).chr(154) => 'S',chr(197).chr(155) => 's',
chr(197).chr(156) => 'S',chr(197).chr(157) => 's',
chr(197).chr(158) => 'S',chr(197).chr(159) => 's',
chr(197).chr(160) => 'S', chr(197).chr(161) => 's',
chr(197).chr(162) => 'T', chr(197).chr(163) => 't',
chr(197).chr(164) => 'T', chr(197).chr(165) => 't',
chr(197).chr(166) => 'T', chr(197).chr(167) => 't',
chr(197).chr(168) => 'U', chr(197).chr(169) => 'u',
chr(197).chr(170) => 'U', chr(197).chr(171) => 'u',
chr(197).chr(172) => 'U', chr(197).chr(173) => 'u',
chr(197).chr(174) => 'U', chr(197).chr(175) => 'u',
chr(197).chr(176) => 'U', chr(197).chr(177) => 'u',
chr(197).chr(178) => 'U', chr(197).chr(179) => 'u',
chr(197).chr(180) => 'W', chr(197).chr(181) => 'w',
chr(197).chr(182) => 'Y', chr(197).chr(183) => 'y',
chr(197).chr(184) => 'Y', chr(197).chr(185) => 'Z',
chr(197).chr(186) => 'z', chr(197).chr(187) => 'Z',
chr(197).chr(188) => 'z', chr(197).chr(189) => 'Z',
chr(197).chr(190) => 'z', chr(197).chr(191) => 's',
// Euro Sign
chr(226).chr(130).chr(172) => 'E',
// GBP (Pound) Sign
chr(194).chr(163) => '');
$string = strtr($string, $chars);
} else {
// Assume ISO-8859-1 if not UTF-8
$chars['in'] = chr(128).chr(131).chr(138).chr(142).chr(154).chr(158)
.chr(159).chr(162).chr(165).chr(181).chr(192).chr(193).chr(194)
.chr(195).chr(196).chr(197).chr(199).chr(200).chr(201).chr(202)
.chr(203).chr(204).chr(205).chr(206).chr(207).chr(209).chr(210)
.chr(211).chr(212).chr(213).chr(214).chr(216).chr(217).chr(218)
.chr(219).chr(220).chr(221).chr(224).chr(225).chr(226).chr(227)
.chr(228).chr(229).chr(231).chr(232).chr(233).chr(234).chr(235)
.chr(236).chr(237).chr(238).chr(239).chr(241).chr(242).chr(243)
.chr(244).chr(245).chr(246).chr(248).chr(249).chr(250).chr(251)
.chr(252).chr(253).chr(255);
$chars['out'] = "EfSZszYcYuAAAAAACEEEEIIIINOOOOOOUUUUYaaaaaaceeeeiiiinoooooouuuuyy";
$string = strtr($string, $chars['in'], $chars['out']);
$double_chars['in'] = array(chr(140), chr(156), chr(198), chr(208), chr(222), chr(223), chr(230), chr(240), chr(254));
$double_chars['out'] = array('OE', 'oe', 'AE', 'DH', 'TH', 'ss', 'ae', 'dh', 'th');
$string = str_replace($double_chars['in'], $double_chars['out'], $string);
}
return $string;
}
$str = "ľ š č ť ž ý á í é Č Á Ž Ý";
echo remove_accents($str); // Output: l s c t z y a i e C A Z Y
?>
Generic way to simplify string, remove diacritics
Some ICU magic:
echo "ë ö ø Я Ł ɲ æ å ñ 開 당" | uconv -x any-name | perl -wpne 's/ WITH [^}]+//g;' | uconv -x name-any | uconv -x any-latin -t iso-8859-1 -c | uconv -f iso-8859-1 -t ascii -x latin-ascii -c
yields
e o o A L n ae a n ki dang
This uses the cmdline tool uconv, but the same can be done with ICU's Java or C or C++ API, and ICU has bindings for almost any language.
Note Я -> A because that is the correct behavior. What you want is not how Unicode defines that character - blame KoЯn for abusing it.
Why is there NFC in NFD; [:Nonspacing Mark:] Remove; NFC?
Okay so canonical decomposition isn't limited to diacritics; I was given the example of Hangul syllables which can be split into many jamos. It can then makes sense to recompose such characters.
Related Topics
Why Does Runtime.Exec(String) Work for Some But Not All Commands
What Is @Modelattribute in Spring MVC
How to Create a Rest Client for Java
Initial Bytes Incorrect After Java Aes/Cbc Decryption
Comparing Strings with == Which Are Declared Final in Java
How to Monitor the Computer's Cpu, Memory, and Disk Usage in Java
Places Where Javabeans Are Used
How to Calculate "Time Ago" in Java
Preferred Way of Loading Resources in Java
Why Does Integer Division Code Give the Wrong Answer
How to Write a Custom JSON Deserializer for Gson
Simple Http Server in Java Using Only Java Se API
Compiling a Java Program into an Executable
Hibernate Annotations - Which Is Better, Field or Property Access
Java Client Certificates Over Https/Ssl