How to unescape a Java string literal in Java?
The Problem
The org.apache.commons.lang.StringEscapeUtils.unescapeJava() given here as another answer is really very little help at all.
- It forgets about
\0for null. - It doesn’t handle octal at all.
- It can’t handle the sorts of escapes admitted by the
java.util.regex.Pattern.compile()and everything that uses it, including\a,\e, and especially\cX. - It has no support for logical Unicode code points by number, only for UTF-16.
- This looks like UCS-2 code, not UTF-16 code: they use the depreciated
charAtinterface instead of thecodePointinterface, thus promulgating the delusion that a Javacharis guaranteed to hold a Unicode character. It’s not. They only get away with this because no UTF-16 surrogate will wind up looking for anything they’re looking for.
The Solution
I wrote a string unescaper which solves the OP’s question without all the irritations of the Apache code.
/*
*
* unescape_perl_string()
*
* Tom Christiansen <tchrist@perl.com>
* Sun Nov 28 12:55:24 MST 2010
*
* It's completely ridiculous that there's no standard
* unescape_java_string function. Since I have to do the
* damn thing myself, I might as well make it halfway useful
* by supporting things Java was too stupid to consider in
* strings:
*
* => "?" items are additions to Java string escapes
* but normal in Java regexes
*
* => "!" items are also additions to Java regex escapes
*
* Standard singletons: ?\a ?\e \f \n \r \t
*
* NB: \b is unsupported as backspace so it can pass-through
* to the regex translator untouched; I refuse to make anyone
* doublebackslash it as doublebackslashing is a Java idiocy
* I desperately wish would die out. There are plenty of
* other ways to write it:
*
* \cH, \12, \012, \x08 \x{8}, \u0008, \U00000008
*
* Octal escapes: \0 \0N \0NN \N \NN \NNN
* Can range up to !\777 not \377
*
* TODO: add !\o{NNNNN}
* last Unicode is 4177777
* maxint is 37777777777
*
* Control chars: ?\cX
* Means: ord(X) ^ ord('@')
*
* Old hex escapes: \xXX
* unbraced must be 2 xdigits
*
* Perl hex escapes: !\x{XXX} braced may be 1-8 xdigits
* NB: proper Unicode never needs more than 6, as highest
* valid codepoint is 0x10FFFF, not maxint 0xFFFFFFFF
*
* Lame Java escape: \[IDIOT JAVA PREPROCESSOR]uXXXX must be
* exactly 4 xdigits;
*
* I can't write XXXX in this comment where it belongs
* because the damned Java Preprocessor can't mind its
* own business. Idiots!
*
* Lame Python escape: !\UXXXXXXXX must be exactly 8 xdigits
*
* TODO: Perl translation escapes: \Q \U \L \E \[IDIOT JAVA PREPROCESSOR]u \l
* These are not so important to cover if you're passing the
* result to Pattern.compile(), since it handles them for you
* further downstream. Hm, what about \[IDIOT JAVA PREPROCESSOR]u?
*
*/
public final static
String unescape_perl_string(String oldstr) {
/*
* In contrast to fixing Java's broken regex charclasses,
* this one need be no bigger, as unescaping shrinks the string
* here, where in the other one, it grows it.
*/
StringBuffer newstr = new StringBuffer(oldstr.length());
boolean saw_backslash = false;
for (int i = 0; i < oldstr.length(); i++) {
int cp = oldstr.codePointAt(i);
if (oldstr.codePointAt(i) > Character.MAX_VALUE) {
i++; /****WE HATES UTF-16! WE HATES IT FOREVERSES!!!****/
}
if (!saw_backslash) {
if (cp == '\\') {
saw_backslash = true;
} else {
newstr.append(Character.toChars(cp));
}
continue; /* switch */
}
if (cp == '\\') {
saw_backslash = false;
newstr.append('\\');
newstr.append('\\');
continue; /* switch */
}
switch (cp) {
case 'r': newstr.append('\r');
break; /* switch */
case 'n': newstr.append('\n');
break; /* switch */
case 'f': newstr.append('\f');
break; /* switch */
/* PASS a \b THROUGH!! */
case 'b': newstr.append("\\b");
break; /* switch */
case 't': newstr.append('\t');
break; /* switch */
case 'a': newstr.append('\007');
break; /* switch */
case 'e': newstr.append('\033');
break; /* switch */
/*
* A "control" character is what you get when you xor its
* codepoint with '@'==64. This only makes sense for ASCII,
* and may not yield a "control" character after all.
*
* Strange but true: "\c{" is ";", "\c}" is "=", etc.
*/
case 'c': {
if (++i == oldstr.length()) { die("trailing \\c"); }
cp = oldstr.codePointAt(i);
/*
* don't need to grok surrogates, as next line blows them up
*/
if (cp > 0x7f) { die("expected ASCII after \\c"); }
newstr.append(Character.toChars(cp ^ 64));
break; /* switch */
}
case '8':
case '9': die("illegal octal digit");
/* NOTREACHED */
/*
* may be 0 to 2 octal digits following this one
* so back up one for fallthrough to next case;
* unread this digit and fall through to next case.
*/
case '1':
case '2':
case '3':
case '4':
case '5':
case '6':
case '7': --i;
/* FALLTHROUGH */
/*
* Can have 0, 1, or 2 octal digits following a 0
* this permits larger values than octal 377, up to
* octal 777.
*/
case '0': {
if (i+1 == oldstr.length()) {
/* found \0 at end of string */
newstr.append(Character.toChars(0));
break; /* switch */
}
i++;
int digits = 0;
int j;
for (j = 0; j <= 2; j++) {
if (i+j == oldstr.length()) {
break; /* for */
}
/* safe because will unread surrogate */
int ch = oldstr.charAt(i+j);
if (ch < '0' || ch > '7') {
break; /* for */
}
digits++;
}
if (digits == 0) {
--i;
newstr.append('\0');
break; /* switch */
}
int value = 0;
try {
value = Integer.parseInt(
oldstr.substring(i, i+digits), 8);
} catch (NumberFormatException nfe) {
die("invalid octal value for \\0 escape");
}
newstr.append(Character.toChars(value));
i += digits-1;
break; /* switch */
} /* end case '0' */
case 'x': {
if (i+2 > oldstr.length()) {
die("string too short for \\x escape");
}
i++;
boolean saw_brace = false;
if (oldstr.charAt(i) == '{') {
/* ^^^^^^ ok to ignore surrogates here */
i++;
saw_brace = true;
}
int j;
for (j = 0; j < 8; j++) {
if (!saw_brace && j == 2) {
break; /* for */
}
/*
* ASCII test also catches surrogates
*/
int ch = oldstr.charAt(i+j);
if (ch > 127) {
die("illegal non-ASCII hex digit in \\x escape");
}
if (saw_brace && ch == '}') { break; /* for */ }
if (! ( (ch >= '0' && ch <= '9')
||
(ch >= 'a' && ch <= 'f')
||
(ch >= 'A' && ch <= 'F')
)
)
{
die(String.format(
"illegal hex digit #%d '%c' in \\x", ch, ch));
}
}
if (j == 0) { die("empty braces in \\x{} escape"); }
int value = 0;
try {
value = Integer.parseInt(oldstr.substring(i, i+j), 16);
} catch (NumberFormatException nfe) {
die("invalid hex value for \\x escape");
}
newstr.append(Character.toChars(value));
if (saw_brace) { j++; }
i += j-1;
break; /* switch */
}
case 'u': {
if (i+4 > oldstr.length()) {
die("string too short for \\u escape");
}
i++;
int j;
for (j = 0; j < 4; j++) {
/* this also handles the surrogate issue */
if (oldstr.charAt(i+j) > 127) {
die("illegal non-ASCII hex digit in \\u escape");
}
}
int value = 0;
try {
value = Integer.parseInt( oldstr.substring(i, i+j), 16);
} catch (NumberFormatException nfe) {
die("invalid hex value for \\u escape");
}
newstr.append(Character.toChars(value));
i += j-1;
break; /* switch */
}
case 'U': {
if (i+8 > oldstr.length()) {
die("string too short for \\U escape");
}
i++;
int j;
for (j = 0; j < 8; j++) {
/* this also handles the surrogate issue */
if (oldstr.charAt(i+j) > 127) {
die("illegal non-ASCII hex digit in \\U escape");
}
}
int value = 0;
try {
value = Integer.parseInt(oldstr.substring(i, i+j), 16);
} catch (NumberFormatException nfe) {
die("invalid hex value for \\U escape");
}
newstr.append(Character.toChars(value));
i += j-1;
break; /* switch */
}
default: newstr.append('\\');
newstr.append(Character.toChars(cp));
/*
* say(String.format(
* "DEFAULT unrecognized escape %c passed through",
* cp));
*/
break; /* switch */
}
saw_backslash = false;
}
/* weird to leave one at the end */
if (saw_backslash) {
newstr.append('\\');
}
return newstr.toString();
}
/*
* Return a string "U+XX.XXX.XXXX" etc, where each XX set is the
* xdigits of the logical Unicode code point. No bloody brain-damaged
* UTF-16 surrogate crap, just true logical characters.
*/
public final static
String uniplus(String s) {
if (s.length() == 0) {
return "";
}
/* This is just the minimum; sb will grow as needed. */
StringBuffer sb = new StringBuffer(2 + 3 * s.length());
sb.append("U+");
for (int i = 0; i < s.length(); i++) {
sb.append(String.format("%X", s.codePointAt(i)));
if (s.codePointAt(i) > Character.MAX_VALUE) {
i++; /****WE HATES UTF-16! WE HATES IT FOREVERSES!!!****/
}
if (i+1 < s.length()) {
sb.append(".");
}
}
return sb.toString();
}
private static final
void die(String foa) {
throw new IllegalArgumentException(foa);
}
private static final
void say(String what) {
System.out.println(what);
}
If it helps others, you’re welcome to it — no strings attached. If you improve it, I’d love for you to mail me your enhancements, but you certainly don’t have to.
How to automatically unescape the escape characters in a string
I think Apache Commons work pretty good. It has StringEscapeUtils class with bunch of different static methods for escaping and unescaping strings, so i think you should check it.
Good luck!
Escaping characters in a string
StringEscapeUtils in the apache commons library has many methods that will escape, or unesecape a string as required.
StringEscapeUtils.escapeJava(someString);
StringEscapeUtils.unescapeJava(someOtherString);
In Java, is there a way to write a string literal without having to escape quotes?
The answer is no, and the proof resides in the Java Language Specification:
StringLiteral:
"StringCharacters"
StringCharacters:
StringCharacter
| StringCharacters StringCharacter
StringCharacter:
InputCharacter but not " or \
| EscapeSequence
As you can see a StringLiteral can just be bound by " and cannot contain special character without escapes..
A side note: you can embed Groovy inside your project, this will extend the syntax of Java allowing you to use '''multi line string ''', ' "string with single quotes" ' and also "string with ${variable}".
how to decode Java strings with Unicode escapes etc. from Scala JavaTokenParsers into unescaped strings?
It seems that there is no such function—at least, none is used in the Scala compiler. That's not a conclusive answer though, maybe a library function was introduced afterwards.
In case you want to read (or copy-n-paste) this code, here's the related code I found.
The tokenization logic of the Scala compiler is distributed among different files.
The top level method seems to be fetchToken in src/compiler/scala/tools/nsc/ast/parser/Scanners.scala, which in turn delegates to logic in src/compiler/scala/tools/nsc/util/CharArrayReader.scala (one of its ancestors), in particular nextChar and potentialUnicode. Other escapes are handled in getLitChar, again in Scanners.scala.
Dynamic regex based string replacement in Java
It is pretty simple using Matcher.appendReplacement() and Matcher.appendTrail():
// Compile the pattern
Pattern p = Pattern.compile("\\\\u([0-9a-f]{4})");
// Create a matcher for our input
Matcher m = p.matcher(testString);
// Create a buffer to hold the resulting string
StringBuffer result = new StringBuffer();
// Iterate over matches
while(m.find()) {
// Convert to UTF8 characters
int codePoint = Integer.parseInt(m.group(1), 16);
char[] chars = Character.toChars(codePoint);
// Append to buffer
m.appendReplacement(result, new String(chars));
}
// Append rest of string
m.appendTail(result);
// Display result
System.out.println(result);
You can test it here.
Related Topics
Resolving Dependency Problems in Apache Spark
How to Lock a File Using Java (If Possible)
Difference Between Javac and the Eclipse Compiler
Why Do I Get an Unsupportedoperationexception When Trying to Remove an Element from a List
What Are All the Escape Characters
What Are Java Command Line Options to Set to Allow Jvm to Be Remotely Debugged
Do Subclasses Inherit Private Fields
How to Use Scanner to Accept Only Valid Int as Input
Causes of Getting a Java.Lang.Verifyerror
What Are the Rules For Evaluation Order in Java
Is There a Concise Way to Iterate Over a Stream With Indices in Java 8
How to Use Wait and Notify in Java Without Illegalmonitorstateexception
Are Getters and Setters Poor Design? Contradictory Advice Seen