Is This the Best Way to Unescape Unicode Escape Sequences in Ruby

Is this the best way to unescape unicode escape sequences in Ruby?

Your regex, /\u(....)/, has some problems.

First of all, \u doesn't work the way you think it does, in 1.9 you'll get an error and in 1.8 it will just match a single u rather than the \u pair that you're looking for; you should use /\\u/ to find the literal \u that you want.

Secondly, your (....) group is much too permissive, that will allow any four characters through and that's not what you want. In 1.9, you want (\h{4}) (four hexadecimal digits) but in 1.8 you'd need ([\da-fA-F]{4}) as \h is a new thing.

So if you want your regex to work in both 1.8 and 1.9, you should use /\\u([\da-fA-F]{4})/. This gives you the following in 1.8 and 1.9:

>> s = 'Where is \u03bc pancakes \u03BD house? And u1123!'
=> "Where is \\u03bc pancakes \\u03BD house? And u1123!"
>> s.gsub(/\\u([\da-fA-F]{4})/) {|m| [$1].pack("H*").unpack("n*").pack("U*")}
=> "Where is μ pancakes ν house? And u1123!"

Using pack and unpack to mangle the hex number into a Unicode character is probably good enough but there may be better ways.

Ruby: unescape unicode string

Are you trying it from irb, or outputting the string with p?

String#inspect (called from irb and p str) transform unicode characters into \uxxxx format to allow the string to be printed anywhere. Also, when you type "CEO Frye \u2013 response to...", this is a escaped sequence resolved by the ruby parser. It is a unicode character in the final string.

str1 = "a\u2013b"
str1.size #=> 3
str2 = "a\\u2013b"
str2.size #=> 8
unescape_unicode(str2) == str1 #=> true

How do I escape a Unicode string with Ruby?

In Ruby 1.8.x, String#inspect may be what you are looking for, e.g.

>> multi_byte_str = "hello\330\271!"
=> "hello\330\271!"

>> multi_byte_str.inspect
=> "\"hello\\330\\271!\""

>> puts multi_byte_str.inspect
"hello\330\271!"
=> nil

In Ruby 1.9 if you want multi-byte characters to have their component bytes escaped, you might want to say something like:

>> multi_byte_str.bytes.to_a.map(&:chr).join.inspect
=> "\"hello\\xD8\\xB9!\""

In both Ruby 1.8 and 1.9 if you are instead interested in the (escaped) unicode code points, you could do this (though it escapes printable stuff too):

>> multi_byte_str.unpack('U*').map{ |i| "\\u" + i.to_s(16).rjust(4, '0') }.join
=> "\\u0068\\u0065\\u006c\\u006c\\u006f\\u0639\\u0021"

Best way to escape and unescape strings in Ruby?

Ruby 2.5 added String#undump as a complement to String#dump:

$ irb
irb(main):001:0> dumped_newline = "\n".dump
=> "\"\\n\""
irb(main):002:0> undumped_newline = dumped_newline.undump
=> "\n"

With it:

def escape(s)
  s.dump[1..-2]
end

def unescape(s)
  "\"#{s}\"".undump
end

$irb
irb(main):001:0> escape("\n \" \\")
=> "\\n \\\" \\\\"
irb(main):002:0> unescape("\\n \\\" \\\\")
=> "\n \" \\"

How do I unescape c-style escape sequences from ruby?

Okay, if you don't like eval solution, I've hacked a simple state machine in Ruby to parse simple "\n" and "\t" in strings correctly, including pre-escaping of backslash itself. Here it is:

BACKSLASH = "\\"

def unescape_c_string(s)
    state = 0
    res = ''
    s.each_char { |c|
        case state
        when 0
            case c
            when BACKSLASH then state = 1
            else res << c
            end
        when 1
            case c
            when 'n' then res << "\n"; state = 0
            when 't' then res << "\t"; state = 0
            when BACKSLASH then res << BACKSLASH; state = 0
            else res << BACKSLASH; res << c; state = 0
            end
        end
    }
    return res
end

This one can be easily extended to support more characters, including multi-character entities, like \123. Test unit to prove that it works:

require 'test/unit'

class TestEscapeCString < Test::Unit::TestCase
    def test_1
        assert_equal("abc\nasd", unescape_c_string('abc\nasd'))
    end
    def test_2
        assert_equal("abc\tasd", unescape_c_string('abc\tasd'))
    end
    def test_3
        assert_equal("abc\\asd", unescape_c_string('abc' + BACKSLASH * 2 + 'asd'))
    end
    def test_4
        assert_equal("abc\\nasd", unescape_c_string('abc' + BACKSLASH * 2 + 'nasd'))
    end
    def test_5
        assert_equal("abc\\\nasd", unescape_c_string('abc' + BACKSLASH * 3 + 'nasd'))
    end
    def test_6
        assert_equal("abc\\\\nasd", unescape_c_string('abc' + BACKSLASH * 4 + 'nasd'))
    end
end

Best way to escape and unescape strings in Ruby?

Ruby 2.5 added String#undump as a complement to String#dump:

$ irb
irb(main):001:0> dumped_newline = "\n".dump
=> "\"\\n\""
irb(main):002:0> undumped_newline = dumped_newline.undump
=> "\n"

With it:

def escape(s)
  s.dump[1..-2]
end

def unescape(s)
  "\"#{s}\"".undump
end

$irb
irb(main):001:0> escape("\n \" \\")
=> "\\n \\\" \\\\"
irb(main):002:0> unescape("\\n \\\" \\\\")
=> "\n \" \\"

ruby unicode escapes as command line arguments

To unescape the unicode escaped command line argument and create a new file with the user supplied unicode string in the filename, I used @mu is too short's method of using pack and unpack, like so:

filetype  = ARGV[1]
unicode   = ARGV[0].gsub(/\\u([\da-fA-F]{4})/) {|m| [$1].pack("H*").unpack("n*").pack("U*")}
path      = unicode + "." + filetype
File.new(path, "w")

Print Unicode escape codes from variable

One can not use \u along with string interpolation, since \u takes precedence. What one might do, is to Array#pack an array of integers:

▶ data.map { |e| e.to_i(16) }.pack 'U*'
#⇒ "br>

How to un-escape a backslash-escaped string?

>>> print '"Hello,\\nworld!"'.decode('string_escape')
"Hello,
world!"

---

Related Topics