How to Fix Ruby Script Which Fails with Encoding Error: "\Xd8" on Us-Ascii

How can I fix ERROR: \xC3 on US-ASCII when bootstrapping chef-client?

Usually the culprit is one of the ohai plugins gathering data from the system. The quick fix is to disable the offending plugin.

rake db:migrate aborted! on US-ASCII using rake 0.9.2.2 and rails 3.0.10

After hours of googling and digging in I found the solution that (I hope) may by helpful for someone with a similar or the same problem.

I did bundle exec rake --trace db:migrate on the staging server and got the following error messages:

rake aborted!
"\xC5" on US-ASCII
/var/www/myapp/test.myapp.com/releases/20111230233802/config/application.rb:5:in `read'
/var/www/myapp/test.myapp.com/releases/20111230233802/config/application.rb:5:in `<top (required)>'
/var/www/myapp/test.myapp.com/releases/20111230233802/Rakefile:4:in `require'
/var/www/myapp/test.myapp.com/releases/20111230233802/Rakefile:4:in `<top (required)>'
/var/www/myapp/test.myapp.com/shared/bundle/ruby/1.9.1/gems/rake-0.9.2.2/lib/rake/rake_module.rb:25:in `load'
/var/www/myapp/test.myapp.com/shared/bundle/ruby/1.9.1/gems/rake-0.9.2.2/lib/rake/rake_module.rb:25:in `load_rakefile'
/var/www/myapp/test.myapp.com/shared/bundle/ruby/1.9.1/gems/rake-0.9.2.2/lib/rake/application.rb:501:in `raw_load_rakefile'
/var/www/myapp/test.myapp.com/shared/bundle/ruby/1.9.1/gems/rake-0.9.2.2/lib/rake/application.rb:82:in `block in load_rakefile'
/var/www/myapp/test.myapp.com/shared/bundle/ruby/1.9.1/gems/rake-0.9.2.2/lib/rake/application.rb:133:in `standard_exception_handling'
/var/www/myapp/test.myapp.com/shared/bundle/ruby/1.9.1/gems/rake-0.9.2.2/lib/rake/application.rb:81:in `load_rakefile'
/var/www/myapp/test.myapp.com/shared/bundle/ruby/1.9.1/gems/rake-0.9.2.2/lib/rake/application.rb:65:in `block in run'
/var/www/myapp/test.myapp.com/shared/bundle/ruby/1.9.1/gems/rake-0.9.2.2/lib/rake/application.rb:133:in `standard_exception_handling'
/var/www/myapp/test.myapp.com/shared/bundle/ruby/1.9.1/gems/rake-0.9.2.2/lib/rake/application.rb:63:in `run'
/var/www/myapp/test.myapp.com/shared/bundle/ruby/1.9.1/gems/rake-0.9.2.2/bin/rake:33:in `<top (required)>'
/var/www/myapp/test.myapp.com/shared/bundle/ruby/1.9.1/bin/rake:19:in `load'
/var/www/myapp/test.myapp.com/shared/bundle/ruby/1.9.1/bin/rake:19:in `<main>'

So I jumped in to the config/application.rb file to find out what could rise the error. Line 5 of that file loads an external config file:

require 'yaml'
APP_CONFIG = YAML.load(File.read(File.expand_path('../app_config.yml', __FILE__)))

and that external file contains UTF-8 chars, not US-ASCII. So I tried a couple of different solutions to solve that problem.

The only one which worked for me was to add an extra few lines of code on top of config/application.rb file:

if RUBY_VERSION =~ /1.9/
  Encoding.default_external = Encoding::UTF_8
  Encoding.default_internal = Encoding::UTF_8
end

just to tell rake to load external files using utf-8 encoding. After that change everything went smooth and exactly as expected. Problem solved!

PS.

I really don't know why developers of rake 0.9 have changed previous behavior of rake 0.8 which worked fine for me and probably for you as well for a long time. Maybe you have an idea why? I am very curious.

Cucumber fails with json Encoding::InvalidByteSequenceError

I've had the same issue, it looks it is because your shell's encoding

Take a look at this page https://github.com/cucumber/gherkin

Linux

export LANG=en_US.UTF-8

OS X

export LC_CTYPE=en_US.UTF-8

SyntaxError of Non-ASCII character

You should define source code encoding, add this to the top of your script:

# -*- coding: utf-8 -*-

The reason why it works differently in console and in the IDE is, likely, because of different default encodings set. You can check it by running:

import sys
print sys.getdefaultencoding()

Also see:

• Why declare unicode by string in python?
• Changing default encoding of Python?
• Correct way to define Python source code encoding

Handling non-standard American English Characters and Symbols in a CSV, using Python

You can use decode and encode:

print a
péché
Álak
óundle

print a.decode('latin9').encode('utf8'),
péché
Ãlak
óundle

I had to do the reverse...

How do I escape a Unicode string with Ruby?

In Ruby 1.8.x, String#inspect may be what you are looking for, e.g.

>> multi_byte_str = "hello\330\271!"
=> "hello\330\271!"

>> multi_byte_str.inspect
=> "\"hello\\330\\271!\""

>> puts multi_byte_str.inspect
"hello\330\271!"
=> nil

In Ruby 1.9 if you want multi-byte characters to have their component bytes escaped, you might want to say something like:

>> multi_byte_str.bytes.to_a.map(&:chr).join.inspect
=> "\"hello\\xD8\\xB9!\""

In both Ruby 1.8 and 1.9 if you are instead interested in the (escaped) unicode code points, you could do this (though it escapes printable stuff too):

>> multi_byte_str.unpack('U*').map{ |i| "\\u" + i.to_s(16).rjust(4, '0') }.join
=> "\\u0068\\u0065\\u006c\\u006c\\u006f\\u0639\\u0021"

---

Related Topics