Encoding Problems in Rails on Ruby 1.9.1

Encoding problems in rails on ruby 1.9.1

I just had this as well so I think its worth having the correct answer.

The 2.8.1 MySql gem is not utf-8 friendly, so it sometimes will return UTF strings and lie to Rails, telling it that they are ASCII when in fact they are UTF-8. This makes things explode.

So: you can either monkey patch or get a compatible MySql gem. See: http://gnuu.org/2009/11/06/ruby19-rails-mysql-utf8/

Set UTF-8 as default for Ruby 1.9.3

To change the source encoding (i.e. the encoding your actual written source code is in), you have to use the magic comment currently:

# encoding: utf-8

It is not enough to either set the internal encoding (the encoding of the internal string representation after conversion) or the external encoding (the assumed encoding of read files). You actually have to set the magic encoding comment on top of files to set the source encoding.

In ChiliProject we have a rake task which sets the correct encoding header in all files automatically before a release.

As for encoding defaults:

• Ruby 1.8 and below didn't knew the concept of string encodings at all. Strings were more or less byte arrays.
• Ruby 1.9: default string encoding is US_ASCII everywhere.
• Ruby 2.0 and above: default string encoding is UTF-8.

Thus, if you use Ruby 2.0, you could skip the encoding comment and correctly assume UTF-8 encoding everywhere by default.

Rails + Ruby 1.9 invalid byte squence in US-ASCII

I found the solution:

The problem is:

Fetching data from any database (Mysql, Postgresql, Sqlite2 & 3), all configured to have UTF-8 as it's character set, returns the data with ASCII-8BIT in ruby 1.9.1 and rails 2.3.2.1.
(Taken from: https://rails.lighthouseapp.com/projects/8994/tickets/2476)

My attempt to use the patched mysql adapter likely failed because my database was not configured to natively use utf8, so the patched adapter failed to work properly.

The fix ended up being to use the patch file available here: http://gnuu.org/2009/11/06/ruby19-rails-mysql-utf8/

require 'mysql'

class Mysql::Result
  def encode(value, encoding = "utf-8")
    String === value ? value.force_encoding(encoding) : value
  end

  def each_utf8(&block)
    each_orig do |row|
      yield row.map {|col| encode(col) }
    end
  end
  alias each_orig each
  alias each each_utf8

  def each_hash_utf8(&block)
    each_hash_orig do |row|
      row.each {|k, v| row[k] = encode(v) }
      yield(row)
    end
  end
  alias each_hash_orig each_hash
  alias each_hash each_hash_utf8
end

(Placed in lib/mysql_utf8fix.rb and required in enviornment.rb using require 'lib/mysql_utf8fix.rb')

Has anyone successfully deployed a Rails project with Ruby 1.9.1?

Matz recently spoke at RubyFoo in London about ruby 1.9.1 adoption. Quite simply, ruby 1.9.1 is not production ready and should not be used for deployment just yet.

Ruby 1.9.2 will be production ready, but until then you should only use ruby 1.9.1 for play and testing.

Although many people out there have had successful deployments using 1.9.1, I would recommend sticking with REE 1.8.7 until 1.9.2 is out. Rails 3.0 will favor 1.9.2, but also work quite happily with 1.8.7 (it will NOT work with 1.8.6).

Method fails due to Encoding::UndefinedConversionError: U+03B1 from UTF-8 to ISO-8859-1

I fixed this thanks to Domon's suggestion. The webpage that Mechanize was interacting with was in ISO-8859-1 format (see more about how to detect that here), whereas my system was trying to read the page in UTF-8 formatting. To fix this, I entered agent.page.encoding = 'utf-8' into my scrape method's script as illustrated in Niels Kristian's answer here. (See also denis.peplin's answer there for further clarification on where to write it.) This allowed a forced conversion of the webpage into the proper formatting (UTF-8) that my system required to read it.

How to avoid Encoding::CompatibilityError with ruby 1.9 and mechanize form submit?

Well, looks like a bug in the 0.9.3 version of mechanize (probably issue 25). Using the current git version of mechanize fixes the posting issues for me.

But a mechanize 1.0 version should include the fix as well.

Mechanize on Ruby 1.9.3 encoding issue

just install the previous mechanize version:

gem install mechanize -v 2.0.1

or

gem 'mechanize', '2.0.1' and `bundle update`

I've been getting the same issue with 2.1
btw, there's an ongoing discussion on mechanize github regarding it

---

Related Topics