Json Encoding Wrongly Escaped (Rails 3, Ruby 1.9.2)

JSON encoding wrongly escaped (Rails 3, Ruby 1.9.2)

If you dig through the source you'll eventually come to ActiveSupport::JSON::Encoding and the escape method:

def escape(string)
  if string.respond_to?(:force_encoding)
    string = string.encode(::Encoding::UTF_8, :undef => :replace).force_encoding(::Encoding::BINARY)
  end
  json = string.
    gsub(escape_regex) { |s| ESCAPED_CHARS[s] }.
    gsub(/([\xC0-\xDF][\x80-\xBF]|
           [\xE0-\xEF][\x80-\xBF]{2}|
           [\xF0-\xF7][\x80-\xBF]{3})+/nx) { |s|
    s.unpack("U*").pack("n*").unpack("H*")[0].gsub(/.{4}/n, '\\\\u\&')
  }
  json = %("#{json}")
  json.force_encoding(::Encoding::UTF_8) if json.respond_to?(:force_encoding)
  json
end

The various gsub calls are forcing non-ASCII UTF-8 to the \uXXXX notation that you're seeing. Hex encoded UTF-8 should be acceptable to anything that processes JSON but you could always post-process the JSON (or monkey patch in a modified JSON escaper) to convert the \uXXXX notation to raw UTF-8 if necessary.

I'd agree that forcing JSON to be 7bit-clean is a bit bogus but there you go.

Short answer: no.

Rails: Wrong encoding in format.json

Your render call is rendering your JSON document explicitly as text, which is likely the source of your problem. Try this instead:

format.json { render json: @articles.as_json }

This will go through the normal Rails JSON encoding path, which is very much intended to properly handle UTF-8.

P.S. For your code examples, those don't look like the output of puts calls but instead the output of IRB's representation of values -- in which case, the escaping of UTF-8 characters is happening simply so that IRB can render you a pastable-back-into-IRB version of your string.

How do you deal with the conflict between ActiveSupport::JSON and the JSON gem?

Update This fix is only applicable to Rails < 2.3. As Giles mentions below, they fixed this in 2.3 internally using much the same technique. But beware the json gem's earlier attempt at Rails compatibility (json/add/rails), which, if required explicitly will break everything all over again.

Do you mean the require 'json' statement itself raises that Exception? Or do you mean when you call @something.to_json(:something => value) you get the error? The latter is what I would expect, if you have a problem requiring the JSON gem then I'm not sure what's going on.

I just ran into this problem with the oauth gem. In my case, there is not a true conflict, because the oauth gem doesn't depend on to_json implementation. Therefore the problem is that JSON is clobbering the ActiveSupport declarations. I solved this by simply requiring json before ActiveSupport is loaded. Putting

require 'json'

inside the Rails::Initializer did the trick (though putting it after the block did NOT).

That allows ActiveSupport to clobber the default JSON implementation instead.

Now if you are using a gem that actually depends on the JSON implementation of to_json then you are up a creek. This is definitely the worst of meta-programming, and I would advocate for the Rails and JSON gem developers to resolve the conflict, though it will be painful because one or the other will have to break backwards compatibility.

In the short term, gem authors may be able to bridge the gap by supporting both implementations. This is more or less feasible depending on how the gem uses the method. A worst case scenario is an official fork (ie. gem and gem-rails).

Rails how to render JSON as UTF-8?

I am pretty sure that you could fix this with your database but the quick fix may be:

new_kategoris = @kategoris.map {|v| v.force_encoding('UTF-8') }
format.json { render :json => new_kategoris.map(&:attributes) }

Prevent Rails from encoding the ampersands in a URL when outputting JSON

Add to your application.rb file:

config.active_support.escape_html_entities_in_json = false

incompatible character encodings: ASCII-8BIT and UTF-8

I have a suspicion that you either copy/pasted a part of your Haml template into the file, or you're working with a non-Unicode/non-UTF-8 friendly editor.

See if you can recreate that file from the scratch in a UTF-8 friendly editor. There are plenty for any platform and see whether this fixes your problem. Start by erasing the line with #content and retyping it manually.

how to convert the old emoji encoding to the latest encoding in iOS5?

iOS 5 and OS X 10.7 (Lion) use the Unicode 6.0 standard ‘unified’ code points for emoji.

iOS 4 on SoftBank iPhones used a set of unofficial code points in the Unicode Private Use Area, and so aren't compatible with any other systems. To convert from this format to proper Unicode 6.0 characters, you'll need to run a big lookup table from Softbank code to Unified over all your current data and all new form data as it gets submitted. You might also want to do Unicode normalisation at this point, so that eg. fullwidth letters match normal ASCII letters.

See for example this table from a library that does emoji conversion tasks for PHP.

Emoji in usernames though?

---

Related Topics