String#Encode Not Fixing "Invalid Byte Sequence in Utf-8" Error

String#encode not fixing invalid byte sequence in UTF-8 error

It would seem that ruby thinks that the string encoding is already utf8, so when you do

line.encode!('UTF-8', :undef => :replace, :invalid => :replace, :replace => "")

it doesn't actually do anything because the destination encoding is the same as the current encoding (at least that's my interpretation of the code in transcode.c)

The real question here is whether your starting data is valid in some encoding that isn't utf-8 or whether this is data that is supposed to be utf-8 but has a few warts in it that you want to discard.

In the first case, the correct thing to do is tell ruby what this encoding is. You can do this when you open the file

File.open('somefile', 'r:iso-8859-1')

will open the file, interpreting its contents as iso-8859-1

You can even get ruby to transcode for you

File.open('somefile', 'r:iso-8859-1:utf-8')

will open the file as iso-8859-1, but when you read data from it the bytes will be converted to utf-8 for you.

You can also call force_encoding to tell ruby what a string's encoding is (this doesn't modify the bytes at all, it just tells ruby how to interpret them).

In the second case, where you just want to dump whatever nasty stuff has got into your utf-8, you can't just call encode! as you have because that's a no-op. In ruby 2.1 and higher, you can use String#scrub, in previous versions you can do this

line.encode!('UTF-16', :undef => :replace, :invalid => :replace, :replace => "")
line.encode!('UTF-8')

We first convert to utf-16. Since this is a different encoding, ruby will actually replace our invalid sequences. We can then convert back to utf-8. This won't lose us any extra data because utf-8 and utf-16 are just two different ways of encoding the same underlying character set.

How do I find/fix: ArgumentError: invalid byte sequence in UTF-8?

Thanks to @muistooshort 's help, I opened the file in ISO mode and then, reading line by line, convert to UTF-8.

myfile = File.open( 'thefile.txt', 'r:iso8859-1' )
  while rawline = myfile.gets
  line = rawline.force_encoding( 'utf-8' )
  # proceed...
end

Why do I get an Invalid Byte Sequence in UTF-8 error reading a text file?

Obviously your input file is not UTF-8 (or at least, not entirely). If you don't care about non-ascii characters, you can simply assume your file is ascii-8bit encoded. BTW, your separator (break_char) is not causing problems as comma is encoded the same way in UTF-8 as in ASCII.

fname = 'test.in'

# create example file and fill it with invalid UTF-8 sequence
File.open(fname, 'w') do |f|
  f.write "\xc3\x28"
end

# then try to read and parse it
s = File.open(fname) do |f| # file opened as UTF-8
#s = File.open(fname, 'r:ascii-8bit') do |f| # file opened as ascii-8bit
  f.read
end
p s.split ','

ArgumentError invalid byte sequence in UTF-8

You get these errors because the Zip gem assumes the filenames to be encoded in UTF-8 but they are actually in a different encoding.

To fix the error, you first have to find the correct encoding. Let's re-create the string from its bytes:

bytes = [111, 117, 116, 112, 117, 116, 50, 48, 50, 48, 49,
         50, 48, 55, 95, 49, 52, 49, 54, 48, 50, 47, 87,
         78, 83, 95, 85, 80, 151, 112, 131, 102, 129, 91,
         131, 94, 46, 116, 120, 116]

string = bytes.pack('c*')
#=> "output20201207_141602/WNS_UP\x97p\x83f\x81[\x83^.txt"

We can now traverse the Encoding.list and select those that return the expected result:

Encoding.list.select do |enc|
  s = string.encode('UTF-8', enc) rescue next
  s.end_with?('WNS_UP用データ.txt')
end
#=> [
#     #<Encoding:Windows-31J>,
#     #<Encoding:Shift_JIS>,
#     #<Encoding:SJIS-DoCoMo>,
#     #<Encoding:SJIS-KDDI>,
#     #<Encoding:SJIS-SoftBank>
#   ]

All of the above encodings result in the correct output.

Back to your code, you could use:

path = entry.name.encode('UTF-8', 'Windows-31J')
#=> "output20201207_141602/WNS_UP用データ.txt"

ext = File.extname(path)
#=> ".txt"

file_name = File.basename(path)
#=> "WNS_UP用データ.txt"

The Zip gem also has an option to set an explicit encoding for non-ASCII file names. You might want to give it a try by setting Zip.force_entry_names_encoding = 'Windows-31J' (haven't tried it)

Why do I get ArgumentError - invalid byte sequence in UTF-8?

This is from the CSV.open documentation:

You must provide a mode with an embedded Encoding designator unless your data is in Encoding::default_external(). CSV will check the Encoding of the underlying IO object (set by the mode you pass) to determine how to parse the data. You may provide a second Encoding to have the data transcoded as it is read just as you can with a normal call to IO::open(). For example, "rb:UTF-32BE:UTF-8" would read UTF-32BE data from the file but transcode it to UTF-8 before CSV parses it.

That applies to any method in CSV that opens a file.

Also start reading in the documentation at the part beginning with:

CSV and Character Encodings (M17n or Multilingualization)

Ruby is expecting UTF-8 but is seeing characters that don't fit. I'd suspect WIN-1252 or ISO-8859-1 or a variant.

Ruby 2.0.0 String#Match ArgumentError: invalid byte sequence in UTF-8

In Ruby 2.0 the encode method is a no-op when encoding a string to its current encoding:

Please note that conversion from an encoding enc to the same encoding enc is a no-op, i.e. the receiver is returned without any changes, and no exceptions are raised, even if there are invalid bytes.

This changed in 2.1, which also added the scrub method as an easier way to do this.

If you are unable to upgrade to 2.1, you’ll have to encode into a different encoding and back in order to remove invalid bytes, something like:

if ! s.valid_encoding?
  s = s.encode("UTF-16be", :invalid=>:replace, :replace=>"?").encode('UTF-8')
end

Rails, Heroku and invalid byte sequence in UTF-8 error

On Heroku, when your app receives the message "niño" from Redis, it is actually getting the four bytes:

 0x6e 0x69 0xf1 0x6f

which, when interpreted as ISO-8859-1 correspond to the characters n, i, ñ and o.

However, your Rails app assumes that these bytes should be interpreted as UTF-8, and at some point it tries to decode them this way. The third byte in this sequence, 0xf1 looks like this:

1 1 1 1 0 0 0 1

If you compare this to the table on the Wikipedia page, you can see this byte is the leading byte of a four byte character (it matches the pattern 11110xxx), and as such should be followed by three more continuation bytes that all match the pattern 10xxxxxx. It's not, instead the next byte is 0x6f (01101111), and so this is invalid utf-8 byte sequence and you get the error you see.

Using:

string = message.encode('utf-8', 'iso-8859-1')

(or the Iconv equivalent) tells Ruby to read message as ISO-8859-1 encoded, and then to create the equivalent string in UTF-8 encoding, which you can then use without problems. (An alternative could be to use force_encoding to tell Ruby the correct encoding of the string, but that will likely cause problems later when you try to mix UTF-8 and ISO-8859-1 strings).

In UTF-8, the string "niño" corresponds to the bytes:

0x6e 0x69 0xc3 0xb1 0x6f

Note that the first, second and last bytes are the same. The ñ character is encoded as the two bytes 0xc3 0xb1. If you write these out in binary and compare to the table in the Wikipedia again article you'll see they encode 0xf1, which is the ISO-8859-1 encoding of ñ (since the first 256 unicode codepoints match ISO-8859-1).

If you take these five bytes and treat them as being ISO-8859-1, then they correspond to the string

niño

Looking at the ISO-8859-1 codepage, 0xc3 maps to Â, and 0xb1 maps to ±.

So what's happening on your local machine is that your app is receiving the five bytes 0x6e 0x69 0xc3 0xb1 0x6f from Redis, which is the UTF-8 representation of "niño". On Heroku it's receiving the four bytes 0x6e 0x69 0xf1 0x6f, which is the ISO-8859-1 representation.

The real fix to your problem will be to make sure the strings being put into Redis are all already UTF-8 (or at least all the same encoding). I haven't used Redis, but from what I can tell from a brief Google, it doesn't concern itself with string encodings but simply gives back whatever bytes it's been given. You should look at whatever process is putting the data into Redis, and ensure that it handles the encoding properly.

---

Related Topics