Ruby 2.0.0 String#Match ArgumentError: invalid byte sequence in UTF-8
In Ruby 2.0 the encode
method is a no-op when encoding a string to its current encoding:
Please note that conversion from an encoding
enc
to the same encodingenc
is a no-op, i.e. the receiver is returned without any changes, and no exceptions are raised, even if there are invalid bytes.
This changed in 2.1, which also added the scrub
method as an easier way to do this.
If you are unable to upgrade to 2.1, you’ll have to encode into a different encoding and back in order to remove invalid bytes, something like:
if ! s.valid_encoding?
s = s.encode("UTF-16be", :invalid=>:replace, :replace=>"?").encode('UTF-8')
end
ArgumentError invalid byte sequence in UTF-8
You get these errors because the Zip gem assumes the filenames to be encoded in UTF-8 but they are actually in a different encoding.
To fix the error, you first have to find the correct encoding. Let's re-create the string from its bytes:
bytes = [111, 117, 116, 112, 117, 116, 50, 48, 50, 48, 49,
50, 48, 55, 95, 49, 52, 49, 54, 48, 50, 47, 87,
78, 83, 95, 85, 80, 151, 112, 131, 102, 129, 91,
131, 94, 46, 116, 120, 116]
string = bytes.pack('c*')
#=> "output20201207_141602/WNS_UP\x97p\x83f\x81[\x83^.txt"
We can now traverse the Encoding.list
and select
those that return the expected result:
Encoding.list.select do |enc|
s = string.encode('UTF-8', enc) rescue next
s.end_with?('WNS_UP用データ.txt')
end
#=> [
# #<Encoding:Windows-31J>,
# #<Encoding:Shift_JIS>,
# #<Encoding:SJIS-DoCoMo>,
# #<Encoding:SJIS-KDDI>,
# #<Encoding:SJIS-SoftBank>
# ]
All of the above encodings result in the correct output.
Back to your code, you could use:
path = entry.name.encode('UTF-8', 'Windows-31J')
#=> "output20201207_141602/WNS_UP用データ.txt"
ext = File.extname(path)
#=> ".txt"
file_name = File.basename(path)
#=> "WNS_UP用データ.txt"
The Zip gem also has an option to set an explicit encoding for non-ASCII file names. You might want to give it a try by setting Zip.force_entry_names_encoding = 'Windows-31J'
(haven't tried it)
Invalid byte sequence in UTF-8 (ArgumentError)
Probably your string is not in UTF-8 format, so use
if ! file_content.valid_encoding?
s = file_content.encode("UTF-16be", :invalid=>:replace, :replace=>"?").encode('UTF-8')
s.gsub(/dr/i,'med')
end
See "Ruby 2.0.0 String#Match ArgumentError: invalid byte sequence in UTF-8".
Ruby 2.1.5 - ArgumentError: invalid byte sequence in UTF-8
That doesn't look like utf-8 data so this exception is normal. Sounds like you need to tell ruby what encoding the string is actually in:
some_string.force_encoding("windows-1254")
You can then convert to UTF8 with the encode
method. There are gems (eg charlock_holmes) that have heuristics for auto detecting encodings if you're getting a mix of encodings
String#encode not fixing invalid byte sequence in UTF-8 error
It would seem that ruby thinks that the string encoding is already utf8, so when you do
line.encode!('UTF-8', :undef => :replace, :invalid => :replace, :replace => "")
it doesn't actually do anything because the destination encoding is the same as the current encoding (at least that's my interpretation of the code in transcode.c
)
The real question here is whether your starting data is valid in some encoding that isn't utf-8 or whether this is data that is supposed to be utf-8 but has a few warts in it that you want to discard.
In the first case, the correct thing to do is tell ruby what this encoding is. You can do this when you open the file
File.open('somefile', 'r:iso-8859-1')
will open the file, interpreting its contents as iso-8859-1
You can even get ruby to transcode for you
File.open('somefile', 'r:iso-8859-1:utf-8')
will open the file as iso-8859-1, but when you read data from it the bytes will be converted to utf-8 for you.
You can also call force_encoding
to tell ruby what a string's encoding is (this doesn't modify the bytes at all, it just tells ruby how to interpret them).
In the second case, where you just want to dump whatever nasty stuff has got into your utf-8, you can't just call encode!
as you have because that's a no-op. In ruby 2.1 and higher, you can use String#scrub, in previous versions you can do this
line.encode!('UTF-16', :undef => :replace, :invalid => :replace, :replace => "")
line.encode!('UTF-8')
We first convert to utf-16. Since this is a different encoding, ruby will actually replace our invalid sequences. We can then convert back to utf-8. This won't lose us any extra data because utf-8 and utf-16 are just two different ways of encoding the same underlying character set.
Related Topics
Iterate Over Days, Starting from X Date Through an End Date
How to Create a Symbol from a String That Has Whitespaces
How to Write a Rails Mixin That Spans Across Model, Controller, and View
Sending a Delete Request from Sinatra
Rails Change Routing of Submit in Form_For
Get Time Object at Start of Day in a Particular Time Zone
Devise Nomethoderror 'For' Parametersanitizer
Deleting the Current Session with Rack::Session::Cookie
Convert Hash to Openstruct Recursively
How to Stop God from Leaving Stale Resque Worker Processes
How to Disable a Form Submit Button "A Là Ruby on Rails Way"
Rails Activerecord Query Date Range
Ruby on Rails Webpacker Can't Find Images Under Asset_Pack_Path
Rails Two-Legged Oauth Provider
Refactoring Activerecord Models with a Base Class Versus a Base Module