PHP JSON_encode() is getting Malformed UTF-8 characters, possibly incorrectly encoded (error)
SOLVED!
The issue was in the function mb_detect_order()
, this function just don't work as I was expecting. I was thinking this was a list of full supporting encoding order by mostly used in order to speed up the detection process.
But I just found that this function return just 2 encoding:
//print_r(mb_detect_order());
Array
(
[0] => ASCII
[1] => UTF-8
)
Which is almost completly useless in my case.
MB functions can detect much more charset.
You can check them out by run mb_list_encodings()
and get the full list:
//print_r(mb_list_encodings());
Array
(
[0] => pass
[1] => auto
[2] => wchar
[3] => byte2be
[4] => byte2le
[5] => byte4be
[6] => byte4le
[7] => BASE64
[8] => UUENCODE
[9] => HTML-ENTITIES
[10] => Quoted-Printable
[11] => 7bit
[12] => 8bit
[13] => UCS-4
[14] => UCS-4BE
[15] => UCS-4LE
[16] => UCS-2
[17] => UCS-2BE
[18] => UCS-2LE
[19] => UTF-32
[20] => UTF-32BE
[21] => UTF-32LE
[22] => UTF-16
[23] => UTF-16BE
[24] => UTF-16LE
[25] => UTF-8
[26] => UTF-7
[27] => UTF7-IMAP
[28] => ASCII
[29] => EUC-JP
[30] => SJIS
[31] => eucJP-win
[32] => EUC-JP-2004
[33] => SJIS-win
[34] => SJIS-Mobile#DOCOMO
[35] => SJIS-Mobile#KDDI
[36] => SJIS-Mobile#SOFTBANK
[37] => SJIS-mac
[38] => SJIS-2004
[39] => UTF-8-Mobile#DOCOMO
[40] => UTF-8-Mobile#KDDI-A
[41] => UTF-8-Mobile#KDDI-B
[42] => UTF-8-Mobile#SOFTBANK
[43] => CP932
[44] => CP51932
[45] => JIS
[46] => ISO-2022-JP
[47] => ISO-2022-JP-MS
[48] => GB18030
[49] => Windows-1252
[50] => Windows-1254
[51] => ISO-8859-1
[52] => ISO-8859-2
[53] => ISO-8859-3
[54] => ISO-8859-4
[55] => ISO-8859-5
[56] => ISO-8859-6
[57] => ISO-8859-7
[58] => ISO-8859-8
[59] => ISO-8859-9
[60] => ISO-8859-10
[61] => ISO-8859-13
[62] => ISO-8859-14
[63] => ISO-8859-15
[64] => ISO-8859-16
[65] => EUC-CN
[66] => CP936
[67] => HZ
[68] => EUC-TW
[69] => BIG-5
[70] => CP950
[71] => EUC-KR
[72] => UHC
[73] => ISO-2022-KR
[74] => Windows-1251
[75] => CP866
[76] => KOI8-R
[77] => KOI8-U
[78] => ArmSCII-8
[79] => CP850
[80] => JIS-ms
[81] => ISO-2022-JP-2004
[82] => ISO-2022-JP-MOBILE#KDDI
[83] => CP50220
[84] => CP50220raw
[85] => CP50221
[86] => CP50222
)
I was in wrong, thinking that mb_detect_order
was just an ordered version of this list. The mb_detect_order
is just.... useless. In order to encode in UTF8 in the right way use the following code:
$my_encoding_list = [
"UTF-8",
"UTF-7",
"UTF-16",
"UTF-32",
"ISO-8859-16",
"ISO-8859-15",
"ISO-8859-10",
"ISO-8859-1",
"Windows-1254",
"Windows-1252",
"Windows-1251",
"ASCII",
//add yours preferred
];
//remove unsupported encodings
$encoding_list = array_intersect($my_encoding_list, mb_list_encodings());
//detect 'finally' the encoding
$this->encoding = mb_detect_encoding($source,$encoding_list,true);
This worked and solved my issue with bad data saved in the database.
php json_encode Malformed UTF-8 characters, possibly incorrectly encoded
MySQL and PHP need to be handling the same character set which is "UTF-8", please check whether you have implemented it by adding:
mysqli_set_charset($connection, "utf8");
after connect to the database. Another way is you can try encoding to EUC-JP or SHIFT-JIS as follow:
mb_convert_encoding("日本語", 'UTF-8', array('EUC-JP', 'SHIFT-JIS',
'AUTO'));
PHP Array to JSON with json_encode didnt work
RE: json_last_error_msg()
Malformed UTF-8 characters, possibly incorrectly encoded
It's a common issue.
function utf8convert($mixed, $key = null)
{
if (is_array($mixed)) {
foreach ($mixed as $key => $value) {
$mixed[$key] = utf8convert($value, $key); //recursive
}
} elseif (is_string($mixed)) {
$fixed = mb_convert_encoding($mixed, "UTF-8", "UTF-8");
return $fixed;
}
return $mixed;
}
It's almost like I just copied this code from something I wrote before ... lol ... this give me lot of headaches in the past. So, been there done that.
Related Topics
PHP Passing C Struct Data Throught Socket Programming
How to Join Three Table by Laravel Eloquent Model
Creating and Update Laravel Eloquent
Call to a Member Function Execute() on Boolean In
Why We Need Curlopt_Ssl_Verifypeer in Windows
Why Don't PHP Attributes Allow Functions
Two Simultaneous Ajax Requests Won't Run in Parallel
Unserialize PHP Array in JavaScript
Magento - Retrieve Products with a Specific Attribute Value
Laravel Socialite: Invalidstateexception
Split Camelcase Word into Words with PHP Preg_Match (Regular Expression)
Sum Values in Foreach Loop PHP
How to Add Text to an Image with PHP Gd Library
Create New Variables from Array Keys in PHP
Why Are "Echo" Short Tags Permanently Enabled as of PHP 5.4
Sorting Laravel Collection via Array of Id'S
In PHP, Is There a Short Way to Compare a Variable to Multiple Values