Are There Different Types of Double Quotes in Utf-8 (Php, Str_Replace)

Are there different types of double quotes in utf-8 (PHP, str_replace)?

There are many characters that look like quotation marks, most of them are used infrequently. The ones that are used more often are these three:

"   U+0022 QUOTATION MARK
“   U+201C LEFT DOUBLE QUOTATION MARK
”   U+201D RIGHT DOUBLE QUOTATION MARK

Some rarer ones are FULLWIDTH QUOTATION MARK, the DITTO MARK, the DOUBLE PRIME, the DOUBLE PRIME QUOTATION MARK, and so on. The Unicode.org "confusables" tool finds 15 characters similar to ".

Why don't you copy and paste the offending character here so we can identify it? Or you could use the HEX function to get the hexadecimal encoding of the character, that's another way of identifying it.

str_replace double quote from word

<?php
function mb_str_replace($needle, $replacement, $haystack) {
   return implode($replacement, mb_split($needle, $haystack));
}
$t = "as“da”sd";
$t = mb_str_replace("”", "", $t); 
$t = mb_str_replace("“", "", $t); 
#and all the other weird quotes :)
echo $t;
?>

http://php.net/manual/en/ref.mbstring.php

http://www.regular-expressions.info/unicode.html

i would suggest to use preg_replace instead

$t = "as“da”sd";
$t = preg_replace("/[”“]/u","",$t); #just create a character class
echo $t;

this might be useful as well:
http://axonflux.com/handy-regexes-for-smart-quotes

How to replace Microsoft-encoded quotes in PHP

Considering you only want to replace a few specific and well identified characters, I would go for str_replace with an array: you obviously don't need the heavy artillery regex will bring you ;-)

And if you encounter some other special characters (damn copy-paste from Microsoft Word...), you can just add them to that array whenever is necessary / whenever they are identified.

The best answer I can give to your comment is probably this link: Convert Smart Quotes with PHP

And the associated code (quoting that page):

function convert_smart_quotes($string) 
{ 
    $search = array(chr(145), 
                    chr(146), 
                    chr(147), 
                    chr(148), 
                    chr(151)); 

    $replace = array("'", 
                     "'", 
                     '"', 
                     '"', 
                     '-'); 

    return str_replace($search, $replace, $string); 
} 

(I don't have Microsoft Word on this computer, so I can't test by myself)

I don't remember exactly what we used at work (I was not the one having to deal with that kind of input), but it was the same kind of stuff...

Convert all types of smart quotes with PHP

You need something like this (assuming UTF-8 input, and ignoring CJK (Chinese, Japanese, Korean)):

$chr_map = array(
   // Windows codepage 1252
   "\xC2\x82" => "'", // U+0082⇒U+201A single low-9 quotation mark
   "\xC2\x84" => '"', // U+0084⇒U+201E double low-9 quotation mark
   "\xC2\x8B" => "'", // U+008B⇒U+2039 single left-pointing angle quotation mark
   "\xC2\x91" => "'", // U+0091⇒U+2018 left single quotation mark
   "\xC2\x92" => "'", // U+0092⇒U+2019 right single quotation mark
   "\xC2\x93" => '"', // U+0093⇒U+201C left double quotation mark
   "\xC2\x94" => '"', // U+0094⇒U+201D right double quotation mark
   "\xC2\x9B" => "'", // U+009B⇒U+203A single right-pointing angle quotation mark

   // Regular Unicode     // U+0022 quotation mark (")
                          // U+0027 apostrophe     (')
   "\xC2\xAB"     => '"', // U+00AB left-pointing double angle quotation mark
   "\xC2\xBB"     => '"', // U+00BB right-pointing double angle quotation mark
   "\xE2\x80\x98" => "'", // U+2018 left single quotation mark
   "\xE2\x80\x99" => "'", // U+2019 right single quotation mark
   "\xE2\x80\x9A" => "'", // U+201A single low-9 quotation mark
   "\xE2\x80\x9B" => "'", // U+201B single high-reversed-9 quotation mark
   "\xE2\x80\x9C" => '"', // U+201C left double quotation mark
   "\xE2\x80\x9D" => '"', // U+201D right double quotation mark
   "\xE2\x80\x9E" => '"', // U+201E double low-9 quotation mark
   "\xE2\x80\x9F" => '"', // U+201F double high-reversed-9 quotation mark
   "\xE2\x80\xB9" => "'", // U+2039 single left-pointing angle quotation mark
   "\xE2\x80\xBA" => "'", // U+203A single right-pointing angle quotation mark
);
$chr = array_keys  ($chr_map); // but: for efficiency you should
$rpl = array_values($chr_map); // pre-calculate these two arrays
$str = str_replace($chr, $rpl, html_entity_decode($str, ENT_QUOTES, "UTF-8"));

Here comes the background:

Every Unicode character belongs to exactly one "General Category", of which the ones that can contain quote characters are the following:

• Ps "Punctuation, Open"
• Pe "Punctuation, Close"
• Pi "Punctuation, Initial quote (may behave like Ps or Pe depending on usage)"
• Pf "Punctuation, Final quote (may behave like Ps or Pe depending on usage)"
• Po "Punctuation, Other"

(these pages are handy for checking that you didn't miss anything - there is also an index of categories)

It is sometimes useful to match these categories in a Unicode-enabled regex.

Furthermore, Unicode characters have "properties", of which the one you are interested in is Quotation_Mark. Unfortunately, these are not accessible in a regex.

In Wikipedia you can find the group of characters with the Quotation_Mark property. The final reference is PropList.txt on unicode.org, but this is an ASCII textfile.

In case you need to translate CJK characters too, you only have to get their code points, decide their translation, and find their UTF-8 encoding, e.g., by looking it up in fileformat.info (e.g., for U+301E: http://www.fileformat.info/info/unicode/char/301e/index.htm).

Regarding Windows codepage 1252: Unicode defines the first 256 code points to represent exactly the same characters as ISO-8859-1, but ISO-8859-1 is often confused with Windows codepage 1252, so that all browsers render the range 0x80-0x9F, which is "empty" in ISO-8859-1 (more exactly: it contains control characters), as if it were Windows codepage 1252. The table in the Wikipedia page lists the Unicode equivalents.

Note: strtr() is often slower than str_replace(). Time it with your input and your PHP version. If it's fast enough, you can directly use a map like my $chr_map.

---

If you are not sure that your input is UTF-8 encoded, AND are willing to assume that if it's not, then it's ISO-8859-1 or Windows codepage 1252, then you can do this before anything else:

if ( !preg_match('/^\\X*$/u', $str)) {
   $str = utf8_encode($str);
}

Warning: this regex can in very rare cases fail to detect a non-UTF-8 encoding, though. E.g.: "Gruß…"/*CP-1252*/=="Gru\xDF\x85" looks like UTF-8 to this regex (U+07C5 is the N'ko digit 5). This regex can be slightly enhanced, but unfortunately it can be shown that there exists NO completely foolproof solution to the problem of encoding detection.

---

If you want to normalize the range 0x80-0x9F that stems from Windows codepage 1252 to regular Unicode codepoints, you can do this (and remove the first part of the $chr_map above):

$normalization_map = array(
   "\xC2\x80" => "\xE2\x82\xAC", // U+20AC Euro sign
   "\xC2\x82" => "\xE2\x80\x9A", // U+201A single low-9 quotation mark
   "\xC2\x83" => "\xC6\x92",     // U+0192 latin small letter f with hook
   "\xC2\x84" => "\xE2\x80\x9E", // U+201E double low-9 quotation mark
   "\xC2\x85" => "\xE2\x80\xA6", // U+2026 horizontal ellipsis
   "\xC2\x86" => "\xE2\x80\xA0", // U+2020 dagger
   "\xC2\x87" => "\xE2\x80\xA1", // U+2021 double dagger
   "\xC2\x88" => "\xCB\x86",     // U+02C6 modifier letter circumflex accent
   "\xC2\x89" => "\xE2\x80\xB0", // U+2030 per mille sign
   "\xC2\x8A" => "\xC5\xA0",     // U+0160 latin capital letter s with caron
   "\xC2\x8B" => "\xE2\x80\xB9", // U+2039 single left-pointing angle quotation mark
   "\xC2\x8C" => "\xC5\x92",     // U+0152 latin capital ligature oe
   "\xC2\x8E" => "\xC5\xBD",     // U+017D latin capital letter z with caron
   "\xC2\x91" => "\xE2\x80\x98", // U+2018 left single quotation mark
   "\xC2\x92" => "\xE2\x80\x99", // U+2019 right single quotation mark
   "\xC2\x93" => "\xE2\x80\x9C", // U+201C left double quotation mark
   "\xC2\x94" => "\xE2\x80\x9D", // U+201D right double quotation mark
   "\xC2\x95" => "\xE2\x80\xA2", // U+2022 bullet
   "\xC2\x96" => "\xE2\x80\x93", // U+2013 en dash
   "\xC2\x97" => "\xE2\x80\x94", // U+2014 em dash
   "\xC2\x98" => "\xCB\x9C",     // U+02DC small tilde
   "\xC2\x99" => "\xE2\x84\xA2", // U+2122 trade mark sign
   "\xC2\x9A" => "\xC5\xA1",     // U+0161 latin small letter s with caron
   "\xC2\x9B" => "\xE2\x80\xBA", // U+203A single right-pointing angle quotation mark
   "\xC2\x9C" => "\xC5\x93",     // U+0153 latin small ligature oe
   "\xC2\x9E" => "\xC5\xBE",     // U+017E latin small letter z with caron
   "\xC2\x9F" => "\xC5\xB8",     // U+0178 latin capital letter y with diaeresis
);
$chr = array_keys  ($normalization_map); // but: for efficiency you should
$rpl = array_values($normalization_map); // pre-calculate these two arrays
$str = str_replace($chr, $rpl, $str);

php str_replace array quotation marks

The ’ is nowhere to be seen in your replacement array, only ' is in it ( ’ vs ' not the same character). And you don't have to escape the double quotes ( " ) if you're putting it in single quotes.

try :

$txt= str_replace(array(',','.','-','\'','"', '’'),'',$txt);

if you want to keep only alphanumeric characters, maybe something regexp based is a better solution :

$txt = preg_replace('/[^A-Za-z0-9]/', '', $txt);

This code will replace anything that isn't a capital letter, a letter or a number.

php replace all forms of quotes

When you echo the contents from the database, make sure you have the charset meta tag set to utf-8.

HTML DOCUMENT

<meta http-equiv="Content-Type" content="text/html; charset=utf-8">

And in PHP before any output:

header('Content-Type: text/html; charset=utf-8');

And lastly set the MySQL transfer charset:

PDO:

$db->query("SET NAMES utf8");

Or oldschool:

mysql_query("SET NAMES utf8");

and while you're at it, change the MySQL table collation to UTF8_unicode or UTF8_general

You can even add some magic to .htaccess:

AddDefaultCharset utf-8

I've had these charset problems plenty of times, so I know where to find them :)

How to replace or encode single and right double quotation mark in JavaScript?

You need to use the escape character \ to prevent JS to interpret "\u201c" and "\u201d" as unicode characters.

Check this:

var stmt = "select “No.”";
stmt = stmt.replace('“', '\\u201c');
stmt = stmt.replace('”', '\\u201d');

console.log(stmt);

Skip Over Words/Phrases if In Quotes while doing String Replacement/Removal

This would be easy using Regular Expressions, more easier using PHP (PCRE). With PCRE you have this ability to match and skip using (*SKIP) backtracking verb. You match a double-quoted string then make engine to skip this part from overall match and type your desired pattern in second side of alternation.

"[^"\\]*(?:\\.[^"\\]*)*"(*SKIP)(*F)

Above regex matches a double quoted string (including escaped double quotation marks) and then tells engine to forget.

This would be the PHP code that implements this feature along with gathering stop words within a regex:

echo preg_replace('/"[^"\\\\]*(?:\\\\.[^"\\\\]*)*"(*SKIP)(*F)|\b(?:'
    . implode('|', array_map('preg_quote', $stopwords))
    . ')\b\h*/', '', $string);

Live demo

Can't replace a smart quote in POST request with PHP

Upon testing (and I had my doubts about it being an encoding issue; I accidentally deleted my comment about that), I was able to find out why your code is failing.

It's because your file's encoding may be set to UTF-8 without BOM.

If that is the case, change it to be with BOM (byte order mark) and it will work as expected.

Reference:

• https://en.wikipedia.org/wiki/Byte_order_mark

---

Nota:

Saving the file as ANSI encoding, did also replace the curly quote with a regular quote, so you have a choice. As ANSI, or UTF-8 with BOM.

You can use an editor such as Notepad++ for this.

• https://notepad-plus-plus.org/

From the dropdown menu, you would choose:

• Encoding, Convert to UTF-8 with BOM, then save.
• Or, Encoding, Convert to ANSI, then save.
• The choice is yours.

Important sidenote: Do not choose "Encode in...", because that will not convert your file once you save it. You must choose "Convert to".

There are other code editors out there that you can use which will give you the same result.

---

Related Topics