What Do the Ent_Html5, Ent_Html401, ... Modifiers on HTML_Entity_Decode Do

What do the ENT_HTML5, ENT_HTML401, ... modifiers on html_entity_decode do?

I started wondering what behavior these constants have when I saw these constants at the htmlspecialchars page. The documentation was rubbish, so I started digging in the source code of PHP.

Basically, these constants affect whether certain entities are encoded or not (or decoded for html_entity_decode). The most obvious effect is whether the apostrophe (') is encoded to ' (for ENT_HTML401) or ' (for others). Similarly, it determines whether ' is decoded or not when using html_entity_decode. (' is always decoded).

All usages can be found in ext/standard/html.c and its header file. From ext/standard/html.h:

#define ENT_HTML_DOC_HTML401            0
#define ENT_HTML_DOC_XML1                       16
#define ENT_HTML_DOC_XHTML                      32
#define ENT_HTML_DOC_HTML5                      (16|32)

(replace ENT_HTML_DOC_ by ENT_ to get their PHP constant names)

I started looking for all occurrences of these constants, and can share the following on the behaviour of the ENT_* constants:

• It affects which numeric entities will be decoded or not. For example, ﷐ gets decoded to an unreadable/invalid character for ENT_HTML401, and ENT_XHTML and ENT_XML1. For ENT_HTML5 however, this is considered an invalid character and hence it stays ﷐. (C function unicode_cp_is_allowed)
• With ENT_SUBSTITUTE enabled, invalid code unit sequences for a specified character set are replaced with �. (does not depend on document type!)
• With ENT_DISALLOWED enabled, code points that are disallowed for the specified document type are replaced with �. (does not depend on charset!)
• With ENT_IGNORE, the same invalid code unit sequences from ENT_SUBSTITUTE are removed and no replacement is done (depends on choice of "document type", e.g. ENT_HTML5)
• Disallow for ENT_HTML5 (line 976)
• ENT_XHTML shares the entity map with ENT_HTML401. The only difference is that ' will be converted to an apostrophe with ENT_XHTML while ENT_HTML401 does not convert it (see this line)
• ENT_HTML401 and ENT_XHTML use exactly the same entity map (minus the difference from the previous point). ENT_HTML5 uses its own map. Others (currently ENT_XML1) have a very limited decoding map (>, &, <, ', " and their numeric equivalents). (see C function unescape_inverse_map)
• Note for the previous point: when only a few entities must be escaped (think of htmlspecialchars), all entities map will use the same one as ENT_XML1, except for ENT_HTML401. That one will not use ', but '.

That covers almost everything. I am not going to list all entity differences, instead I would like to point at https://github.com/php/php-src/tree/php-5.4.11/ext/standard/html_tables for some text files that contain the mappings for each type.

What ENT_* should I use for htmlspecialchars?

When using htmlspecialchars with ENT_COMPAT (default) or ENT_NOQUOTES, it does not matter which one you pick (see below). I saw some answers here on SO that boils down to this:

<input value="<?php echo htmlspecialchars($str, ENT_HTML5);?>" >

This is insecure. It will override the default value ENT_HTML401 | ENT_COMPAT which has as difference that HTML5 entities are used, but also that quotes are not escaped anymore! In addition, this is redundant code. The entities that have to be encoded by htmlspecialchars are the same for all ENT_HTML401, ENT_HTML5, etc.

Just use ENT_COMPAT or ENT_QUOTES instead. The latter also works when you use apostrophes for attributes (value='foo'). If you only have two arguments for htmlspecialchars, do not include the argument at all since it is the default (ENT_HTML401 is 0, remember?).

When you want to print something on the page (between tags, not attributes), it does not matter at all which one you pick as it will have equal effect. It is even sufficient to use ENT_NOQUOTES | ENT_HTML401 which equals to the numeric value 0.

See also below, about ENT_SUBTITUTE and ENT_DISALLOWED.

What ENT_* should I use for htmlentities?

If your text editor or database is so crappy that you cannot include non-US-ASCII characters (e.g. UTF-8), you can use htmlentities. Otherwise, save some bytes and use htmlspecialchars instead (see above).

Whether you need to use ENT_HTML401, ENT_HTML5 or something else depends on how your page is served. When you have a HTML5 page (<!doctype html>), use ENT_HTML5. XHTML or XML? Use the corresponding ENT_XHTML or ENT_XML1. With no doctype or plain ol' HTML4, use ENT_HTML401 (which is the default when omitted).

Should I use ENT_DISALLOWED, ENT_IGNORE or ENT_SUBSTITUTE?

By default, byte sequences that are invalid for the given character set are removed. To have a � in place of an invalid byte sequence, specify ENT_SUBSTITUTE. (note that &#FFFD; is shown for non-UTF-8 charsets). When you specify ENT_IGNORE though, these characters are not shown even if you specified ENT_SUBSTITUTE.

Invalid characters for a document type are substituted by the same replacement character (or its entity) above when ENT_DISALLOWED is specified. This happens regardless of having ENT_IGNORE set (which has nothing to do with invalid chars for doctypes).

Is htmlentities() sufficient for creating xml-safe values?

htmlentities() is not a guaranteed way to build legal XML.

Use htmlspecialchars() instead of htmlentities() if this is all you are worried about. If you have encoding mismatches between the representation of your data and the encoding of your XML document, htmlentities() may serve to work around/cover them up (it will bloat your XML size in doing so). I believe it's better to get your encodings consistent and just use htmlspecialchars().

Also, be aware that if you pump the return value of htmlspecialchars() inside XML attributes delimited with single quotes, you will need to pass the ENT_QUOTES flag as well so that any single quotes in your source string are properly encoded as well. I suggest doing this anyway, as it makes your code immune to bugs resulting from someone using single quotes for XML attributes in the future.

Edit: To clarify:

htmlentities() will convert a number of non-ANSI characters (I assume this is what you mean by UTF-8 data) to entities (which are represented with just ANSI characters). However, it cannot do so for any characters which do not have a corresponding entity, and so cannot guarantee that its return value consists only of ANSI characters. That's why I 'm suggesting to not use it.

If encoding is a possible issue, handle it explicitly (e.g. with iconv()).

Edit 2: Improved answer taking into account Josh Davis's comment belowis .

PHP File Editor - Working out some bugs

You should not call html_entity_decode().

htmlentities() is necessary to insert arbitrary content into HTML source.
However, the POST from the browser is raw text, and is not HTML encoded.

Show spanish characters in a form

You can try explicitly adding content type at the top of your file as below

<meta http-equiv="Content-Type" content="text/html; charset=utf-8" />

if it's already encoded as html then you need to decode it now..you can use html_entity_decode($string);

Your string to be echoed in the form should be á as returned from database and not á

$string = 'á'; // your string as fetched from database
echo html_entity_decode($string);// this will display á in the textarea

and before saving to database you need to

htmlentities($_POST['txtAreaName'], ENT_QUOTES, "UTF-8"); // return `á`

Converting from HTML entities to UTF-8

My understanding is, though I might be wrong, that unicode characters should be represented by their codepoint, and not by encoding individual UTF-8 bytes, which is what you have. So, Ö would be better represented using Ö or in the named form, Ö.

The ENT_XML1 flag to html_entity_decode does seem to make this work, though I'm not entirely sure what it does under the hood. If you want something more explicit:

preg_replace_callback('/&#x([A-Fa-f0-9]{2});/', function ($m) {
    return chr(hexdec($m[1]));
}, $str);

What's the easiest way to re-populate a form when the page is reloaded?

If you mean reloaded before it is submitted you will need to use JavaScript to capture and store the values in a cookie, or to the server using Ajax, and then refill the form on page load.

If you mean after it is submitted then will need to use PHP to re-populate the form with the submitted form values:

<input type="text" name="somefield" value="<?php if (isset($_POST['somefield'])) echo htmlspecialchars($_POST['somefield'], ENT_HTML5, 'utf-8'); ?>">

htmlentities 'Invalid Multibyte Sequence' error

For PHP 5.3.0 and below, the default charset for htmlentities() is ISO-8859-1. (Manual)

You are probably applying it to a UTF-8 string. Specify the character set using

htmlentities($string, (whatever), "UTF-8");

Since PHP 5.4.0, the default charset is UTF-8.

htmlentities 'Invalid Multibyte Sequence' error

For PHP 5.3.0 and below, the default charset for htmlentities() is ISO-8859-1. (Manual)

You are probably applying it to a UTF-8 string. Specify the character set using

htmlentities($string, (whatever), "UTF-8");

Since PHP 5.4.0, the default charset is UTF-8.

---

Related Topics