4 ms·
> You might think "but BOMs are super evil." Checking a BOM is extremely, extremely easy. Checking the BOM is not enough, you also have to handle it. Non-LE BO
by account42 5y ago
> You might think "but BOMs are super evil." Checking a BOM is extremely, extremely easy.
Checking the BOM is not enough, you also have to handle it. Non-LE BOMs (like surrogates, at least before the emojipocalypse) are rare enough that many "UTF-16" based tools simply only support UTF-16LE. They are also stupid because you need to know that you are dealing with UTF-16LE or UTF-16BE in the first place - either through heuristics or because it is specidified and then you can also guess/specify the byte order.
And like surrogates, the BOM is another wasted Unicode character that is only needed for the UTF-16 mess.
> Furthermore, you don't get to bail out of checking anything just by using UTF-8, you have to check to ensure you have _valid_ UTF-8. That's right, you gotta scan the whole bytestream anyway, so you may as well just check the 2-byte BOM at the beginning too.
You don't need validation for most UTF-8 tasks, GIGO is often more reasonable for things entered by humans.
> you definitely can't store UTF-8 in something expecting ASCII.
Unix tools would disagree.
$ echo "Hello Wörld!" | tr '!' '?'
Hello Wörld?
Many tools only care about finding substrings. UTF-8 gurantees if the byte sequence making up the encoding of a Unicode character (or sequence) appears in valid UTF-8 encoded text then it also decodes to that Unicode sequence.
> Practically all HTML is gzipped
Not when it is being parsed it isn't.
- camgunz 5y ago> Non-LE BOMs (like surrogates, at least before the emojipocalypse) are rare enough that many "UTF-16" based tools simply only support UTF-16LE. Gymnastics like this always occur when you don't know the encoding. If you didn't know you were getting UTF-8, welcome to heuristics or whatever. > And like surrogates, the BOM is another wasted Unicode character that is only needed for the UTF-16 mess. This isn't a big deal at all. > You don't need validation for most UTF-8 tasks, GIGO is often more reasonable for things entered by humans. You can't build robust systems this way. You always have to validate, and in order to do that, you gotta know what encoding you're getting. > Unix tools would disagree. Most of those use locales, or they don't expect ASCII they expect bytes. Again you can get lucky with this, but you can't build a robust system with luck. > Many tools only care about finding substrings. You have to scan through the string or save offsets (after an initial scan) either way, UTF-8 or 16. > Not when it is being parsed it isn't. Fair point!