4 ms·

Maybe browsers should default to UTF-8 already. It's 2021.

by dheera 5y ago

lxgr 5y ago
Why? Defaulting to UTF-8 for modern HTML, and to ISO-8859-1 for legacy pages, makes a lot of sense. Pages that haven‘t been adapted to HTML 5 in the last 10 years or so are exceedingly unlikely to do so in year 11.
- dheera 5y ago
  ISO-8859-1 is a subset of UTF-8 isn't it? No harm done by defaulting to the superset.
  - CodesInChaos 5y ago
    No. ASCII is a subset of UTF-8, ISO-8859-1 is not. The first 256 codepoints of unicode match ISO-8859-1, which is probably the source of your confusion. However Codepoints 128-255 are encoded differently in UTF-8. They are represented by a single byte when encoded as ISO-8859-1, while they turn into two bytes encoded in UTF-8. Plus "ISO-8859-1" is treated as Windows-1252 by browsers, while unicode uses ISO-8859-1 extended with the ISO 6429 control characters for its initial 256 codepoints.
    - dheera 5y ago
      Ah I see, thanks.
  - anyfoo 5y ago
    If it were, the characters in question would already display correctly for this website, since they are within ISO-8859-1. ASCII is a subset of UTF-8.
  - deleted 5y ago
    [deleted]
magicalhippo 5y ago
We need to handle a lot of crappy data-in-text-files at work, and for most of them using the UTF-8 duck test seems to be the most reliable. If it decodes successfully as UTF-8 it's probably UTF-8.
- wolfgang42 5y ago
  That requires scanning the whole file before guessing the encoding, which browsers don’t do for performance reasons (and also because an HTML document may never end, it’s perfectly valid for the server to keep appending to the document indefinitely). The HTML5 spec does recommend doing this on the first 1024 bytes, though.
  - magicalhippo 5y ago
    Browsers are quite happy on re-rendering the whole document multiple times though, so it could just switch and re-decode when UTF-8 fails. Sure it wouldn't be the fast path, but sure beats looking at hieroglyphs. And yeah, add some sensible limits to this logic of course. Most web pages aren't never-ending nor multi-GB of text.
    - colejohnson66 5y ago
      The “sensible limit” is 1024. That’s what the standard is saying.
      - magicalhippo 5y ago
        Not really. I can write a lot of Norwegian text, especially with HTML overhead of scripts, inline CSS and whatnot, before needing a non-ASCII Norwegian character. Also the only reference I find to this 1024 limit is for the charset meta element[1] in HTML5, which also seems highly redundant given that the very same page states UTF-8 must be used regardless. Limits are good, but 1k is way to low. [1]: https://html.spec.whatwg.org/multipage/semantics.html#charset https://html.spec.whatwg.org/multipage/semantics.html#charse...
hsivonen 5y ago
https://hsivonen.fi/utf-8-detection/ https://hsivonen.fi/utf-8-detection/ explains why Firefox doesn’t.

Maybe browsers should default to UTF-8 already. It's 2021.

by dheera 5y ago

Maybe browsers should default to UTF-8 already. It's 2021.

lxgr 5y ago

Why? Defaulting to UTF-8 for modern HTML, and to ISO-8859-1 for legacy pages, makes a lot of sense. Pages that haven‘t been adapted to HTML 5 in the last 10 years or so are exceedingly unlikely to do so in year 11.

dheera 5y ago
ISO-8859-1 is a subset of UTF-8 isn't it? No harm done by defaulting to the superset.
- CodesInChaos 5y ago
  No. ASCII is a subset of UTF-8, ISO-8859-1 is not. The first 256 codepoints of unicode match ISO-8859-1, which is probably the source of your confusion. However Codepoints 128-255 are encoded differently in UTF-8. They are represented by a single byte when encoded as ISO-8859-1, while they turn into two bytes encoded in UTF-8. Plus "ISO-8859-1" is treated as Windows-1252 by browsers, while unicode uses ISO-8859-1 extended with the ISO 6429 control characters for its initial 256 codepoints.
  - dheera 5y ago
    Ah I see, thanks.
- anyfoo 5y ago
  If it were, the characters in question would already display correctly for this website, since they are within ISO-8859-1. ASCII is a subset of UTF-8.
- deleted 5y ago
  [deleted]

magicalhippo 5y ago

We need to handle a lot of crappy data-in-text-files at work, and for most of them using the UTF-8 duck test seems to be the most reliable. If it decodes successfully as UTF-8 it's probably UTF-8.

wolfgang42 5y ago
That requires scanning the whole file before guessing the encoding, which browsers don’t do for performance reasons (and also because an HTML document may never end, it’s perfectly valid for the server to keep appending to the document indefinitely). The HTML5 spec does recommend doing this on the first 1024 bytes, though.
- magicalhippo 5y ago
  Browsers are quite happy on re-rendering the whole document multiple times though, so it could just switch and re-decode when UTF-8 fails. Sure it wouldn't be the fast path, but sure beats looking at hieroglyphs. And yeah, add some sensible limits to this logic of course. Most web pages aren't never-ending nor multi-GB of text.
  - colejohnson66 5y ago
    The “sensible limit” is 1024. That’s what the standard is saying.
    - magicalhippo 5y ago
      Not really. I can write a lot of Norwegian text, especially with HTML overhead of scripts, inline CSS and whatnot, before needing a non-ASCII Norwegian character. Also the only reference I find to this 1024 limit is for the charset meta element[1] in HTML5, which also seems highly redundant given that the very same page states UTF-8 must be used regardless. Limits are good, but 1k is way to low. [1]: https://html.spec.whatwg.org/multipage/semantics.html#charset https://html.spec.whatwg.org/multipage/semantics.html#charse...

hsivonen 5y ago

https://hsivonen.fi/utf-8-detection/ https://hsivonen.fi/utf-8-detection/ explains why Firefox doesn’t.