4 ms·

This is because the page has no doctype, thus putting the browser in "quirks mode", defaulting to a charset of ISO-8859-1 (as the page does not specify one). Th

• dheera 5y ago
  Maybe browsers should default to UTF-8 already. It's 2021.
  • lxgr 5y ago
    Why? Defaulting to UTF-8 for modern HTML, and to ISO-8859-1 for legacy pages, makes a lot of sense. Pages that haven‘t been adapted to HTML 5 in the last 10 years or so are exceedingly unlikely to do so in year 11.
    • dheera 5y ago
      ISO-8859-1 is a subset of UTF-8 isn't it? No harm done by defaulting to the superset.
      • CodesInChaos 5y ago
        No. ASCII is a subset of UTF-8, ISO-8859-1 is not. The first 256 codepoints of unicode match ISO-8859-1, which is probably the source of your confusion. However Codepoints 128-255 are encoded differently in UTF-8. They are represented by a single byte when encoded as ISO-8859-1, while they turn into two bytes encoded in UTF-8. Plus "ISO-8859-1" is treated as Windows-1252 by browsers, while unicode uses ISO-8859-1 extended with the ISO 6429 control characters for its initial 256 codepoints.
        • dheera 5y ago
          Ah I see, thanks.
      • anyfoo 5y ago
        If it were, the characters in question would already display correctly for this website, since they are within ISO-8859-1. ASCII is a subset of UTF-8.
      • deleted 5y ago
        [deleted]
  • magicalhippo 5y ago
    We need to handle a lot of crappy data-in-text-files at work, and for most of them using the UTF-8 duck test seems to be the most reliable. If it decodes successfully as UTF-8 it's probably UTF-8.
    • wolfgang42 5y ago
      That requires scanning the whole file before guessing the encoding, which browsers don’t do for performance reasons (and also because an HTML document may never end, it’s perfectly valid for the server to keep appending to the document indefinitely). The HTML5 spec does recommend doing this on the first 1024 bytes, though.
      • magicalhippo 5y ago
        Browsers are quite happy on re-rendering the whole document multiple times though, so it could just switch and re-decode when UTF-8 fails. Sure it wouldn't be the fast path, but sure beats looking at hieroglyphs. And yeah, add some sensible limits to this logic of course. Most web pages aren't never-ending nor multi-GB of text.
        • colejohnson66 5y ago
          The “sensible limit” is 1024. That’s what the standard is saying.
          • magicalhippo 5y ago
            Not really. I can write a lot of Norwegian text, especially with HTML overhead of scripts, inline CSS and whatnot, before needing a non-ASCII Norwegian character. Also the only reference I find to this 1024 limit is for the charset meta element[1] in HTML5, which also seems highly redundant given that the very same page states UTF-8 must be used regardless. Limits are good, but 1k is way to low. [1]: https://html.spec.whatwg.org/multipage/semantics.html#charset https://html.spec.whatwg.org/multipage/semantics.html#charse...
  • hsivonen 5y ago
    https://hsivonen.fi/utf-8-detection/ https://hsivonen.fi/utf-8-detection/ explains why Firefox doesn’t.
• wolfgang42 5y ago
  > HTML5 defaults to UTF-8 I’m not sure this is correct, though the WHATWG docs[1] are kind of confusing. From what I can tell, it seems like HTML5 documents are required to be UTF-8, but also this is required to be explicitly declared either in the Content-Type header, a leading BOM, or a <meta> tag in the first 1024 bytes of the file. Reading this blog post[2] it sounds like there is a danger that if you don’t do this then heuristics will kick in and try to guess the charset instead; the documented algorithm for this doesn't seem to consider the doctype at all. [1]: https://html.spec.whatwg.org/dev/semantics.html#charset https://html.spec.whatwg.org/dev/semantics.html#charset [2]: https://blog.whatwg.org/the-road-to-html-5-character-encoding https://blog.whatwg.org/the-road-to-html-5-character-encodin...
• Aardwolf 5y ago
  The single quot for doesn't is an ASCII character though, why does that one become ’?
  • iudqnolq 5y ago
    Here's the heuridtic-based hypothesis of the python package ftfy >>> ftfy.fix_and_explain("’") ExplainedText( text="'", explanation=[ ('encode', 'sloppy-windows-1252'), ('decode', 'utf-8'), ('apply', 'uncurl_quotes') ] )
    • wolfgang42 5y ago
      Note that uncurl_quotes is a FTFY fix unrelated to character encoding, it’s basically just s/’/'/. (FTFY turns all of its fixes on by default, which sometimes results in it doing more than you might want it to.) You can play around with FTFY here (open the “Decoding steps” to see the explanation of what it did and why): https://www.linestarve.com/tools/mojibake/?mojibake=’ https://www.linestarve.com/tools/mojibake/?mojibake=’
      • deleted 5y ago
        [deleted]
      • iudqnolq 5y ago
        Thanks!
  • bdowling 5y ago
    It's not. It's a Unicode 'RIGHT SINGLE QUOTATION MARK' (U+2019), which in UTF-8 is encoded as 0xe2 0x80 0x99. 0xe2 is â in iso8859-1. 0x80 is not in iso8859-1, but is € in windows-1252. 0x99 is not in iso8859-1, but is ™ in windows-1252. So, the browser here appears to be defaulting to windows-1252.
• globular-toast 5y ago
  Oddly if you select "automatic" then Firefox correctly detects it as UTF-8. I would have expected quirks mode to use automatic rather than ISO-8859-1.