4 ms·

It's what Haskell's 'text' type uses internally. This stackoverflow question seems to discuss why that is the case: https://stackoverflow.com/questions/2376590

• marcosdumay 8y ago
  Just to add, there are a few use cases in what UTF-8 makes a large difference, and there is a UTF-8 version of Text on Hackage for those cases.
• spullara 8y ago
  Sadly every language developed in the 90s has this flaw. Stuck somewhere between bytes and actual UNICODE.
  • eridius 8y ago
    To be more clear, this flaw happened because way back when we thought UCS-2 was sufficient. Then UTF-16 happened and all these systems designed for UCS-2 had to be hacked up for UTF-16 instead.
    • stouset 8y ago
      To be even more clear, this happened because early versions of Unicode consisted of fewer than 2^16 code points, and this was thought to be enough. In 1996, Unicode 2.0 was published and allowed characters whose encoding required more than 16 bits. This necessitated UTF-16 which allows expansion up to 32 bits to represent characters in higher planes (and is compatible with most text that had been encoded in UCS-2).
      • chrisseaton 8y ago
        > allows expansion up to 32 bits It doesn't allow expansion up to 32 bits. How would that even be possible? Some of the bits are needed to indicate to use the second two bytes, so there's a hole in the range and it's less than 32 bits.
        • gowld 8y ago
          32 bit-representations (compared to UCS-2), not 32 bits worth of values, so it can represent all of Unicode. UCS-2 is limited to only 16-bit representation, making it _impossible_ to express Unicode values outside the basic multilingual plane (BMP).
          • stouset 8y ago
            Correct, this is what I meant. UTF-16 can use up to 32 bits to represent a larger number of characters, but cannot uniquely represent 2^32 possible characters.
        • theoh 8y ago
          "Surrogate pairs" use two code units (32 bits total) to increase the number of code points that can be represented. It seems a lot more complicated than UTF-8. I have no idea how many code points can actually be represented in theory. There are under 21 bits' worth of Unicode codepoints, though, so I'm sure the claim that UTF-16 can represent all of them isn't some kind of trick.
          • eridius 8y ago
            A surrogate pair can represent 20 bits, or 2^20 values. A single UTF-16 byte is of course 16 bits, or 2^16 values. 2^20 + 2^16 == 0x110000, which is precisely as many unicode code points as there are (0–0x10FFFF). Decoding UTF-16 (with surrogate pairs) is pretty similar to decoding UTF-8 except the state machine is simpler (as it's just 1–2 code units per codepoint, instead of 1–3). It also means that if you validate that you have a high surrogate and a low surrogate, the combination is guaranteed to be a valid codepoint (whereas with UTF-8 there are invalid encodings, e.g. any 1-byte codepoint can actually be encoded using 2 or 3 bytes instead, which the state machine must reject).
            • theoh 8y ago
              One of us should try to make this clearer on the relevant Wikipedia page.
  • gowld 8y ago
    The U in UTF-16 is for Unicode.
    • deleted 8y ago
      [deleted]