5 ms·

I don’t know what you mean by “all the UTF-8 codepoints can be held inside an 8bit char”. All Unicode codepoints obviously cannot be held in 8-bits. The UTF-8 e

I don’t know what you mean by “all the UTF-8 codepoints can be held inside an 8bit char”. All Unicode codepoints obviously cannot be held in 8-bits. The UTF-8 encoding matches ASCII over the first 7-bits, but that’s not relevant. You can UTF-8 encode Unicode codepoints into a bunch of 8-bit chars, but then you can encode anything you like into a bunch of 8-bit chars; a JPEG file for instance.

shakna 6y ago
> All Unicode codepoints obviously cannot be held in 8-bits. I mentioned UTF-8 specifically, because the UTF-8 encoding actually does specify this particular feature: > UTF-8 is capable of encoding all 1,112,064 valid character code points in Unicode using one to four one-byte (8-bit) code units. Code points with lower numerical values, which tend to occur more frequently, are encoded using fewer bytes. [0] [0] https://en.wikipedia.org/wiki/UTF-8 https://en.wikipedia.org/wiki/UTF-8
- stonemetal12 6y ago
  >Unicode using one to four one-byte UTF-8 is a variable length encoding, some characters need 8 bits, other characters need 32 bits per your own quote.
  - shakna 6y ago
    > one to four one-byte (8-bit) code units And it is designed so that it fits in one-byte units. That was a central goal of the encoding. So... If you have a char buffer, like we have been talking about, you can toss any valid UTF-8 sequence inside it.
    - Brian_K_White 6y ago
      ...merely, a[n] is not necessarily the nth charater. Up to you to make sense of the data in singles, pairs, fours, and before that to declare the buffer as the appropriate multiplier plus 1.
    - Xophmeister 6y ago
      It’s very rare that any data encoding doesn’t come in a multiple of 8-bits. Have you ever seen a 1.5 byte (12-bit) file? UTF-8 isn’t special in that regard. You can put literally anything into a char buffer and decode it how you like.
      - alarge 6y ago
        The issue isn't whether or not your character encoding is always a multiple of 8 bits. It is whether or not you can use standard (octet-focused) parsing functions to deal with those strings. This is what makes utf-8 "special". No byte of a utf-8 multibyte sequence will ever have a value < 127. So for most "syntactic" parsing problems, you can use standard C functions to deal with utf-8 strings - something that is not true with most other multibyte character encodings.
        Xophmeister 6y ago
        Bytes are bytes. We’re not debating whether it’s easier to write a UTF-8 decoder; I’m asserting that (almost?) any data can be represented as a sequence of bytes and UTF-8 is not special in that regard.
        account42 6y ago
        UTF-8 has an even stronger guarantee: If a byte sequence at any position in a UTF-8 string matches the byte sequence of a UTF-8 encoding of a Unicode code point then that part of the string represents that code point. This means you cannot just use standard C functions like strchr with UTF-8 strings and ASCII characters but you can alos use e.g. strstr to find UTF-8 substrings in UTF-8 strings.
    - globular-toast 6y ago
      And what good is that? All computer memory is just an array of addressable bytes. So all you've said is you can store UTF-8 strings in memory. You still can't do random access on the string (ie. s[i] will not give you the ith character).
      - account42 6y ago
        For any definition of a character that is useful to anyone except a text shaping engine, neither will s[i] with UCS-4 (and definitely not with UTF-16).
        globular-toast 6y ago
        Erm... ASCII? The original comment in this thread was asking whether this supports anything but ASCII.
  - Google234 6y ago
    You can fit 32 bits inside 4 8 bit chars...
__d 6y ago
The most valuable property of UTF-8 from a C-string point of view is that it guarantees there are no embedded NULs in a UTF-8 string. If you naively put the bytes of UTF-16 or UTF-32 encodings into a buffer, they might contain NUL (zero) byte values. Which, for C strings, means "end of string". UTF-8 makes sure this doesn't happen, which makes it compatible with existing C string functions.

shakna 6y ago

> All Unicode codepoints obviously cannot be held in 8-bits. I mentioned UTF-8 specifically, because the UTF-8 encoding actually does specify this particular feature: > UTF-8 is capable of encoding all 1,112,064 valid character code points in Unicode using one to four one-byte (8-bit) code units. Code points with lower numerical values, which tend to occur more frequently, are encoded using fewer bytes. [0] [0] https://en.wikipedia.org/wiki/UTF-8 https://en.wikipedia.org/wiki/UTF-8

stonemetal12 6y ago
>Unicode using one to four one-byte UTF-8 is a variable length encoding, some characters need 8 bits, other characters need 32 bits per your own quote.
- shakna 6y ago
  > one to four one-byte (8-bit) code units And it is designed so that it fits in one-byte units. That was a central goal of the encoding. So... If you have a char buffer, like we have been talking about, you can toss any valid UTF-8 sequence inside it.
  - Brian_K_White 6y ago
    ...merely, a[n] is not necessarily the nth charater. Up to you to make sense of the data in singles, pairs, fours, and before that to declare the buffer as the appropriate multiplier plus 1.
  - Xophmeister 6y ago
    It’s very rare that any data encoding doesn’t come in a multiple of 8-bits. Have you ever seen a 1.5 byte (12-bit) file? UTF-8 isn’t special in that regard. You can put literally anything into a char buffer and decode it how you like.
    - alarge 6y ago
      The issue isn't whether or not your character encoding is always a multiple of 8 bits. It is whether or not you can use standard (octet-focused) parsing functions to deal with those strings. This is what makes utf-8 "special". No byte of a utf-8 multibyte sequence will ever have a value < 127. So for most "syntactic" parsing problems, you can use standard C functions to deal with utf-8 strings - something that is not true with most other multibyte character encodings.
      - Xophmeister 6y ago
        Bytes are bytes. We’re not debating whether it’s easier to write a UTF-8 decoder; I’m asserting that (almost?) any data can be represented as a sequence of bytes and UTF-8 is not special in that regard.
      - account42 6y ago
        UTF-8 has an even stronger guarantee: If a byte sequence at any position in a UTF-8 string matches the byte sequence of a UTF-8 encoding of a Unicode code point then that part of the string represents that code point. This means you cannot just use standard C functions like strchr with UTF-8 strings and ASCII characters but you can alos use e.g. strstr to find UTF-8 substrings in UTF-8 strings.
  - globular-toast 6y ago
    And what good is that? All computer memory is just an array of addressable bytes. So all you've said is you can store UTF-8 strings in memory. You still can't do random access on the string (ie. s[i] will not give you the ith character).
    - account42 6y ago
      For any definition of a character that is useful to anyone except a text shaping engine, neither will s[i] with UCS-4 (and definitely not with UTF-16).
      - globular-toast 6y ago
        Erm... ASCII? The original comment in this thread was asking whether this supports anything but ASCII.
- Google234 6y ago
  You can fit 32 bits inside 4 8 bit chars...

__d 6y ago

The most valuable property of UTF-8 from a C-string point of view is that it guarantees there are no embedded NULs in a UTF-8 string. If you naively put the bytes of UTF-16 or UTF-32 encodings into a buffer, they might contain NUL (zero) byte values. Which, for C strings, means "end of string". UTF-8 makes sure this doesn't happen, which makes it compatible with existing C string functions.