4 ms·

>Unicode using one to four one-byte UTF-8 is a variable length encoding, some characters need 8 bits, other characters need 32 bits per your own quote.

shakna 6y ago
> one to four one-byte (8-bit) code units And it is designed so that it fits in one-byte units. That was a central goal of the encoding. So... If you have a char buffer, like we have been talking about, you can toss any valid UTF-8 sequence inside it.
- Brian_K_White 6y ago
  ...merely, a[n] is not necessarily the nth charater. Up to you to make sense of the data in singles, pairs, fours, and before that to declare the buffer as the appropriate multiplier plus 1.
- Xophmeister 6y ago
  It’s very rare that any data encoding doesn’t come in a multiple of 8-bits. Have you ever seen a 1.5 byte (12-bit) file? UTF-8 isn’t special in that regard. You can put literally anything into a char buffer and decode it how you like.
  - alarge 6y ago
    The issue isn't whether or not your character encoding is always a multiple of 8 bits. It is whether or not you can use standard (octet-focused) parsing functions to deal with those strings. This is what makes utf-8 "special". No byte of a utf-8 multibyte sequence will ever have a value < 127. So for most "syntactic" parsing problems, you can use standard C functions to deal with utf-8 strings - something that is not true with most other multibyte character encodings.
    - Xophmeister 6y ago
      Bytes are bytes. We’re not debating whether it’s easier to write a UTF-8 decoder; I’m asserting that (almost?) any data can be represented as a sequence of bytes and UTF-8 is not special in that regard.
    - account42 6y ago
      UTF-8 has an even stronger guarantee: If a byte sequence at any position in a UTF-8 string matches the byte sequence of a UTF-8 encoding of a Unicode code point then that part of the string represents that code point. This means you cannot just use standard C functions like strchr with UTF-8 strings and ASCII characters but you can alos use e.g. strstr to find UTF-8 substrings in UTF-8 strings.
- globular-toast 6y ago
  And what good is that? All computer memory is just an array of addressable bytes. So all you've said is you can store UTF-8 strings in memory. You still can't do random access on the string (ie. s[i] will not give you the ith character).
  - account42 6y ago
    For any definition of a character that is useful to anyone except a text shaping engine, neither will s[i] with UCS-4 (and definitely not with UTF-16).
    - globular-toast 6y ago
      Erm... ASCII? The original comment in this thread was asking whether this supports anything but ASCII.
Google234 6y ago
You can fit 32 bits inside 4 8 bit chars...

shakna 6y ago

> one to four one-byte (8-bit) code units And it is designed so that it fits in one-byte units. That was a central goal of the encoding. So... If you have a char buffer, like we have been talking about, you can toss any valid UTF-8 sequence inside it.

Brian_K_White 6y ago
...merely, a[n] is not necessarily the nth charater. Up to you to make sense of the data in singles, pairs, fours, and before that to declare the buffer as the appropriate multiplier plus 1.
Xophmeister 6y ago
It’s very rare that any data encoding doesn’t come in a multiple of 8-bits. Have you ever seen a 1.5 byte (12-bit) file? UTF-8 isn’t special in that regard. You can put literally anything into a char buffer and decode it how you like.
- alarge 6y ago
  The issue isn't whether or not your character encoding is always a multiple of 8 bits. It is whether or not you can use standard (octet-focused) parsing functions to deal with those strings. This is what makes utf-8 "special". No byte of a utf-8 multibyte sequence will ever have a value < 127. So for most "syntactic" parsing problems, you can use standard C functions to deal with utf-8 strings - something that is not true with most other multibyte character encodings.
  - Xophmeister 6y ago
    Bytes are bytes. We’re not debating whether it’s easier to write a UTF-8 decoder; I’m asserting that (almost?) any data can be represented as a sequence of bytes and UTF-8 is not special in that regard.
  - account42 6y ago
    UTF-8 has an even stronger guarantee: If a byte sequence at any position in a UTF-8 string matches the byte sequence of a UTF-8 encoding of a Unicode code point then that part of the string represents that code point. This means you cannot just use standard C functions like strchr with UTF-8 strings and ASCII characters but you can alos use e.g. strstr to find UTF-8 substrings in UTF-8 strings.
globular-toast 6y ago
And what good is that? All computer memory is just an array of addressable bytes. So all you've said is you can store UTF-8 strings in memory. You still can't do random access on the string (ie. s[i] will not give you the ith character).
- account42 6y ago
  For any definition of a character that is useful to anyone except a text shaping engine, neither will s[i] with UCS-4 (and definitely not with UTF-16).
  - globular-toast 6y ago
    Erm... ASCII? The original comment in this thread was asking whether this supports anything but ASCII.

Google234 6y ago

You can fit 32 bits inside 4 8 bit chars...