4 ms·

Strings of ASCII characters only? Or can this library be used with Unicode as well? Just asking.

shakna 6y ago
Depends what you mean by Unicode. All the UTF-8 codepoints can be held inside an 8bit char, which is what this library seems to use under the covers. You might need to add a couple UTF-specific methods if you want number of graphemes rather than number of bytes, but there's nothing to stop you placing UTF8 data inside a char buffer.
- aidenn0 6y ago
  See my link in sibling comment; the library supports decoding arbitrary encodings to unicode, even those with embedded NULLs.
- Xophmeister 6y ago
  I don’t know what you mean by “all the UTF-8 codepoints can be held inside an 8bit char”. All Unicode codepoints obviously cannot be held in 8-bits. The UTF-8 encoding matches ASCII over the first 7-bits, but that’s not relevant. You can UTF-8 encode Unicode codepoints into a bunch of 8-bit chars, but then you can encode anything you like into a bunch of 8-bit chars; a JPEG file for instance.
  - shakna 6y ago
    > All Unicode codepoints obviously cannot be held in 8-bits. I mentioned UTF-8 specifically, because the UTF-8 encoding actually does specify this particular feature: > UTF-8 is capable of encoding all 1,112,064 valid character code points in Unicode using one to four one-byte (8-bit) code units. Code points with lower numerical values, which tend to occur more frequently, are encoded using fewer bytes. [0] [0] https://en.wikipedia.org/wiki/UTF-8 https://en.wikipedia.org/wiki/UTF-8
    - stonemetal12 6y ago
      >Unicode using one to four one-byte UTF-8 is a variable length encoding, some characters need 8 bits, other characters need 32 bits per your own quote.
      - shakna 6y ago
        > one to four one-byte (8-bit) code units And it is designed so that it fits in one-byte units. That was a central goal of the encoding. So... If you have a char buffer, like we have been talking about, you can toss any valid UTF-8 sequence inside it.
        Brian_K_White 6y ago
        ...merely, a[n] is not necessarily the nth charater. Up to you to make sense of the data in singles, pairs, fours, and before that to declare the buffer as the appropriate multiplier plus 1.
        Xophmeister 6y ago
        It’s very rare that any data encoding doesn’t come in a multiple of 8-bits. Have you ever seen a 1.5 byte (12-bit) file? UTF-8 isn’t special in that regard. You can put literally anything into a char buffer and decode it how you like.
        alarge 6y ago
        The issue isn't whether or not your character encoding is always a multiple of 8 bits. It is whether or not you can use standard (octet-focused) parsing functions to deal with those strings. This is what makes utf-8 "special". No byte of a utf-8 multibyte sequence will ever have a value < 127. So for most "syntactic" parsing problems, you can use standard C functions to deal with utf-8 strings - something that is not true with most other multibyte character encodings.
        Xophmeister 6y ago
  - __d 6y ago
    The most valuable property of UTF-8 from a C-string point of view is that it guarantees there are no embedded NULs in a UTF-8 string. If you naively put the bytes of UTF-16 or UTF-32 encodings into a buffer, they might contain NUL (zero) byte values. Which, for C strings, means "end of string". UTF-8 makes sure this doesn't happen, which makes it compatible with existing C string functions.
aidenn0 6y ago
https://github.com/maxim2266/str#unicode-support https://github.com/maxim2266/str#unicode-support
fjfaase 6y ago
There is some support for reading a string as an encoded in the current program locale. See: https://github.com/maxim2266/str#unicode-support https://github.com/maxim2266/str#unicode-support

Depends what you mean by Unicode. All the UTF-8 codepoints can be held inside an 8bit char, which is what this library seems to use under the covers. You might need to add a couple UTF-specific methods if you want number of graphemes rather than number of bytes, but there's nothing to stop you placing UTF8 data inside a char buffer.

aidenn0 6y ago
See my link in sibling comment; the library supports decoding arbitrary encodings to unicode, even those with embedded NULLs.
Xophmeister 6y ago
I don’t know what you mean by “all the UTF-8 codepoints can be held inside an 8bit char”. All Unicode codepoints obviously cannot be held in 8-bits. The UTF-8 encoding matches ASCII over the first 7-bits, but that’s not relevant. You can UTF-8 encode Unicode codepoints into a bunch of 8-bit chars, but then you can encode anything you like into a bunch of 8-bit chars; a JPEG file for instance.
- shakna 6y ago
  > All Unicode codepoints obviously cannot be held in 8-bits. I mentioned UTF-8 specifically, because the UTF-8 encoding actually does specify this particular feature: > UTF-8 is capable of encoding all 1,112,064 valid character code points in Unicode using one to four one-byte (8-bit) code units. Code points with lower numerical values, which tend to occur more frequently, are encoded using fewer bytes. [0] [0] https://en.wikipedia.org/wiki/UTF-8 https://en.wikipedia.org/wiki/UTF-8
  - stonemetal12 6y ago
    >Unicode using one to four one-byte UTF-8 is a variable length encoding, some characters need 8 bits, other characters need 32 bits per your own quote.
    - shakna 6y ago
      > one to four one-byte (8-bit) code units And it is designed so that it fits in one-byte units. That was a central goal of the encoding. So... If you have a char buffer, like we have been talking about, you can toss any valid UTF-8 sequence inside it.
      - Brian_K_White 6y ago
        ...merely, a[n] is not necessarily the nth charater. Up to you to make sense of the data in singles, pairs, fours, and before that to declare the buffer as the appropriate multiplier plus 1.
      - Xophmeister 6y ago
        It’s very rare that any data encoding doesn’t come in a multiple of 8-bits. Have you ever seen a 1.5 byte (12-bit) file? UTF-8 isn’t special in that regard. You can put literally anything into a char buffer and decode it how you like.
        alarge 6y ago
        The issue isn't whether or not your character encoding is always a multiple of 8 bits. It is whether or not you can use standard (octet-focused) parsing functions to deal with those strings. This is what makes utf-8 "special". No byte of a utf-8 multibyte sequence will ever have a value < 127. So for most "syntactic" parsing problems, you can use standard C functions to deal with utf-8 strings - something that is not true with most other multibyte character encodings.
        Xophmeister 6y ago
- __d 6y ago
  The most valuable property of UTF-8 from a C-string point of view is that it guarantees there are no embedded NULs in a UTF-8 string. If you naively put the bytes of UTF-16 or UTF-32 encodings into a buffer, they might contain NUL (zero) byte values. Which, for C strings, means "end of string". UTF-8 makes sure this doesn't happen, which makes it compatible with existing C string functions.

aidenn0 6y ago

https://github.com/maxim2266/str#unicode-support https://github.com/maxim2266/str#unicode-support

fjfaase 6y ago

There is some support for reading a string as an encoded in the current program locale. See: https://github.com/maxim2266/str#unicode-support https://github.com/maxim2266/str#unicode-support