5 ms·

> One of the big ironies of all this is how well C has stood the test of time. C strings interoperate beautifully with UTF8 - all thats missing is some libc hel

> One of the big ironies of all this is how well C has stood the test of time. C strings interoperate beautifully with UTF8 - all thats missing is some libc helper methods to count & iterate through UTF8 codepoints. strlen / strncpy / strcmp / etc all work perfectly when dealing with UTF8. The only change is that you need to supply lengths in bytes not characters. That's exaggerating the truth, I think. Programmers can and do presume strlen's result is things like the length of the string in characters (and the typename "char" does not help), or the number of terminal columns wide a string is. E.g., I'm pretty sure the mysql client can still, to this day, not properly format a table, and IIRC, it is written in C. strcmp() does a byte-for-byte comparison of strings, which in Unicode may very well be meaningless. strncpy() has its own footguns w/ leaving the buffer unterminated. (But yes, as long as your indexes into the string make "sense" — aligned to code points that will make sense at the destination — it works fine.) If you have to write any of the helper methods to count & iterate through UTF8 codepoints, you have to watch out for the footgun that char is sometimes signed, sometimes not, so attempting to get the high bits to determine things like "is this a continuation byte?" with >> or & is either implementation defined or UB, depending on the specifics of the platform. (>> is UB on negatives, which char can be, and often is; & on a signed int doesn't make much sense, and it's implementation defined what it does in C (it operates on the bitwise representation of whatever the underlying signed integer representation is, which is implementation defined. Yes, yes, all the world is two's complement. The earlier UB is much more likely to bite you.)) unsigned char * is better for both raw binary buffers and dealing w/ buffers of UTF-8 data. (Honestly, I'm not even sure that C requires the character literal 'A' to correspond to an ASCII "A". But all the world is ASCII…)

kevin_thibedeau 8y ago
> Honestly, I'm not even sure that C requires the character literal 'A' to correspond to an ASCII "A". It doesn't. Nor do chars have to be 8-bits.
xeeeeeeeeeeenu 8y ago
>(Honestly, I'm not even sure that C requires the character literal 'A' to correspond to an ASCII "A". But all the world is ASCII…) EBCDIC is still alive and well on IBM mainframes.
uranusjr 8y ago
> unsigned char * is better for both raw binary buffers and dealing w/ buffers of UTF-8 data. Or, even better, uint8_t. Fundamental types don’t have guarantee fixed sizes, only minimum. Explicitly asking for 8 bits only can save sanity in edge cases in a long running code base where no one is sure what other people is doing anymore.
- maccard 8y ago
  > Or, even better, uint8_t. Fundamental types don’t have guarantee fixed sizes This is overly pedantic IMO. Practically speaking, a char will always be 8bits, unless you're working in embedded (even then, only some ancient DSP devices IIRC have non 8 bit Char). If you're writing code for POSIX, a chae will always be 8 bits, as will windows^. Almost evrry embedded device also works with an 8 bit Char, and anything that doesn't you probably know about in advance, and probably aren't handling Unicode text on. ^ Citation needed
  - mikeash 8y ago
    Char is also the smallest data type the language can have, so if char isn’t 8 bits then uint8_t won’t be available and your code won’t build anyway. That’s preferable to building and misbehaving, but if you got that far without realizing you’re dealing with an unusually-sized char then you’re probably doomed anyway.
  - dagenix 8y ago
    > This is overly pedantic IMO. Practically speaking, a char will always be 8bits ... [List of some cases where it isn't] If you want an 8-bit unsigned type, uint8_t is that type. If it's available, going with it isn't "overly pedantic", not going with it is irresponsible. What does "overly pedantic" even mean here? Whenever I see that phrase, it's often a case of someone who is wrong trying to shame someone who is a right for the crime of being right.

kevin_thibedeau 8y ago

> Honestly, I'm not even sure that C requires the character literal 'A' to correspond to an ASCII "A". It doesn't. Nor do chars have to be 8-bits.

xeeeeeeeeeeenu 8y ago

>(Honestly, I'm not even sure that C requires the character literal 'A' to correspond to an ASCII "A". But all the world is ASCII…) EBCDIC is still alive and well on IBM mainframes.

uranusjr 8y ago

> unsigned char * is better for both raw binary buffers and dealing w/ buffers of UTF-8 data. Or, even better, uint8_t. Fundamental types don’t have guarantee fixed sizes, only minimum. Explicitly asking for 8 bits only can save sanity in edge cases in a long running code base where no one is sure what other people is doing anymore.

maccard 8y ago
> Or, even better, uint8_t. Fundamental types don’t have guarantee fixed sizes This is overly pedantic IMO. Practically speaking, a char will always be 8bits, unless you're working in embedded (even then, only some ancient DSP devices IIRC have non 8 bit Char). If you're writing code for POSIX, a chae will always be 8 bits, as will windows^. Almost evrry embedded device also works with an 8 bit Char, and anything that doesn't you probably know about in advance, and probably aren't handling Unicode text on. ^ Citation needed
- mikeash 8y ago
  Char is also the smallest data type the language can have, so if char isn’t 8 bits then uint8_t won’t be available and your code won’t build anyway. That’s preferable to building and misbehaving, but if you got that far without realizing you’re dealing with an unusually-sized char then you’re probably doomed anyway.
- dagenix 8y ago
  > This is overly pedantic IMO. Practically speaking, a char will always be 8bits ... [List of some cases where it isn't] If you want an 8-bit unsigned type, uint8_t is that type. If it's available, going with it isn't "overly pedantic", not going with it is irresponsible. What does "overly pedantic" even mean here? Whenever I see that phrase, it's often a case of someone who is wrong trying to shame someone who is a right for the crime of being right.