10 ms·

Cuneicode, and the Future of Text in C

125 points by g0xA52A2A 3y ago

eesmith 3y ago
> The reason we always write out Unicode data using fwrite rather than fprintf/printf or similar is because on Microsoft Windows, the default assumption of input strings is that they have a locale encoding. ... even if I put UTF-8 data into fprintf("%s", (const char*)u8" meow ");, it can assume that the data I put in is not UTF-8 but, in fact, ISO 8859-1 or Mac Cyrillic or GBK I had no clue! I'm just a happy little Python developer for Unix, and I think I'll stay that way.
- mananaysiempre 3y ago
  > I'm just a happy little Python developer for Unix, and I think I'll stay that way. I assume you haven’t supported a twenty-year-old website entirely encoded in KOI8-R—the ability to occasionally just set LC_ALL=ru_RU.KOI8-R can be invaluable, and the same caveats would apply there. Granted, I don’t think modern Python versions will even run properly by default under such conditions. (ETA: Ah, no, I was thinking about PEP 528 and 529, and those only apply to Windows. So dunno.)
  - eesmith 3y ago
    Nope! Happy little me pretty much only needs to deal with monolingual English, and prices in £ and €. Verifying your "Ah, no" on a Mac: % python3.11 >>> print("\N{CYRILLIC CAPITAL LETTER SHORT I}") Й >>> ^D % env LC_ALL=ru_RU.KOI8-R python3.11 >>> print("\N{CYRILLIC CAPITAL LETTER SHORT I}") � >>> ^D Going to make some happy little clouds now. :)
    - mananaysiempre 3y ago
      To be entirely fair, that’s a different problem: just because you launched something with a KOI8-R locale inside your terminal emulator doesn’t mean the emulator itself stopped expecting UTF-8. (I mean, it didn’t change its UI language to Russian either, did it?) If you use luit -encoding koi8-r env LC_ALL=ru_RU.KOI8-R python3.11 instead, for example, things do work out on my Linux machine, so apparently I was wrong as far as Unix versions of Python go.
    - keithalewis 3y ago
      If English was good enough for Jesus Christ, it's good enough for me. (/s of course)
      - eesmith 3y ago
        But Jesus required more than ASCII! According to the divinely inspired translators of the 1611 King James Bible ("the Only True Bible"), John 3:16, from https://archive.org/details/1611-the-authorized-king-james-bible/page/n1321/mode/2up https://archive.org/details/1611-the-authorized-king-james-b... , is: 𝔉𝔬𝔯 𝔊𝔬𝔡 ſ𝔬 𝔩𝔬𝔳𝔢𝔡 𝔶ͤ 𝔴𝔬𝔯𝔩𝔡,𝔱𝔥𝔞𝔱 𝔥𝔢 𝔤𝔞𝔲𝔢 𝔥𝔦𝔰 𝔬𝔫𝔩𝔶 𝔟𝔢𝔤𝔬𝔱𝔱𝔢𝔫 𝔖𝔬𝔫𝔫𝔢: 𝔱𝔥𝔞𝔱 𝔴𝔥𝔬ſ𝔬𝔢𝔲𝔢𝔯 𝔟𝔢𝔩𝔢𝔢𝔲𝔢𝔱𝔥 𝔦𝔫 𝔥𝔦𝔪 , ſ𝔥𝔬𝔲𝔩𝔡 𝔫𝔬𝔱 𝔭𝔢𝔯𝔦ſ𝔥, 𝔟𝔲𝔱 𝔥𝔞𝔳𝔢 𝔢𝔲𝔢𝔯𝔩𝔞𝔰𝔱𝔦𝔫𝔤 𝔩𝔦𝔣𝔢. ASCII can't represent the yͤ shorthand for "the", nor the long-s that Jesus used. /s
- TazeTSchnitzel 3y ago
  Python has its own set of Unicode problems. Sometimes it thinks the terminal shouldn't be in Unicode, when it is, or vice-versa. Java has the same problem. But both languages' Unicode problems are so tiny compared to C that you don't need to worry too much.
rurban 3y ago
Finally a sane string encoding conversion API, but insane function names. If they get overlong, accept it and write them out. And better, write a proper string API. It's still missing.
- ape4 3y ago
  I agree about the function names. At least have official alternate long name for the function like wideToUtf8().
dingdingdang 3y ago
Thank God someone has the capacity to attack this issue in depth. UTF8 situation with regards to systems level programming is scary and annoying at same time edit: I note that HN filters out utf8 emojis..
- jagged-chisel 3y ago
  Falling back to emoticons is always an option ;-)
  - deleted 3y ago
    [deleted]
  - vanderZwan 3y ago
    Back when I was stil single some people unmatched me on dating apps for using them. Great way to filter out people not worth your time, really
    - bee_rider 3y ago
      You could do that, or you could set your age minimum to 30 or so.
      - vanderZwan 3y ago
        I'm sure it would have been worse with people in their tweens, but it's not exclusive to them weirdly enough. Another good one is being completely honest about your height (mostly just filters out American expats but still). Anyway, the only reason I used emoticons was because I use this keyboard: http://www.exideas.com/ME/index.php http://www.exideas.com/ME/index.php ... but it apparently had some side-effects.
  - serentty 3y ago
    The inability to represent text in the vast majority of the world’s languages is a far bigger issue than the lack of emoji.
- 3y ago
xeeeeeeeeeeenu 3y ago
The author is mistaken about fwrite() being exempt from locale conversions on Windows. Here's a snippet to test this: #include <stdio.h> #include <locale.h> #include <stdlib.h> #include <windows.h> #include <io.h> int main() { if (!setlocale(LC_ALL, "Japanese_Japan.932")) { puts("setlocale failed"); exit(1); } // U+00A7 SECTION SIGN encoded in CP932 char cp932_string[] = "\x81\x98\n"; // this converts the encoding: printf("%s", cp932_string); // this too: fwrite(cp932_string, 1, sizeof(cp932_string) - 1, stdout); // this too: _write(1, cp932_string, sizeof(cp932_string) - 1); // this outputs raw bytes: DWORD bytes_written; WriteFile( GetStdHandle(STD_OUTPUT_HANDLE), cp932_string, sizeof(cp932_string) - 1, &bytes_written, NULL ); return 0; } And this is the output (compiled with Visual C++ 2022 on Windows 10): >chcp Active code page: 852 >.\a.exe § § § üś
- eesmith 3y ago
  https://learn.microsoft.com/en-us/cpp/c-runtime-library/reference/fwrite?view=msvc-170 https://learn.microsoft.com/en-us/cpp/c-runtime-library/refe... says: > When stream is opened in Unicode translation mode—for example, if stream is opened by calling fopen and using a mode parameter that includes ccs=UNICODE, ccs=UTF-16LE, or ccs=UTF-8, or if the mode is changed to a Unicode translation mode by using _setmode and a mode parameter that includes _O_WTEXT, _O_U16TEXT, or _O_U8TEXT—buffer is interpreted as a pointer to an array of wchar_t that contains UTF-16 data. An attempt to write an odd number of bytes in this mode causes a parameter validation error. I presume stdout is open in that mode, but can't verify it. Try adding: _setmode(_fileno(stdout), _O_BINARY); https://learn.microsoft.com/en-us/cpp/c-runtime-library/reference/setmode?view=msvc-170 https://learn.microsoft.com/en-us/cpp/c-runtime-library/refe... But you are right, that's not mentioned in the linked-to examples. Does setlocale() have a side-effect of modifying the mode?
  - xeeeeeeeeeeenu 3y ago
    I think it isn't documented, but the console is a special case. The default mode of stdout is _O_TEXT. _write(), which is used internally by stdio, performs encoding translation if all of the following are true: - The current locale isn't "C". - The _O_TEXT flag is set. - The fd points to the console. When you redirect the stdout (e.g. with > in a shell), it's no longer considered a console handle, so the translation won't be applied. Personally, I think it was a weird design decision to include encoding and CRLF translation in what Microsoft calls "low-level I/O".
dark-star 3y ago
From the tables in the article, it looks to me that the existing(?) ztd.text already solves all issues he mentions (at least the ztd.text column has checkmarks everywhere). I have not understood why there is a need for this new ztd.cuneicode ? But I have only skimmed the article so maybe I missed something
- vasama 3y ago
  ztd.text is a C++ library. cuneicode is a C library.
  - dark-star 3y ago
    that explains it, thanks!
nemetroid 3y ago
The author is conflating terms. The meaning of execution character set is [1]: > The execution character set is the encoding used for the text of your program that is input to the compilation phase after all preprocessing steps. This character set is used for the internal representation of any string or character literals in the compiled code. But the author seems to be using "execution encoding" to mean "the current locale of the running program". These are completely different things. See the following links for the test program that's outputting "Execution Encoding": https://github.com/soasis/cuneicode/blob/018d284088fce910ac88171b63a7d8e95325a7c7/tests/basic_run_time/source/main.cpp#L43 https://github.com/soasis/cuneicode/blob/018d284088fce910ac8... https://github.com/soasis/idk/blob/ada0f119da3ccce0f1f58abc958d6f9663dbc1e3/source/ztd/idk/encoding_detection.c.cpp#L216 https://github.com/soasis/idk/blob/ada0f119da3ccce0f1f58abc9... 1: https://learn.microsoft.com/en-us/cpp/build/reference/execution-charset-set-execution-character-set?view=msvc-170#remarks https://learn.microsoft.com/en-us/cpp/build/reference/execut...
- jsmith45 3y ago
  The C standard defines the execution character set. First it defines the base execution character set. That includes: upper and lower case A to Z, digits 0 to 9, these symbols !"#%&'*+,-./:;<=>?[\]^_{|}~ , space, horizontal tab, vertical tab, form feed, alert, backspace, CR, LF. These will all have single byte encoding. The full execution character set is further defined as the basic execution characters (single byte), plus additional locale-specific characters (either single or multi-byte): > The source character set may contain multibyte characters, used to represent members of the extended character set. The execution character set may also contain multibyte characters, which need not have the same encoding as for the source character set. For both character sets, the following shall hold: > - The basic character set shall be present and each character shall be encoded as a single byte. > - The presence, meaning, and representation of any additional members is locale-specific. [ED: emphasis added] > - [allows for shift-dependent encodings] > - A byte with all bits zero shall be interpreted as a null character independent of shift state. Such a byte shall not occur as part of any other multibyte character. --- Which is to say, that the basic characters must have identical encodings in all supported char* strings, and everything else is dependent on the locale, which means setting the locale effectively modifies the execution character set. If you want to correctly interpret any string literal that exceeds the basic character set, you need to know what locale the compiler used when converting to the execution character set, and set the locale to match. Any encoding that does not have the base characters having the same byte values as are not technically valid as a string, but could be represented as an arbitrary byte array, even if you choose to spell that byte array as `char*`. This also implies that no locale may legally use such as encoding.
classified 3y ago
Not just C, the language. On Linux, if you create a `FILE *` with `fopencookie`, your program suddenly terminates if you apply `fgetwc` to it. The Linux C standard library is unable to read Unicode in 2023.
pulse65535 3y ago
Why is text such a shitshow?
- adrianN 3y ago
  Because human language is very diverse and not optimized for computers.
  - zokier 3y ago
    And the current day thinking is very conservative, especially Unicode wants to be able to capture and preserve every single little detail of writing that ever existed. Historically human languages were far more flexible, and did adapt to changes; not being able to reproduce everything exactly as hand-writing didn't stop printing press, and various languages added and removed letters from their alphabets. Try writing things in a different way these days and you will be told that stuff is misspelled or otherwise wrong just because it doesn't match what some grumpy old men decreed hundred (or two) years ago.
- ben_w 3y ago
  Like everything else, because of all the edge cases. Every symbol in Morse? Easy. Every symbol in ASCII? Easy. Add accents, and now you need to ask: are these (1) unique letters with their own sort order, or (2) modifiers that go "on top of" other letters? Answers may vary by language. Add ligatures, and now you're also forced to care about character length even if you would rather not, e.g. ﷽ (U+FDFD). Emoji? Simple ones are fine… but the general case has ways to combine characters to make different icons, making them a systematic synthlang in the same style as Chinese and Japanese[0]. What even is the right sort order of "4", "四", and "៤"? Or "a" vs. "A"? What about writing directions? LTR, RTL, the ancient Greek thing whose name I forget with alternating directions on each line, the Egyptian hieroglyphics where it can be either depending on which way the heads face, or vertical? What about alphabets like Arabic where glyphs are not in general separate, and where they can take different forms depending on if they're the first in a word, the last in a word, in the middle of a word, or isolated? What about archaic letter forms? e.g. ſ, þ, and ð in old English. [0] where I will probably embarrass myself by suggesting 犭(dog) + 瓜 (melon) = 狐 (fox), which, though cute, feels like as much a false etymology as suggesting that in English "congress" is the opposite of "progress". Or perhaps Japanese foxes really are melon dogs — I don't know, I can barely count to 4 in that specific writing system.
foul 3y ago
Didn't read the post thoroughly, will do, but why no libgrapheme in there?
- lifthrasiir 3y ago
  In my understanding this library is meant to be a part of the future C standard library (but that attempt was not successful). As such, it has to keep all past craziness imposed by ISO C---libgrapheme on the other hand only supports UTF-8 and pseudo-UTF-32 ("pseudo" because of uint_least32_t).
smcameron 3y ago
These aren't mistakes of C, these are mistakes made hundreds and hundreds of years ago in the design of various language writing systems. English gets a lot right with its writing system, mostly in its choice of a small alphabet, a small set of punctuation marks, and an almost complete absence of diacritical marks.
- mikepurvis 3y ago
  Maybe, but we also have to understand that by an accident of history, computers and the internet were originated by English-speaking people, so it makes sense that the defaults (ASCII and so on) were geared toward the specific needs of that language— it makes sense that everything not-English is going to have some degree of feeling "bolted on" to an underlying framework built for English. If French or Greek or Russian or Japanese had happened to be the lingua franca of the Internet, the features those writing systems require would be first class, and the implementation of English would be the one that felt like a bit more of a kludge. Imagine if computers were designed around unicase Arabic languages [1], and then years later there was a "Han Unificiation"-type summit where it was decided that capitalization was to be handled with extra metadata, since an upper- lower-case latin letter are really just the same thing and why bother wasting double the codepoints on that. (I don't have a link handy, but I believe I read somewhere that you can see some of this effect in NES-era RPGs from Japan— that some of the narrative had to be simplified down for the English translations because the text boxes were originally built for a small number of complex characters rather than the larger number of simpler ones that would be required to capture the same meaning.) [1]: https://en.wikipedia.org/wiki/Unicase https://en.wikipedia.org/wiki/Unicase
  - dahfizz 3y ago
    The English language evolved alongside the technology. English used to be cursive-only, like something like Arabic. But the language evolved to fit into a modern world, and I think other written languages would benefit from a simplified "print" version as well.
    - dontlaugh 3y ago
Avamander 3y ago
The first thing that comes to mind with C and locales is this legendary MPV commit: https://github.com/mpv-player/mpv/commit/1e70e82baa9193f6f027338b0fab0f5078971fbe https://github.com/mpv-player/mpv/commit/1e70e82baa9193f6f02...
- adamrezich 3y ago
  that was a fun read—you can feel the catharsis felt in writing it all out.
- tehnub 3y ago
  Come back wm4, mpv needs you
vpastore 3y ago
[dead]
kazinator 3y ago
The elephant in the room is that Unicode is a steaming pile of shit. It is the source of all the complexity that results in people picking and choosing what they dare to try implementing, and document other things as being off the plate (the application programmer can fend for themselves as best they can, based on what of the missing cruft they need).
- jakeogh 3y ago
  The lack of superscript uppercase C F Q S X Y and Z really accents U+1F4A9.
- Georgelemental 3y ago
  What would you change?
  - kps 3y ago
    Not them, but: - Combining characters, joiners, modifiers, etc. should have been prefix, not suffix/infix, and in blocks by arity. This would have allowed detecting incomplete sequences, allowed code to skip combinations without knowing their meaning, and made dead-key input trivial. - A standard normalized form should have been specified up front. - Once blocks overflowed, subsequent blocks should have been made large enough that more that one would not be necessary. Unicode has ten different Latin blocks. You've got ‘Kana Extended-B’, ‘Kana Supplement’, ‘Kana Extended-A’ (in that order!), and ‘Small Kana Extension’. (If there's an explanation for ‘Supplement’ vs ‘Extended’ vs ‘Extension’, I've never seen it.) - Character ordering is inconsistent even within scripts, e.g. ÀÁÂ… àáâ but ĀāĂăĄą. - Assigning code points for the UTF-16 encoding is ridiculous. - The contentious cases of CJK unification should have been disambiguated promptly, before use of the originals became entrenched. (I know where to find 2K code points they could have used…) - They recently changed some characters to have ‘emoji presentation’ by default, meaning that existing text needs to be edited to preserve its original appearance.
    - JdeBP 3y ago
      Dead-key input, if done according to ISO 9995, is trivial for exactly that reason. Type all of the combining diacritics, then the base character. * https://jdebp.uk/Softwares/nosh/guide/commands/console-fb-realizer.xml#DEAD-KEYS https://jdebp.uk/Softwares/nosh/guide/commands/console-fb-re...
cryptonector 3y ago
> Standard C’s primary deficiency is its constant clinging to and dependency upon the “multibyte” encoding and the “wide character” encoding. What? No, UTF-8 won for a reason, and that reason is not just that C has a deficiency in this area but, rather, that UTF-8 is: a) simple and much saner than UTF-16, b) self-synchronizing in both directions, c) as -or even more- space efficient than UTF-16 on average even for non-Latin text, d) even UTF-32 doesn't make it possible to turn logical character string indices into 32-bit word string indices. (d) is the real killer. One just cannot assume that a character or glyph requires just one codepoint to express, therefore one can't assume that a character or glyph will require some fixed number of code units to express, therefore one might as well use UTF-8 because it's saner and more efficient than the other UTFs, therefore... <drumroll/> a "constant clinging to and dependency upon the “multibyte” encoding" is NOT a deficiency of C but an advantage of C. C does have problems here though, namely all the usual problems it has: - C strings (NUL-terminated) suck - C doesn't have a first class string type, only pointer to char - C doesn't have a first class string type that indicates what encoding the string uses Now, C's wchar_t is not even a deficiency but a disaster.
- Taywee 3y ago
  I don't think that's what the complaint is. The complaint is that "multibyte" is not necessarily UTF-8. You can't just blindly convert to multibyte assuming that it's UTF-8, because it might not be. You can't convert between two encodings by just going through "multibyte", because it might actually not support all characters you might need to support. So it really is a deficiency in C. It's nearly useless to have a "multibyte" or "wide character" encoding when those can mean anything. Having conversion between UTF-8 and UTF-32 is useful. Having conversion between "implemetation and platform dependent 'multibyte'" and "implementation and platform dependent 'wide character'" strings is nearly useless.
  - cryptonector 3y ago
    > You can't just blindly convert to multibyte assuming that it's UTF-8, because it might not be. I mentioned that. TFA didn't say that specifically.

125 points by g0xA52A2A 3y ago

> The reason we always write out Unicode data using fwrite rather than fprintf/printf or similar is because on Microsoft Windows, the default assumption of input strings is that they have a locale encoding. ... even if I put UTF-8 data into fprintf("%s", (const char*)u8" meow ");, it can assume that the data I put in is not UTF-8 but, in fact, ISO 8859-1 or Mac Cyrillic or GBK I had no clue! I'm just a happy little Python developer for Unix, and I think I'll stay that way.

mananaysiempre 3y ago
> I'm just a happy little Python developer for Unix, and I think I'll stay that way. I assume you haven’t supported a twenty-year-old website entirely encoded in KOI8-R—the ability to occasionally just set LC_ALL=ru_RU.KOI8-R can be invaluable, and the same caveats would apply there. Granted, I don’t think modern Python versions will even run properly by default under such conditions. (ETA: Ah, no, I was thinking about PEP 528 and 529, and those only apply to Windows. So dunno.)
- eesmith 3y ago
  Nope! Happy little me pretty much only needs to deal with monolingual English, and prices in £ and €. Verifying your "Ah, no" on a Mac: % python3.11 >>> print("\N{CYRILLIC CAPITAL LETTER SHORT I}") Й >>> ^D % env LC_ALL=ru_RU.KOI8-R python3.11 >>> print("\N{CYRILLIC CAPITAL LETTER SHORT I}") � >>> ^D Going to make some happy little clouds now. :)
  - mananaysiempre 3y ago
    To be entirely fair, that’s a different problem: just because you launched something with a KOI8-R locale inside your terminal emulator doesn’t mean the emulator itself stopped expecting UTF-8. (I mean, it didn’t change its UI language to Russian either, did it?) If you use luit -encoding koi8-r env LC_ALL=ru_RU.KOI8-R python3.11 instead, for example, things do work out on my Linux machine, so apparently I was wrong as far as Unix versions of Python go.
  - keithalewis 3y ago
    If English was good enough for Jesus Christ, it's good enough for me. (/s of course)
    - eesmith 3y ago
      But Jesus required more than ASCII! According to the divinely inspired translators of the 1611 King James Bible ("the Only True Bible"), John 3:16, from https://archive.org/details/1611-the-authorized-king-james-bible/page/n1321/mode/2up https://archive.org/details/1611-the-authorized-king-james-b... , is: 𝔉𝔬𝔯 𝔊𝔬𝔡 ſ𝔬 𝔩𝔬𝔳𝔢𝔡 𝔶ͤ 𝔴𝔬𝔯𝔩𝔡,𝔱𝔥𝔞𝔱 𝔥𝔢 𝔤𝔞𝔲𝔢 𝔥𝔦𝔰 𝔬𝔫𝔩𝔶 𝔟𝔢𝔤𝔬𝔱𝔱𝔢𝔫 𝔖𝔬𝔫𝔫𝔢: 𝔱𝔥𝔞𝔱 𝔴𝔥𝔬ſ𝔬𝔢𝔲𝔢𝔯 𝔟𝔢𝔩𝔢𝔢𝔲𝔢𝔱𝔥 𝔦𝔫 𝔥𝔦𝔪 , ſ𝔥𝔬𝔲𝔩𝔡 𝔫𝔬𝔱 𝔭𝔢𝔯𝔦ſ𝔥, 𝔟𝔲𝔱 𝔥𝔞𝔳𝔢 𝔢𝔲𝔢𝔯𝔩𝔞𝔰𝔱𝔦𝔫𝔤 𝔩𝔦𝔣𝔢. ASCII can't represent the yͤ shorthand for "the", nor the long-s that Jesus used. /s
TazeTSchnitzel 3y ago
Python has its own set of Unicode problems. Sometimes it thinks the terminal shouldn't be in Unicode, when it is, or vice-versa. Java has the same problem. But both languages' Unicode problems are so tiny compared to C that you don't need to worry too much.

rurban 3y ago

Finally a sane string encoding conversion API, but insane function names. If they get overlong, accept it and write them out. And better, write a proper string API. It's still missing.

ape4 3y ago
I agree about the function names. At least have official alternate long name for the function like wideToUtf8().

dingdingdang 3y ago

Thank God someone has the capacity to attack this issue in depth. UTF8 situation with regards to systems level programming is scary and annoying at same time edit: I note that HN filters out utf8 emojis..

jagged-chisel 3y ago
Falling back to emoticons is always an option ;-)
- deleted 3y ago
  [deleted]
- vanderZwan 3y ago
  Back when I was stil single some people unmatched me on dating apps for using them. Great way to filter out people not worth your time, really
  - bee_rider 3y ago
    You could do that, or you could set your age minimum to 30 or so.
    - vanderZwan 3y ago
      I'm sure it would have been worse with people in their tweens, but it's not exclusive to them weirdly enough. Another good one is being completely honest about your height (mostly just filters out American expats but still). Anyway, the only reason I used emoticons was because I use this keyboard: http://www.exideas.com/ME/index.php http://www.exideas.com/ME/index.php ... but it apparently had some side-effects.
- serentty 3y ago
  The inability to represent text in the vast majority of the world’s languages is a far bigger issue than the lack of emoji.
3y ago

xeeeeeeeeeeenu 3y ago

The author is mistaken about fwrite() being exempt from locale conversions on Windows. Here's a snippet to test this: #include <stdio.h> #include <locale.h> #include <stdlib.h> #include <windows.h> #include <io.h> int main() { if (!setlocale(LC_ALL, "Japanese_Japan.932")) { puts("setlocale failed"); exit(1); } // U+00A7 SECTION SIGN encoded in CP932 char cp932_string[] = "\x81\x98\n"; // this converts the encoding: printf("%s", cp932_string); // this too: fwrite(cp932_string, 1, sizeof(cp932_string) - 1, stdout); // this too: _write(1, cp932_string, sizeof(cp932_string) - 1); // this outputs raw bytes: DWORD bytes_written; WriteFile( GetStdHandle(STD_OUTPUT_HANDLE), cp932_string, sizeof(cp932_string) - 1, &bytes_written, NULL ); return 0; } And this is the output (compiled with Visual C++ 2022 on Windows 10): >chcp Active code page: 852 >.\a.exe § § § üś

eesmith 3y ago
https://learn.microsoft.com/en-us/cpp/c-runtime-library/reference/fwrite?view=msvc-170 https://learn.microsoft.com/en-us/cpp/c-runtime-library/refe... says: > When stream is opened in Unicode translation mode—for example, if stream is opened by calling fopen and using a mode parameter that includes ccs=UNICODE, ccs=UTF-16LE, or ccs=UTF-8, or if the mode is changed to a Unicode translation mode by using _setmode and a mode parameter that includes _O_WTEXT, _O_U16TEXT, or _O_U8TEXT—buffer is interpreted as a pointer to an array of wchar_t that contains UTF-16 data. An attempt to write an odd number of bytes in this mode causes a parameter validation error. I presume stdout is open in that mode, but can't verify it. Try adding: _setmode(_fileno(stdout), _O_BINARY); https://learn.microsoft.com/en-us/cpp/c-runtime-library/reference/setmode?view=msvc-170 https://learn.microsoft.com/en-us/cpp/c-runtime-library/refe... But you are right, that's not mentioned in the linked-to examples. Does setlocale() have a side-effect of modifying the mode?
- xeeeeeeeeeeenu 3y ago
  I think it isn't documented, but the console is a special case. The default mode of stdout is _O_TEXT. _write(), which is used internally by stdio, performs encoding translation if all of the following are true: - The current locale isn't "C". - The _O_TEXT flag is set. - The fd points to the console. When you redirect the stdout (e.g. with > in a shell), it's no longer considered a console handle, so the translation won't be applied. Personally, I think it was a weird design decision to include encoding and CRLF translation in what Microsoft calls "low-level I/O".

dark-star 3y ago

From the tables in the article, it looks to me that the existing(?) ztd.text already solves all issues he mentions (at least the ztd.text column has checkmarks everywhere). I have not understood why there is a need for this new ztd.cuneicode ? But I have only skimmed the article so maybe I missed something

vasama 3y ago
ztd.text is a C++ library. cuneicode is a C library.
- dark-star 3y ago
  that explains it, thanks!

nemetroid 3y ago

The author is conflating terms. The meaning of execution character set is [1]: > The execution character set is the encoding used for the text of your program that is input to the compilation phase after all preprocessing steps. This character set is used for the internal representation of any string or character literals in the compiled code. But the author seems to be using "execution encoding" to mean "the current locale of the running program". These are completely different things. See the following links for the test program that's outputting "Execution Encoding": https://github.com/soasis/cuneicode/blob/018d284088fce910ac88171b63a7d8e95325a7c7/tests/basic_run_time/source/main.cpp#L43 https://github.com/soasis/cuneicode/blob/018d284088fce910ac8... https://github.com/soasis/idk/blob/ada0f119da3ccce0f1f58abc958d6f9663dbc1e3/source/ztd/idk/encoding_detection.c.cpp#L216 https://github.com/soasis/idk/blob/ada0f119da3ccce0f1f58abc9... 1: https://learn.microsoft.com/en-us/cpp/build/reference/execution-charset-set-execution-character-set?view=msvc-170#remarks https://learn.microsoft.com/en-us/cpp/build/reference/execut...

jsmith45 3y ago
The C standard defines the execution character set. First it defines the base execution character set. That includes: upper and lower case A to Z, digits 0 to 9, these symbols !"#%&'*+,-./:;<=>?[\]^_{|}~ , space, horizontal tab, vertical tab, form feed, alert, backspace, CR, LF. These will all have single byte encoding. The full execution character set is further defined as the basic execution characters (single byte), plus additional locale-specific characters (either single or multi-byte): > The source character set may contain multibyte characters, used to represent members of the extended character set. The execution character set may also contain multibyte characters, which need not have the same encoding as for the source character set. For both character sets, the following shall hold: > - The basic character set shall be present and each character shall be encoded as a single byte. > - The presence, meaning, and representation of any additional members is locale-specific. [ED: emphasis added] > - [allows for shift-dependent encodings] > - A byte with all bits zero shall be interpreted as a null character independent of shift state. Such a byte shall not occur as part of any other multibyte character. --- Which is to say, that the basic characters must have identical encodings in all supported char* strings, and everything else is dependent on the locale, which means setting the locale effectively modifies the execution character set. If you want to correctly interpret any string literal that exceeds the basic character set, you need to know what locale the compiler used when converting to the execution character set, and set the locale to match. Any encoding that does not have the base characters having the same byte values as are not technically valid as a string, but could be represented as an arbitrary byte array, even if you choose to spell that byte array as `char*`. This also implies that no locale may legally use such as encoding.

classified 3y ago

Not just C, the language. On Linux, if you create a `FILE *` with `fopencookie`, your program suddenly terminates if you apply `fgetwc` to it. The Linux C standard library is unable to read Unicode in 2023.

pulse65535 3y ago

Why is text such a shitshow?

adrianN 3y ago
Because human language is very diverse and not optimized for computers.
- zokier 3y ago
  And the current day thinking is very conservative, especially Unicode wants to be able to capture and preserve every single little detail of writing that ever existed. Historically human languages were far more flexible, and did adapt to changes; not being able to reproduce everything exactly as hand-writing didn't stop printing press, and various languages added and removed letters from their alphabets. Try writing things in a different way these days and you will be told that stuff is misspelled or otherwise wrong just because it doesn't match what some grumpy old men decreed hundred (or two) years ago.
ben_w 3y ago
Like everything else, because of all the edge cases. Every symbol in Morse? Easy. Every symbol in ASCII? Easy. Add accents, and now you need to ask: are these (1) unique letters with their own sort order, or (2) modifiers that go "on top of" other letters? Answers may vary by language. Add ligatures, and now you're also forced to care about character length even if you would rather not, e.g. ﷽ (U+FDFD). Emoji? Simple ones are fine… but the general case has ways to combine characters to make different icons, making them a systematic synthlang in the same style as Chinese and Japanese[0]. What even is the right sort order of "4", "四", and "៤"? Or "a" vs. "A"? What about writing directions? LTR, RTL, the ancient Greek thing whose name I forget with alternating directions on each line, the Egyptian hieroglyphics where it can be either depending on which way the heads face, or vertical? What about alphabets like Arabic where glyphs are not in general separate, and where they can take different forms depending on if they're the first in a word, the last in a word, in the middle of a word, or isolated? What about archaic letter forms? e.g. ſ, þ, and ð in old English. [0] where I will probably embarrass myself by suggesting 犭(dog) + 瓜 (melon) = 狐 (fox), which, though cute, feels like as much a false etymology as suggesting that in English "congress" is the opposite of "progress". Or perhaps Japanese foxes really are melon dogs — I don't know, I can barely count to 4 in that specific writing system.

foul 3y ago

Didn't read the post thoroughly, will do, but why no libgrapheme in there?

lifthrasiir 3y ago
In my understanding this library is meant to be a part of the future C standard library (but that attempt was not successful). As such, it has to keep all past craziness imposed by ISO C---libgrapheme on the other hand only supports UTF-8 and pseudo-UTF-32 ("pseudo" because of uint_least32_t).

smcameron 3y ago

These aren't mistakes of C, these are mistakes made hundreds and hundreds of years ago in the design of various language writing systems. English gets a lot right with its writing system, mostly in its choice of a small alphabet, a small set of punctuation marks, and an almost complete absence of diacritical marks.

mikepurvis 3y ago
Maybe, but we also have to understand that by an accident of history, computers and the internet were originated by English-speaking people, so it makes sense that the defaults (ASCII and so on) were geared toward the specific needs of that language— it makes sense that everything not-English is going to have some degree of feeling "bolted on" to an underlying framework built for English. If French or Greek or Russian or Japanese had happened to be the lingua franca of the Internet, the features those writing systems require would be first class, and the implementation of English would be the one that felt like a bit more of a kludge. Imagine if computers were designed around unicase Arabic languages [1], and then years later there was a "Han Unificiation"-type summit where it was decided that capitalization was to be handled with extra metadata, since an upper- lower-case latin letter are really just the same thing and why bother wasting double the codepoints on that. (I don't have a link handy, but I believe I read somewhere that you can see some of this effect in NES-era RPGs from Japan— that some of the narrative had to be simplified down for the English translations because the text boxes were originally built for a small number of complex characters rather than the larger number of simpler ones that would be required to capture the same meaning.) [1]: https://en.wikipedia.org/wiki/Unicase https://en.wikipedia.org/wiki/Unicase
- dahfizz 3y ago
  The English language evolved alongside the technology. English used to be cursive-only, like something like Arabic. But the language evolved to fit into a modern world, and I think other written languages would benefit from a simplified "print" version as well.
  - dontlaugh 3y ago

Avamander 3y ago

The first thing that comes to mind with C and locales is this legendary MPV commit: https://github.com/mpv-player/mpv/commit/1e70e82baa9193f6f027338b0fab0f5078971fbe https://github.com/mpv-player/mpv/commit/1e70e82baa9193f6f02...

adamrezich 3y ago
that was a fun read—you can feel the catharsis felt in writing it all out.
tehnub 3y ago
Come back wm4, mpv needs you

vpastore 3y ago

[dead]

kazinator 3y ago

The elephant in the room is that Unicode is a steaming pile of shit. It is the source of all the complexity that results in people picking and choosing what they dare to try implementing, and document other things as being off the plate (the application programmer can fend for themselves as best they can, based on what of the missing cruft they need).

jakeogh 3y ago
The lack of superscript uppercase C F Q S X Y and Z really accents U+1F4A9.
Georgelemental 3y ago
What would you change?
- kps 3y ago
  Not them, but: - Combining characters, joiners, modifiers, etc. should have been prefix, not suffix/infix, and in blocks by arity. This would have allowed detecting incomplete sequences, allowed code to skip combinations without knowing their meaning, and made dead-key input trivial. - A standard normalized form should have been specified up front. - Once blocks overflowed, subsequent blocks should have been made large enough that more that one would not be necessary. Unicode has ten different Latin blocks. You've got ‘Kana Extended-B’, ‘Kana Supplement’, ‘Kana Extended-A’ (in that order!), and ‘Small Kana Extension’. (If there's an explanation for ‘Supplement’ vs ‘Extended’ vs ‘Extension’, I've never seen it.) - Character ordering is inconsistent even within scripts, e.g. ÀÁÂ… àáâ but ĀāĂăĄą. - Assigning code points for the UTF-16 encoding is ridiculous. - The contentious cases of CJK unification should have been disambiguated promptly, before use of the originals became entrenched. (I know where to find 2K code points they could have used…) - They recently changed some characters to have ‘emoji presentation’ by default, meaning that existing text needs to be edited to preserve its original appearance.
  - JdeBP 3y ago
    Dead-key input, if done according to ISO 9995, is trivial for exactly that reason. Type all of the combining diacritics, then the base character. * https://jdebp.uk/Softwares/nosh/guide/commands/console-fb-realizer.xml#DEAD-KEYS https://jdebp.uk/Softwares/nosh/guide/commands/console-fb-re...

cryptonector 3y ago

> Standard C’s primary deficiency is its constant clinging to and dependency upon the “multibyte” encoding and the “wide character” encoding. What? No, UTF-8 won for a reason, and that reason is not just that C has a deficiency in this area but, rather, that UTF-8 is: a) simple and much saner than UTF-16, b) self-synchronizing in both directions, c) as -or even more- space efficient than UTF-16 on average even for non-Latin text, d) even UTF-32 doesn't make it possible to turn logical character string indices into 32-bit word string indices. (d) is the real killer. One just cannot assume that a character or glyph requires just one codepoint to express, therefore one can't assume that a character or glyph will require some fixed number of code units to express, therefore one might as well use UTF-8 because it's saner and more efficient than the other UTFs, therefore... <drumroll/> a "constant clinging to and dependency upon the “multibyte” encoding" is NOT a deficiency of C but an advantage of C. C does have problems here though, namely all the usual problems it has: - C strings (NUL-terminated) suck - C doesn't have a first class string type, only pointer to char - C doesn't have a first class string type that indicates what encoding the string uses Now, C's wchar_t is not even a deficiency but a disaster.

Taywee 3y ago
I don't think that's what the complaint is. The complaint is that "multibyte" is not necessarily UTF-8. You can't just blindly convert to multibyte assuming that it's UTF-8, because it might not be. You can't convert between two encodings by just going through "multibyte", because it might actually not support all characters you might need to support. So it really is a deficiency in C. It's nearly useless to have a "multibyte" or "wide character" encoding when those can mean anything. Having conversion between UTF-8 and UTF-32 is useful. Having conversion between "implemetation and platform dependent 'multibyte'" and "implementation and platform dependent 'wide character'" strings is nearly useless.
- cryptonector 3y ago
  > You can't just blindly convert to multibyte assuming that it's UTF-8, because it might not be. I mentioned that. TFA didn't say that specifically.