5 ms·
As there are a lot of C-masters lurking in this thread: How can one process unicode (UTF-8) properly in C? As a CJK person, I wish there was a robust solution.
by pcr910303 7y ago
As there are a lot of C-masters lurking in this thread:
How can one process unicode (UTF-8) properly in C? As a CJK person, I wish there was a robust solution. Are there any standardized ways or proposals? (Using wchar doesn't count.)
- DougGwyn 7y agoUTF-8 encoding works "as is" based on byte strings (char[]). The latest versions of the draft standard provide somewhat more support. I recommend heading toward a future where only UTF-8 encoding is used for multibyte characters and UCS-2 or similar for wchar_t. There is no need to support several different encodings.
- deleted 7y ago[deleted]
- rseacord 7y agoAaron Ballman even got a u8 character prefix added to C2x: N2198 2018/01/02 Ballman, Adding the u8 character prefix http://www.open-std.org/jtc1/sc22/wg14/www/docs/n2198.pdf http://www.open-std.org/jtc1/sc22/wg14/www/docs/n2198.pdf
- ori_b 7y agoUCS-2 is a bad choice -- it fails to represent most unicode characters. If you meant UTF-16, that's also a bad choice, because UTF-16 is also a variable width encoding, forcing programmers to use a some for of "extra-wide char". I'm of the opinion that wchar_t should become an alias for char32_t.
- DougGwyn 7y agoYes, I meant the 31-bit code point value (more than 16, anyway). It is the most useful width for doing things with wide characters.
- a1369209993 7y agoUTF-32 is also a variable-width encoding; eg 00000044 00000308 aka "D̈".
- DougGwyn 7y agoI thought it was strictly one character per 32-bit code. Anyway, whatever it is called it is what wchar_t should be.
- a1369209993 7y agoThere are no fixed width encodings with range of encodable characters anywhere near that of Unicode.
- flatfinger 7y agoIt's too bad Unicode wasn't designed around the concept of easily-recognizable grapheme clusters and "write-only" [non-round-trip] forms that are normalized in various ways. A text layout engine shouldn't have to have detailed knowledge of rules that are constantly subject to change, but if there were a standard representation for a Unicode string where all grapheme clusters are marked and everything is listed in left-to-right order, and an OS function was available to convert a Unicode string into such a form, a text-layout using that OS routine would be able to accommodate future additions to the character set and and glyph-joining rules without having to know anything about them.
- a1369209993 7y agoYou can't do that without commiting to not supporting pathological text, otherwise you're stuck adding new special cases to the layout engine every update anyway. I do have some ideas for a better encoding (like, I assume, anyone competent with sufficient free time and interest in text encoding), but there's a lot of reluctance to put effort into something that's already completely eclipsed by a technically inferior but not completely unusable alternative, so I've had it mostly shelved.
- Tronic2 7y agoIgnore all character support in the standard library and handle UTF-8 as opaque binary buffers. If you need complex string algorithms, decode into UCS-4 (UTF-32). You'll find short encoding and decoding functions on StackOverflow. For case-insensitive comparisons and sorting, use an external library that knows the latest Unicode standard.
- barbegal 7y agoExcept that not all binary data is valid UTF-8 so you also need functions that check if a binary buffer is valid UTF-8.
- Tronic2 7y agoThe decoding phase will do that, if needed. Also note that in many cases you must process it as opaque binary, even though it should be valid UTF-8. This is in particular with filenames on POSIX systems because otherwise you could not access any files that happen to have invalid UTF-8 in their names.
- oreally 7y agoCheck this: http://utf8everywhere.org/ http://utf8everywhere.org/ Basically store the text as char arrays, and convert them when needed. Meanwhile, you could use this single file header: https://github.com/RandyGaul/cute_headers/blob/master/cute_utf.h https://github.com/RandyGaul/cute_headers/blob/master/cute_u...
- pascal_cuoq 7y agoAs a reviewer for Robert's upcoming C book “Effective C”, I thought that this aspect was better covered than in existing manuals for learning C. However, the book only describes the available standard functions, so even doing better than other manuals, everything it has to say on this subject fits in one chapter and feel underpowered.
- ori_b 7y agohttps://bitbucket.org/knight666/utf8rewind/src/default/ https://bitbucket.org/knight666/utf8rewind/src/default/
- loeg 7y agoWhat sort of processing do you want to do?
- begriffs 7y agoYour best bet is probably to use a library like ICU. Here are examples of working with unicode in C: https://begriffs.com/posts/2019-05-23-unicode-icu.html https://begriffs.com/posts/2019-05-23-unicode-icu.html