14 ms·
Unicode programming, with examples in C
- macintux 7y agoI found this quite interesting. Perhaps the closest I’ve come to grasping the core concepts, even though I’ve done a fair bit of work around the edges previously.
- jancsika 7y ago> This is called Normalization Form Canonical Composition (NFC). Is there something like a "Round Midnight and No Coffee Form" where the programmer just renders the text to check whether the output of each set of codepoints matches pixel for pixel?
- a1369209993 7y agoThe closest unicode has seems to be NFKC, but last I checked it still didn't correctly handle greek and cyrillic aliases of latin characters, never mind anything more obscure.
- LunaSea 7y agoDo you have examples of this? I wonder if it's implementation specific or due to the NFKC method itself?
- dfox 7y agoThe issue does not have anything to do with normalization per se, but stems from the fact that there are Unicode characters that are semantically different, but nevertheless look exactly same in most fonts. For example 'a' (U+0061 LATIN SMALL LETTER A) vs. 'а' (U+0430 CYRILLIC SMALL LETTER A)
- ygra 7y agoIt's probably also futile to keep homoglyph tables without the context of the font being used, as т and т are the same letter, just with different styles (and in some fonts resemble T and m respectively in Latin). Unicode mentions homoglyphs briefly in UTR #36, but overall I feel that it's not really their job to solve that issue, as visually similar characters already exist in ASCII and other limited character sets and every context where those things are a problem probably needs to be evaluated differently with different mitigations.
- efitz 7y agoI know you were joking but http://patft.uspto.gov/netacgi/nph-Parser?Sect1=PTO2&Sect2=HITOFF&p=1&u=%2Fnetahtml%2FPTO%2Fsearch-bool.html&r=1&f=G&l=50&co1=AND&d=PTXT&s1=10127388&OS=10127388&RS=10127388 http://patft.uspto.gov/netacgi/nph-Parser?Sect1=PTO2&Sect2=H...
- stevenwoo 7y agoI cooped at IBM in the 1980s and this is how we tested and verified the newest IBM PC hardware (at this point IIRC it would have been around the 286/pc junior/os2/smaller form factor ibm pc designs) by creating tests on current systems with manual keyboard entry, and recording all the key strokes/timing and screen buffers and output files at intervals then running the same tests on new hardware with a playback of recorded input and comparing the new output to the saved runs.
- stabbles 7y agoI'm just confused the libicu shared libs are ~25MB big. Why is that?
- CaliforniaKarl 7y agoI think it’s because the library, when standing alone, would need to include the entire Unicode database. What I wonder is, when you install the library from a distribution’s package manager, does it package that data separately (so that it can hopefully be used by related software)?
- derefr 7y agoProbably not. I would guess, from working with similar “static-data-heavy” libraries, that much of the data is “burned into” the libicu in the form of lookup functions that execute native-code representations of decision trees to give their responses. This data is meant to be “used by related software” by just linking libicu and asking it about the data. Though, this does mean that in language runtimes that don’t want to pull in native libraries (because the runtime is trying to ensure some guarantee like soft-realtime or fault-tolerance), libicu can’t be linked, so the runtime actually has to do the same thing it does, if it wants to have efficient Unicode lookup to the level that people expect: ship a copy of the database files from Unicode.org with the source, and convert them into source-code representations of decision-tree functions. The language I’m most familiar with, Elixir, employs this strategy to support Unicode; see e.g. https://github.com/elixir-lang/elixir/blob/master/lib/elixir/unicode/unicode.ex https://github.com/elixir-lang/elixir/blob/master/lib/elixir...
- Pete_D 7y agoI've seen many distros provide a unicode-data package, containing the human-readable text files from the Unicode Character Database. These are sometimes handy to have around, but I'd be surprised if much software used them directly.
- rurban 7y agoBecause it has to store all properties of all valid codepoints 0-0x10fff. It does it via perfect hashes for fastest lookup, not via space saving 3-level arrays as most others do. I described various implementation strategies here : http://perl11.org/blog/foldcase.html http://perl11.org/blog/foldcase.html
- kazinator 7y ago> Reading lines into internal UTF-16 representation Fail. > It’s unwise to use UTF-32 to store strings in memory. In this encoding it’s true that every code unit can hold a full codepoint. wchar_t is 32 bits on a number of platforms such as GNU/Linux, MacOS and Solaris. It behooves you to use that, and all the associated library functionality, rather than roll your own.
- bhaak 7y agoCuriously, the paragraph before the line "it's unwise to use UTF-32" ends with "Use the encoding preferred by your library and convert to/from UTF-8 at the edges of the program." And that is the best advice there is. If you have a choice, use UTF-8, otherwise use whatever your libraries use. Unless you have very special needs, forget about UTF-16.
- Const-me 7y ago> Unless you have very special needs, forget about UTF-16. I don’t think programming for Windows, Android, or iOS, or program in / interop with Java, JavaScript, .NET, qualify as “very special needs”.
- Asooka 7y agoEven on Windows it's best to keep your text in UTF-8 and convert it to and from UTF-16 when interacting with win32 APIs. Java, dotNet and JavaScript are the worst of all worlds because you're both stuck with wide characters (in their native string types) and have the intricacies of UTF-16 to consider. I guess the advice might have been better phrased as "Unless you're forced to, or have very special needs, stay away from UTF-16".
- Const-me 7y ago> it's best to keep your text in UTF-8 and convert it to and from UTF-16 when interacting with win32 APIs It’s extra source code to write and then support, extra machine code to execute, and likely extra memory to malloc/free. Too slow, in my book automatically means “not best”. > Java, dotNet and JavaScript are the worst of all worlds because you're both stuck with wide characters (in their native string types) and have the intricacies of UTF-16 to consider. Just a normal UTF-16, like in WinAPI and many other popular languages, frameworks and libraries. E.g. QT is used a lot in the wild. > the advice might have been better phrased as It says exactly the opposite, “Use the encoding preferred by your library and convert to/from UTF-8 at the edges of the program.”
- jhasse 7y agoBtw: In C99 you don't have to declare all your variables at the top. You can also use // comments.
- vardump 7y agoWe embedded devs might be able to start consistently using C99 in about fifty years or so. Hopefully. Lots of development kits barely support C89.
- goto11 7y agoCouldn't pre-processors be used to add such features? (I don't know anything about embedded development.)
- mbel 7y agoCPP would not be enough, you would need full C99-to-C89 compiler with parsing to AST.
- anilakar 7y agoIronically, most legacy C environments are commercial in nature. Open source tools seem to have a lot better track record in this regard.
- jhasse 7y agoBut the ICU headers require C99 anyway.
- begriffs 7y agoAs it says in the article: > The examples in this article conform to the C89 standard, but we specify C99 in the Makefile because the ICU header files use C99-style (//) comments.
- gpvos 7y agoMoney quote: "Puh-leaze, if your program can’t handle Medieval Irish carvings then I want nothing to do with it."