2 ms·
I learned almost nothing about Unicode as part of my CS degree. When I graduated, all I was aware of was that it existed and I thought that it used just 16-bit
by Maultasche 11y ago
I learned almost nothing about Unicode as part of my CS degree. When I graduated, all I was aware of was that it existed and I thought that it used just 16-bit characters, which was the original Windows Unicode implementation.
That wasn't exactly accurate, and I didn't have any idea of Unicode code points and the different ways of encoding them (UTF-8, UTF-16, UTF-32, and whatever else).
It wasn't until later when I decided to write a C++ library to deal with Unicode strings did I learn about code points, code units, UTF-8 and UTF-16.
https://github.com/Maultasche/UtfString https://github.com/Maultasche/UtfString
The library essentially re-implements the std::string interface, but deals with code points instead of bytes. It'll successfully handle code points of variable size (1 to 4 bytes for UTF-8 and 1 to 2 16-bit values for UTF-16).
Sorry, the documentation needs to be converted to Markdown. It's currently in HTML that gets fed to doxygen. I wrote this before markdown and Github existed.
Here's an interesting thing I learned while implementing the library. UTF-16 is no longer consists of just 16-bit characters. It did originally, but the number of Unicode code points became too large for 16 bits. Like UTF-8, a single code point (which often, but not always, has a 1-1 relationship with a character), can have a variable number of code units. UTF-16 code points usually consist of a single 16-bit value (aka code unit), but the code points higher up on the Unicode map will consist of two 16-bit values.
If I hadn't done the work and research to write the library, I probably would still only have a vague idea of Unicode and its encodings.