4 ms·
The Absolute Minimum Every Software Developer Absolutely, Positively Must Know About Unicode and Character Sets (No Excuses!) from 2003: https://www.joelonsoft
by teddyh 9y ago
The Absolute Minimum Every Software Developer Absolutely, Positively Must Know About Unicode and Character Sets (No Excuses!) from 2003:
https://www.joelonsoftware.com/2003/10/08/the-absolute-minimum-every-software-developer-absolutely-positively-must-know-about-unicode-and-character-sets-no-excuses/ https://www.joelonsoftware.com/2003/10/08/the-absolute-minim...
- nabla9 9y agoThat is not the absolute minimum. Unicode is complex beast and oversimplification is dangerous. When these absolute minimum intros talk just about encoding it misleads people to think that that's enough. I can't count the number of people who have read Joel's article and have the misconception that all user perceived characters are mapped to code points. I was one of those people. Just because ASCII and Latin-1 character sets can be mapped to code points does not mean that's how Unicode works. At minimum every software developer must know four different levels: * bytes, * code points, * combining character sequence * grapheme clusters, extended grapheme clusters Joel stops at the second level. He never gets into point where he explains how encode user perceived characters, how to detect grapheme cluster boundaries in the Unicode encoding. examples: 각 , नी , நி
- TuringTest 9y agoMaybe the perfect is enemy of the good? If a developer knows and understands the concept of character encoding ("It does not make sense to have a string without knowing what encoding it uses"), at least they will know how to read a string from one system and move it to a different system that expects a different encoding. They'll know that they need to call the relevant conversion routine in a specialized library that knows how to handle the conversion. With this, maybe they won't be able to correctly build or modify their own strings directly. But being able to handle strings from an external system that produces them, and passing them to another system that consumes them, without breaking them in the process, IMHO does qualify as "the absolute minimum" they should know.
- jstimpfle 9y agoThanks, but I'm happy to be ignorant about levels three and four. I know they exist, and that's enough for me. Something is really wrong if programmers of most applications domains have to care about that complexity.
- nabla9 9y agoKnowing that their exist is the minimum you must know. Knowing what you don't know is already knowledge. Joel gives the impression that he don't know that he don't know. Knowing that you can't break a Unicode text string or insert text into the middle of Unicode string unless you know what language it uses is usually enough. They are just binary blocks you can't modify unless you have some extra info or uses specific libraries.
- Manishearth 9y agoYou have to know they exist, and know when you should worry about them (and call in to the appropriate APIs) https://manishearth.github.io/blog/2017/01/15/breaking-our-latin-1-assumptions/ https://manishearth.github.io/blog/2017/01/15/breaking-our-l... gives an overview of most of the different things scripts do. Being aware of those helps a lot. It also gives a brief idea of how to deal with this stuff (usually it's just calling an API) https://manishearth.github.io/blog/2017/01/14/stop-ascribing-meaning-to-unicode-code-points/ https://manishearth.github.io/blog/2017/01/14/stop-ascribing... gives an idea of how grapheme clusters work. You don't need to know the algorithm, just the stuff around it.
- cryptonector 9y agoAt minimum every software developer must also know about normalization, and must know what pre-composed and decomposed forms are (especially normal forms, NFC and NFD). These things are things one can mostly ignore, until one can't. Most input modes produce pre-composed (but not necessarily NFC) output. But some things will decompose (e.g., HFS+ will decompose filenames). So... if you cut-n-paste non-ASCII Unicode from an HFS+ file picker UI... you'll get into trouble if the software you paste into is unaware of these things. Ultimately, every software developer needs: - UTF validator (at least for UTF-8) - UTF converters (unless only supporting UTF-8) - case mapping (probably) - normalization (almost certainly) - collation (probably) That's... not too bad. Networking software )may_ also need: - IDNA2008 implementation - UTS#46 implementation Word processing / typesetting software also absolutely needs to know about grapheme clusters in order to determine the size of each grapheme. Also: modern fonts.
- kyberias 9y agoSo you're just basically listing everything you know about Unicode and then state that every software developer MUST know the same. That's bullshit of course.
- cryptonector 9y agoWAT? I gave a breakdown of when you might need support for specific aspects of Unicode. That is not an exhaustive list, just a list that will get 95% of developers covered in most cases. I didn't mention bi-di, for example, though I probably should have.
- johan_larson 9y agoWhat Joel covers is just fine for software developers who a) work in a language written in Latin script and b) aren't specifically responsible for internationalization of their product. Deeper issues can be left to specialists.