6 ms·

I doubt you can handle UTF-8 properly with that attitude. The problems is, there is one very popular OS which is very hard to enforce UTF-8 everywhere, Microso

by ezoe 2y ago

I doubt you can handle UTF-8 properly with that attitude. The problems is, there is one very popular OS which is very hard to enforce UTF-8 everywhere, Microsoft Windows. It's very hard to ensure all the software stack you are depending on it use Unicode version of Win32 API. Actually the native character encoding in Windows is UTF-16 so you can't just assume UTF-8. If you're writing low level code, you have to convert UTF-8 to UTF-16 and back. Even if you don't you have to ensure all the low level code you are depending on it do the same for you. Oh and don't forget about the Unicode Normalizations. There is no THE UTF-8. There are bunch of UTF-8s with different Unicode normalizations. Apple macOS use NFD while other mostly use NFC. These are Just some examples. When people living in ASCII world casually said "I just assume UTF-8", in reality, you still assume it's ASCII.

dgellow 2y ago
I’ve been sharing it multiple times but I love it: WTF-16 spec https://simonsapin.github.io/wtf-8/#ill-formed-utf-16 https://simonsapin.github.io/wtf-8/#ill-formed-utf-16
int_19h 2y ago
> If you're writing low level code, you have to convert UTF-8 to UTF-16 and back. It depends. If you're writing an app, just add the necessary incantation to your manifest, and all the narrow char APIs start talking UTF-8 to you. For a library, yeah.
- ezoe 2y ago
  Don't be surprised such UTF-8 locale program will break so bad on Windows with default language set to Japanese.
  - int_19h 2y ago
    Why would it break? If you just assume that the system codepage is UTF-8, then sure. If you specifically say in your manifest that you want UTF-8, then Windows (10+) will give you UTF-8 regardless of which locale it is: https://learn.microsoft.com/en-us/windows/apps/design/globalizing/use-utf8-code-page https://learn.microsoft.com/en-us/windows/apps/design/global...
josephg 2y ago
> Actually the native character encoding in Windows is UTF-16 so you can't just assume UTF-8. If you're writing low level code, you have to convert UTF-8 to UTF-16 and back. Yes. You should convert your strings. Thankfully, UTF-16 is very difficult to confuse with UTF-8 because they're completely incompatible encodings. Conversion is (or should be) a relatively simple process in basically any modern language or environment. And personally, I've never run into a problem where the difference between NFC and NFD mattered. (Do you have an example?). The different forms are (or should be) visually completely identical for the user - at least on modern computers with decent unicode fonts. The largest problem with UTF-8 (and its biggest strength) is how similar it is to ASCII. It is for this reason we should consider emoji to be a wonderful gift to software correctness everywhere. Correctly handling emoji requires that your software can handle unicode correctly - because they need multi-unit encoding with both UTF-16 and UTF-8. And emoji won't render correctly unless your software can also handle grapheme clusters. > When people living in ASCII world casually said "I just assume UTF-8", in reality, you still assume it's ASCII. Check! If your application deals with text, throw your favorite multi-codepoint emoji into your unit testing data. (Mine is the polar bear). Users love emoji, and your software should handle it correctly. There's no excuse! Even the windows filesystem passes this test today.
- kevin_thibedeau 2y ago
  > your software should handle it correctly. There's no excuse! It is valid for the presentation of compound emoji can fallback to their component parts. You can't expect every platform to have an up to date database of every novel combination. A better test is emoji with color modifiers. Another good one is grandfathered symbols with both a text and emoji presentation and forcing the chosen glyph with the variant selector prefix.
  - josephg 2y ago
    > You can't expect every platform to have an up to date database of every novel combination. On modern desktop OSes and smart phones, I do expect my platform to have an up-to-date unicode database & font set. Certainly for something like the unicode polar bear, which was added in 2020. I'll begrudgingly look the other way for terminals, embedded systems and maybe video games... but generally it should just work everywhere. Server code generally shouldn't interact with unicode grapheme clusters at all. I'm struggling to think of any common, valid reason to use a unicode character database in 'normal' backend server code. > Another good one is grandfathered symbols with both a text and emoji presentation and forcing the chosen glyph with the variant selector prefix. I didn't know about that one. I'll have to try it out.
magicalhippo 2y ago
We make some B2B software running on Windows, integrating with customer systems. We get a lot of interesting files. About a decade ago I wrote some utility code for reading files, where it'll try to detect BOM first, if not scan for invalid UTF-8 sequences. If none are found assume UTF-8 else assume Windows-1252. Worked well for us so far. Still get the occasional flat file in Windows-1252 with one random field containing UTF-8, so some special handling is needed for those cases. But that's rare. Fortunately we don't have to worry about normalization for the most part. If we're parsing then any delimiters will be one of the usual suspects and the rest data.
thayne 2y ago
> Apple macOS use NFD while other mostly use NFC. It's actually worse than that. Older versions of mac did enforce NFD for file names, but more recent names don't, at least at the OS level. But many apple programs, such as finder _will_ use NFD. Except that it isn't even Unicode standardized NFD, it is Apple's own modified version of it. And this can cause issues when for example you create a file in finder, then search for it using `find`, and type the name of the file the exact same way, but it can't find the file because find got an NFC form, but the actual file is in NFD. OTOH, in many applications, you don't really care about the normalization form used. For example, if you are parsing a CSV, you probably don't need to worry about if one of the cells using using a single code point or two code points to represent that accented e.
- neonsunset 2y ago
  Thanks, yet another quantum of knowledge that makes one's life irreversibly ever so slightly worse. But not as bad as encryption (and learning all the terrible ways most applications have broken implementations in)
  - staunton 2y ago
    > most applications have broken implementations What applications? Almost nobody writes their own implementations of encryption nowadays (nor should they). You mean openssl is "broken"?
    - neonsunset 2y ago
      By broken implementations I meant incorrect usage of cryptographic APIs - padding errors, nonce reuse, using weak hash functions, etc.
- ezoe 2y ago
  Yeah, I know that but omit it to make my comment shorter. The world will be a slightly better if there is no macOS.
  - 2y ago
morpheuskafka 2y ago
I got a ticket about some "folders with Chinese characters" showing up on an SMB share at work, my first thought was a Unicode issue and sure enough when you combine two UTF-8/ASCII A-z code points together as one UTF-16 code point, it will usually wind up in the CJK Common Ideographs range of Unicode. Some crappy software had evidently bypassed the appropriate Windows APIs and just directly wrote a C-style ASCII string onto the filesystem without realizing that NTFS is UTF-16.
surfingdino 2y ago
Microsoft Windows is a source of many a headache for me as almost every other client I write code for has to deal with data created by humans using MS Office. Ordinary users could be excused, because they are not devs but even devs don't see a difference between ASCII and UTF-8 and continue to write code today as if it was 1986 and nobody needed to support accented characters.
0xEF 2y ago
Do you know of a resource that explains character encoding in greater detail? Just for my own curiosity. I am learning web development and boy, they brow beat UTF-8 upon us which okay, I'll make sure that call is in my meta data, but none bother to explain how or why we got to that point, or why it seems so splintered.
- screwt 2y ago
  (very late reply, but in case you see it) This Joel On Software article [0] is a good starting point. Incredibly it's now over 20 years old so that makes me feel ancient! But still relevant today. The suggestion that the web should just use utf-8 everywhere is largely true today. But we still have to interact with other software that may not use utf-8 for various legacy reasons - the CSV file example in the original article is a good example. Joel's article also mentions the solution discussed in the original article, i.e. use heuristics to deduce the encoding. [0] https://www.joelonsoftware.com/2003/10/08/the-absolute-minimum-every-software-developer-absolutely-positively-must-know-about-unicode-and-character-sets-no-excuses/ https://www.joelonsoftware.com/2003/10/08/the-absolute-minim...
Comma2976 2y ago
Some[1] would see breaking Windows as a feature [1]Me, surely at least 1 other
- mikhailfranco 2y ago
  Some [1] may also consider working for any company/app that needs to display an emoji, to be a waste of at least one life (your life, and all your users' lives). [1] Me, for sure.
  - Comma2976 2y ago
    Some[1] may have missed the point, emojis were never supported by some[2] in their projects and they consider them to be the mark of the beast[3] [1]you may find him at ::1 [2]HEAD~2 [3]Front-end work

by ezoe 2y ago

dgellow 2y ago

I’ve been sharing it multiple times but I love it: WTF-16 spec https://simonsapin.github.io/wtf-8/#ill-formed-utf-16 https://simonsapin.github.io/wtf-8/#ill-formed-utf-16

int_19h 2y ago

> If you're writing low level code, you have to convert UTF-8 to UTF-16 and back. It depends. If you're writing an app, just add the necessary incantation to your manifest, and all the narrow char APIs start talking UTF-8 to you. For a library, yeah.

ezoe 2y ago
Don't be surprised such UTF-8 locale program will break so bad on Windows with default language set to Japanese.
- int_19h 2y ago
  Why would it break? If you just assume that the system codepage is UTF-8, then sure. If you specifically say in your manifest that you want UTF-8, then Windows (10+) will give you UTF-8 regardless of which locale it is: https://learn.microsoft.com/en-us/windows/apps/design/globalizing/use-utf8-code-page https://learn.microsoft.com/en-us/windows/apps/design/global...

josephg 2y ago

> Actually the native character encoding in Windows is UTF-16 so you can't just assume UTF-8. If you're writing low level code, you have to convert UTF-8 to UTF-16 and back. Yes. You should convert your strings. Thankfully, UTF-16 is very difficult to confuse with UTF-8 because they're completely incompatible encodings. Conversion is (or should be) a relatively simple process in basically any modern language or environment. And personally, I've never run into a problem where the difference between NFC and NFD mattered. (Do you have an example?). The different forms are (or should be) visually completely identical for the user - at least on modern computers with decent unicode fonts. The largest problem with UTF-8 (and its biggest strength) is how similar it is to ASCII. It is for this reason we should consider emoji to be a wonderful gift to software correctness everywhere. Correctly handling emoji requires that your software can handle unicode correctly - because they need multi-unit encoding with both UTF-16 and UTF-8. And emoji won't render correctly unless your software can also handle grapheme clusters. > When people living in ASCII world casually said "I just assume UTF-8", in reality, you still assume it's ASCII. Check! If your application deals with text, throw your favorite multi-codepoint emoji into your unit testing data. (Mine is the polar bear). Users love emoji, and your software should handle it correctly. There's no excuse! Even the windows filesystem passes this test today.

kevin_thibedeau 2y ago
> your software should handle it correctly. There's no excuse! It is valid for the presentation of compound emoji can fallback to their component parts. You can't expect every platform to have an up to date database of every novel combination. A better test is emoji with color modifiers. Another good one is grandfathered symbols with both a text and emoji presentation and forcing the chosen glyph with the variant selector prefix.
- josephg 2y ago
  > You can't expect every platform to have an up to date database of every novel combination. On modern desktop OSes and smart phones, I do expect my platform to have an up-to-date unicode database & font set. Certainly for something like the unicode polar bear, which was added in 2020. I'll begrudgingly look the other way for terminals, embedded systems and maybe video games... but generally it should just work everywhere. Server code generally shouldn't interact with unicode grapheme clusters at all. I'm struggling to think of any common, valid reason to use a unicode character database in 'normal' backend server code. > Another good one is grandfathered symbols with both a text and emoji presentation and forcing the chosen glyph with the variant selector prefix. I didn't know about that one. I'll have to try it out.

magicalhippo 2y ago

We make some B2B software running on Windows, integrating with customer systems. We get a lot of interesting files. About a decade ago I wrote some utility code for reading files, where it'll try to detect BOM first, if not scan for invalid UTF-8 sequences. If none are found assume UTF-8 else assume Windows-1252. Worked well for us so far. Still get the occasional flat file in Windows-1252 with one random field containing UTF-8, so some special handling is needed for those cases. But that's rare. Fortunately we don't have to worry about normalization for the most part. If we're parsing then any delimiters will be one of the usual suspects and the rest data.

thayne 2y ago

> Apple macOS use NFD while other mostly use NFC. It's actually worse than that. Older versions of mac did enforce NFD for file names, but more recent names don't, at least at the OS level. But many apple programs, such as finder _will_ use NFD. Except that it isn't even Unicode standardized NFD, it is Apple's own modified version of it. And this can cause issues when for example you create a file in finder, then search for it using `find`, and type the name of the file the exact same way, but it can't find the file because find got an NFC form, but the actual file is in NFD. OTOH, in many applications, you don't really care about the normalization form used. For example, if you are parsing a CSV, you probably don't need to worry about if one of the cells using using a single code point or two code points to represent that accented e.

neonsunset 2y ago
Thanks, yet another quantum of knowledge that makes one's life irreversibly ever so slightly worse. But not as bad as encryption (and learning all the terrible ways most applications have broken implementations in)
- staunton 2y ago
  > most applications have broken implementations What applications? Almost nobody writes their own implementations of encryption nowadays (nor should they). You mean openssl is "broken"?
  - neonsunset 2y ago
    By broken implementations I meant incorrect usage of cryptographic APIs - padding errors, nonce reuse, using weak hash functions, etc.
ezoe 2y ago
Yeah, I know that but omit it to make my comment shorter. The world will be a slightly better if there is no macOS.
- 2y ago

morpheuskafka 2y ago

I got a ticket about some "folders with Chinese characters" showing up on an SMB share at work, my first thought was a Unicode issue and sure enough when you combine two UTF-8/ASCII A-z code points together as one UTF-16 code point, it will usually wind up in the CJK Common Ideographs range of Unicode. Some crappy software had evidently bypassed the appropriate Windows APIs and just directly wrote a C-style ASCII string onto the filesystem without realizing that NTFS is UTF-16.

surfingdino 2y ago

Microsoft Windows is a source of many a headache for me as almost every other client I write code for has to deal with data created by humans using MS Office. Ordinary users could be excused, because they are not devs but even devs don't see a difference between ASCII and UTF-8 and continue to write code today as if it was 1986 and nobody needed to support accented characters.

0xEF 2y ago

Do you know of a resource that explains character encoding in greater detail? Just for my own curiosity. I am learning web development and boy, they brow beat UTF-8 upon us which okay, I'll make sure that call is in my meta data, but none bother to explain how or why we got to that point, or why it seems so splintered.

screwt 2y ago
(very late reply, but in case you see it) This Joel On Software article [0] is a good starting point. Incredibly it's now over 20 years old so that makes me feel ancient! But still relevant today. The suggestion that the web should just use utf-8 everywhere is largely true today. But we still have to interact with other software that may not use utf-8 for various legacy reasons - the CSV file example in the original article is a good example. Joel's article also mentions the solution discussed in the original article, i.e. use heuristics to deduce the encoding. [0] https://www.joelonsoftware.com/2003/10/08/the-absolute-minimum-every-software-developer-absolutely-positively-must-know-about-unicode-and-character-sets-no-excuses/ https://www.joelonsoftware.com/2003/10/08/the-absolute-minim...

Comma2976 2y ago

Some[1] would see breaking Windows as a feature [1]Me, surely at least 1 other

mikhailfranco 2y ago
Some [1] may also consider working for any company/app that needs to display an emoji, to be a waste of at least one life (your life, and all your users' lives). [1] Me, for sure.
- Comma2976 2y ago
  Some[1] may have missed the point, emojis were never supported by some[2] in their projects and they consider them to be the mark of the beast[3] [1]you may find him at ::1 [2]HEAD~2 [3]Front-end work