4 ms·
The most practical real-world solution across all operating systems is to use "UTF-8 everywhere", and only convert from and to UTF-16 or UTF-32 "at the last min
by flohofwoe 7y ago
The most practical real-world solution across all operating systems is to use "UTF-8 everywhere", and only convert from and to UTF-16 or UTF-32 "at the last minute" when the string data is coming from or going into APIs that don't (yet) accept UTF-8 directly.
Also see: http://utf8everywhere.org/ http://utf8everywhere.org/
- iforgotpassword 7y agoI've always wondered why Windows doesn't allow setting the legacy code page to utf8. It would've been a nice hack to get old apps to work with Unicode, assuming they were at least somewhat MBCS aware. If the programmer assumed a character would always be one byte it might not have worked, but then that program would also have failed with a Japanese code page.
- flohofwoe 7y agoThere's a fairly new experimental setting in Win10 where UTF-8 is set as the system-wide codepage, allowing to call the "narrow string" functions with UTF-8 strings. See https://en.wikipedia.org/wiki/Unicode_in_Microsoft_Windows#UTF-8 https://en.wikipedia.org/wiki/Unicode_in_Microsoft_Windows#U... "With insider build 17035 and the April 2018 update (nominal build 17134) for Windows 10, a "Beta: Use Unicode UTF-8 for worldwide language support" checkbox appeared for setting the locale code page to UTF-8.[a] This allows for calling "narrow" functions, including fopen and SetWindowTextA, with UTF-8 strings."
- iforgotpassword 7y agoThat's just awesome. I've wanted this since the win2k/xp days! Back then there were still a lot of non-Unicode programs around. It's less useful today, plus I'm not even using windows anymore.... But just seeing this happen feels great, satisfies past me.
- rkagerer 7y agoFrom that same link: Another popular work-around is to convert the name to the 8.3 filename equivalent, this is necessary if the fopen is inside a library function that takes a string filename and cannot be altered. 8.3 filenames for the win!
- ChrisSD 7y agoThough do note this is only an issue when using particular "narrow" functions. If you consistently use the recommended "wide" functions then none of these issues occur.
- mark-r 7y agoWindows has had a codepage number for UTF-8 for a very long time. As I understand it, it's not recommended because too many things break when a multi-byte codepoint gets broken apart accidentally.
- iforgotpassword 7y agoThat's just a pseudo codepage unfortunately. You can use it when calling functions like MultibyteToWideChar etc. but not set it as the actual locale for processes.
- mark-r 7y agoThanks, I didn't know that detail. `chcp 65001` seems to work. I like the way they handle it in Python 3.6. They use their own internal encoding, then bypass the regular console byte API when they need to output a string, allowing full Unicode.
- cesarb 7y agoI saw once an explanation, probably in the blog which used to be at http://blogs.msdn.com/michkap/ http://blogs.msdn.com/michkap/, but I'm not finding it anymore. IIRC, the reason was that there is a maximum number of bytes per character for a legacy multibyte code page, I don't recall if it was 2 bytes or 3 bytes (there should be a #define somewhere for it), while for UTF-8 the maximum number of bytes per character is 4 bytes. Allowing a user to set the legacy code page to CP65001 (the legacy code page number for UTF-8) could crash programs which depended on the original limit.
- juped 7y agoMichael Kaplan's blog was indeed great; Microsoft took it down, then he started blogging elsewhere, then he unfortunately died in 2015, taking his second blog with him. Happily, archives are available at http://archives.miloush.net/michkap/archive/ http://archives.miloush.net/michkap/archive/
- iforgotpassword 7y agoOh. I remember reading his blog occasionally back in the day. I faintly remember there was a disagreement with Microsoft about his blog or at least some post in it, so he moved to an external blog. Great to see someone archived that stuff.
- cesarb 7y agoYes, that was it, thanks! It's at http://archives.miloush.net/michkap/archive/2006/07/14/665714.html http://archives.miloush.net/michkap/archive/2006/07/14/66571... (Can the CP_ACP be UTF-8?) which links to http://archives.miloush.net/michkap/archive/2005/02/06/368081.html http://archives.miloush.net/michkap/archive/2005/02/06/36808... (Can a codepage be changed? How about which codepage a locale points to?) and http://archives.miloush.net/michkap/archive/2006/07/04/656051.html http://archives.miloush.net/michkap/archive/2006/07/04/65605... (Behind Norman's 'Who needs Unicode?' post). "It is not possible given both the current architecture (which must work in both user and kernel mode) and also the inherent assumption in several subsystems (like USER) that the ACPs maximum number of bytes per character is 2." And the #define would have been MB_LEN_MAX.
- ChrisSD 7y agoOn Windows this isn't great advice if you're making heavy use of Windows APIs. You end up needlessly converting back and forth between UTF-16 and UTF-8. So it often makes more sense to simple use UTF-16 for your own strings and assume UTF-16 for strings returned by the WinAPI (but handle invalid sequences appropriately).
- hinkley 7y agoUTF-8 has good space behavior for English but as you get higher in the space you end up with 24 bits per character for languages that could expect 16 bits per character with another encoding. That said, UTF-8 + gzip on the wire isn’t a bad solution.
- jeremysalwen 7y agoThe website addresses this issue: http://utf8everywhere.org/#asian http://utf8everywhere.org/#asian . The short answer is that most text is not purely high code points, but contains, e.g. HTML tags.
- 08-15 7y agoHow much of the data transmitted or stored on any given system is actually text, as opposed to images, audio, video, machine code, application specific random stuff? Of the little bit of text, how much is HTML, CSS, JSON, any programming language; all of which would be pretty much all ASCII? And if you can find a niche application where most of your data is CJK text, is it really so much that it fits on a USB pen drive if encoded in BIG-5 but not if encoded in UTF-8? My rough guesses at answers are 0.1%, 90%, and no. Besides, UTF-32 plus gzip isn't a bad solution, and completely ends all discussion about transfer formats... at the cost of requiring a compressor.
- WalterBright 7y agoBack in 2000, when I was doing the initial design of the D programming language, I was convinced that Unicode was the future (things weren't as clear then). Earlier, I had put various multibyte encodings into the Zortech C compiler (Shift-JIS, etc.). But it wasn't clear at all whether to use UTF-8, -16 or -32. Windows and Java used UTF-16. So D became agnostic about that, supporting all three equally. Over time it became clear that you're right, the best approach was to use UTF-8 everywhere, and last-minute convert to/from other formats.