4 ms·

I've always wondered why Windows doesn't allow setting the legacy code page to utf8. It would've been a nice hack to get old apps to work with Unicode, assuming

I've always wondered why Windows doesn't allow setting the legacy code page to utf8. It would've been a nice hack to get old apps to work with Unicode, assuming they were at least somewhat MBCS aware. If the programmer assumed a character would always be one byte it might not have worked, but then that program would also have failed with a Japanese code page.

flohofwoe 7y ago
There's a fairly new experimental setting in Win10 where UTF-8 is set as the system-wide codepage, allowing to call the "narrow string" functions with UTF-8 strings. See https://en.wikipedia.org/wiki/Unicode_in_Microsoft_Windows#UTF-8 https://en.wikipedia.org/wiki/Unicode_in_Microsoft_Windows#U... "With insider build 17035 and the April 2018 update (nominal build 17134) for Windows 10, a "Beta: Use Unicode UTF-8 for worldwide language support" checkbox appeared for setting the locale code page to UTF-8.[a] This allows for calling "narrow" functions, including fopen and SetWindowTextA, with UTF-8 strings."
- iforgotpassword 7y ago
  That's just awesome. I've wanted this since the win2k/xp days! Back then there were still a lot of non-Unicode programs around. It's less useful today, plus I'm not even using windows anymore.... But just seeing this happen feels great, satisfies past me.
- rkagerer 7y ago
  From that same link: Another popular work-around is to convert the name to the 8.3 filename equivalent, this is necessary if the fopen is inside a library function that takes a string filename and cannot be altered. 8.3 filenames for the win!
  - ChrisSD 7y ago
    Though do note this is only an issue when using particular "narrow" functions. If you consistently use the recommended "wide" functions then none of these issues occur.
mark-r 7y ago
Windows has had a codepage number for UTF-8 for a very long time. As I understand it, it's not recommended because too many things break when a multi-byte codepoint gets broken apart accidentally.
- iforgotpassword 7y ago
  That's just a pseudo codepage unfortunately. You can use it when calling functions like MultibyteToWideChar etc. but not set it as the actual locale for processes.
  - mark-r 7y ago
    Thanks, I didn't know that detail. `chcp 65001` seems to work. I like the way they handle it in Python 3.6. They use their own internal encoding, then bypass the regular console byte API when they need to output a string, allowing full Unicode.
cesarb 7y ago
I saw once an explanation, probably in the blog which used to be at http://blogs.msdn.com/michkap/ http://blogs.msdn.com/michkap/, but I'm not finding it anymore. IIRC, the reason was that there is a maximum number of bytes per character for a legacy multibyte code page, I don't recall if it was 2 bytes or 3 bytes (there should be a #define somewhere for it), while for UTF-8 the maximum number of bytes per character is 4 bytes. Allowing a user to set the legacy code page to CP65001 (the legacy code page number for UTF-8) could crash programs which depended on the original limit.
- juped 7y ago
  Michael Kaplan's blog was indeed great; Microsoft took it down, then he started blogging elsewhere, then he unfortunately died in 2015, taking his second blog with him. Happily, archives are available at http://archives.miloush.net/michkap/archive/ http://archives.miloush.net/michkap/archive/
  - iforgotpassword 7y ago
    Oh. I remember reading his blog occasionally back in the day. I faintly remember there was a disagreement with Microsoft about his blog or at least some post in it, so he moved to an external blog. Great to see someone archived that stuff.
  - cesarb 7y ago
    Yes, that was it, thanks! It's at http://archives.miloush.net/michkap/archive/2006/07/14/665714.html http://archives.miloush.net/michkap/archive/2006/07/14/66571... (Can the CP_ACP be UTF-8?) which links to http://archives.miloush.net/michkap/archive/2005/02/06/368081.html http://archives.miloush.net/michkap/archive/2005/02/06/36808... (Can a codepage be changed? How about which codepage a locale points to?) and http://archives.miloush.net/michkap/archive/2006/07/04/656051.html http://archives.miloush.net/michkap/archive/2006/07/04/65605... (Behind Norman's 'Who needs Unicode?' post). "It is not possible given both the current architecture (which must work in both user and kernel mode) and also the inherent assumption in several subsystems (like USER) that the ACPs maximum number of bytes per character is 2." And the #define would have been MB_LEN_MAX.

flohofwoe 7y ago

There's a fairly new experimental setting in Win10 where UTF-8 is set as the system-wide codepage, allowing to call the "narrow string" functions with UTF-8 strings. See https://en.wikipedia.org/wiki/Unicode_in_Microsoft_Windows#UTF-8 https://en.wikipedia.org/wiki/Unicode_in_Microsoft_Windows#U... "With insider build 17035 and the April 2018 update (nominal build 17134) for Windows 10, a "Beta: Use Unicode UTF-8 for worldwide language support" checkbox appeared for setting the locale code page to UTF-8.[a] This allows for calling "narrow" functions, including fopen and SetWindowTextA, with UTF-8 strings."

iforgotpassword 7y ago
That's just awesome. I've wanted this since the win2k/xp days! Back then there were still a lot of non-Unicode programs around. It's less useful today, plus I'm not even using windows anymore.... But just seeing this happen feels great, satisfies past me.
rkagerer 7y ago
From that same link: Another popular work-around is to convert the name to the 8.3 filename equivalent, this is necessary if the fopen is inside a library function that takes a string filename and cannot be altered. 8.3 filenames for the win!
- ChrisSD 7y ago
  Though do note this is only an issue when using particular "narrow" functions. If you consistently use the recommended "wide" functions then none of these issues occur.

mark-r 7y ago

Windows has had a codepage number for UTF-8 for a very long time. As I understand it, it's not recommended because too many things break when a multi-byte codepoint gets broken apart accidentally.

iforgotpassword 7y ago
That's just a pseudo codepage unfortunately. You can use it when calling functions like MultibyteToWideChar etc. but not set it as the actual locale for processes.
- mark-r 7y ago
  Thanks, I didn't know that detail. `chcp 65001` seems to work. I like the way they handle it in Python 3.6. They use their own internal encoding, then bypass the regular console byte API when they need to output a string, allowing full Unicode.

cesarb 7y ago

I saw once an explanation, probably in the blog which used to be at http://blogs.msdn.com/michkap/ http://blogs.msdn.com/michkap/, but I'm not finding it anymore. IIRC, the reason was that there is a maximum number of bytes per character for a legacy multibyte code page, I don't recall if it was 2 bytes or 3 bytes (there should be a #define somewhere for it), while for UTF-8 the maximum number of bytes per character is 4 bytes. Allowing a user to set the legacy code page to CP65001 (the legacy code page number for UTF-8) could crash programs which depended on the original limit.

juped 7y ago
Michael Kaplan's blog was indeed great; Microsoft took it down, then he started blogging elsewhere, then he unfortunately died in 2015, taking his second blog with him. Happily, archives are available at http://archives.miloush.net/michkap/archive/ http://archives.miloush.net/michkap/archive/
- iforgotpassword 7y ago
  Oh. I remember reading his blog occasionally back in the day. I faintly remember there was a disagreement with Microsoft about his blog or at least some post in it, so he moved to an external blog. Great to see someone archived that stuff.
- cesarb 7y ago
  Yes, that was it, thanks! It's at http://archives.miloush.net/michkap/archive/2006/07/14/665714.html http://archives.miloush.net/michkap/archive/2006/07/14/66571... (Can the CP_ACP be UTF-8?) which links to http://archives.miloush.net/michkap/archive/2005/02/06/368081.html http://archives.miloush.net/michkap/archive/2005/02/06/36808... (Can a codepage be changed? How about which codepage a locale points to?) and http://archives.miloush.net/michkap/archive/2006/07/04/656051.html http://archives.miloush.net/michkap/archive/2006/07/04/65605... (Behind Norman's 'Who needs Unicode?' post). "It is not possible given both the current architecture (which must work in both user and kernel mode) and also the inherent assumption in several subsystems (like USER) that the ACPs maximum number of bytes per character is 2." And the #define would have been MB_LEN_MAX.