4 ms·
I'd like to see a bit more explanation why the "UTF-8 Everywhere" (https://utf8everywhere.org/ https://utf8everywhere.org/) advice is poor? Outside Windows API
by flohofwoe 5y ago
I'd like to see a bit more explanation why the "UTF-8 Everywhere" (https://utf8everywhere.org/ https://utf8everywhere.org/) advice is poor?
Outside Windows APIs, UTF-16 is pretty much irrelevant (yes there are a few popular languages around which jumped on the UNICODE bandwagon too early and are now stuck with UTF-16, but those languages usually also offer simple conversion functions to and from other encodings - like UTF-8).
UTF-32 has its place as internal runtime string representation, but not for data exchange.
UTF-8 has won, and for all the right reasons. UTF-16 is a backward compatibility hack for legacy APIs, and UTF-32 is a special-case encoding for easier runtime-manipulation of string data.
- lifthrasiir 5y agoThe author claims that the UTF-8 Everywhere manifesto makes people think that they don't need to implement anything but UTF-8 (which is false and not even what the manifesto actually says). I think that the author's claim itself is also false.
- flohofwoe 5y agoBtw, I'm almost certain that the first code example of expecting the command line arguments as UTF-8 strings won't work on Windows, instead the argument strings will be 8-bit-ASCII encoded with the current system code page. The only way I got this working reliably with the standard main() entry point was to call the Win32 function GetCommandLineW() and then split into args and convert to UTF-8 myself.
- flohofwoe 5y agoPS: found some example code in the github project which seems to take care of the special Windows situation: https://github.com/soasis/text/blob/main/examples/documentation/source/runtime_locale_encoding.cpp https://github.com/soasis/text/blob/main/examples/documentat... Unfortunately this doesn't look so simple anymore.
- nly 5y agoThis code looks like a disaster to me. The issue on Windows is simply that code pages are a per-application setting (or state) and not a system setting. Programs are free to run with, and change between, any "active code page" they want and spit bytes out in that code page. This is all fine and dandy when you're talking to a -A Windows API, since Windows knows the ACP of your application, but it's disaster for I/O and making arbitrary programs talk to one another. Nothing you can do in your code can fix this, it's a contract you have to have with the outside world. On Linux calling setlocale() with a non-UTF-8 locale might not even even work because your /etc/locale.gen file will probably only have a couple of enabled entries, and doing so would be mostly pointless because your terminal and all your other programs are likely using UTF-8. The downside to the Linux approach is that if somebody does actually send you a ISO-8859-1 encoded file you will have to do some conversion. #include <boost/locale.hpp> #include <fstream> #include <iostream> #include <string> int main(int argc, char** argv) { std::ifstream ifs ("test.txt"); std::string str; boost::locale::generator gen; // No dependency on what's in /etc/locale.gen auto loc = gen ("en_US.ISO-8859-1"); ifs.imbue (loc); // Should make formatting input functions work std::getline (ifs, str); str = boost::locale::conv::to_utf<char> (str, loc); for (auto wc: str) { std::cout << sizeof(wc) << ": " << static_cast<uint64_t>(static_cast<unsigned char>(wc)) << "\n"; } }
- malkia 5y agoYou can also https://docs.microsoft.com/en-us/cpp/c-language/using-wmain?view=msvc-160 https://docs.microsoft.com/en-us/cpp/c-language/using-wmain?... e.g. #pragma comment(linker, "/ENTRY:wmainCRTStartup") int wmain(int argc, const wchar_t** argv) { ... // You can use this to actually replace WinMain with proper main (if you use /ENTRY:mainCRTStartup then it's just the good ole main).
- Asooka 5y agoThat's how e.g. Python and Go do it. It's not pretty, but it is reliable. Going the other direction - calling programs with non-ASCII arguments is also a pain. I remember having to manually encode everything in the "mbcs" locale when invoking a subprocess with Python 2. I think Python 3 handles it properly now, but it has been a while.
- MaxBarraclough 5y ago> there are a few popular languages around which jumped on the UNICODE bandwagon too early and are now stuck with UTF-16, but those languages usually also offer simple conversion functions Java does this, right?
- x4e 5y agoJava uses UTF-16 for internally storing strings, however all it’s APIs use the platform default charset which is usually UTF-8. It also uses modified UTF-8 for storing strings inside compiled classes.