4 ms·
Btw, I'm almost certain that the first code example of expecting the command line arguments as UTF-8 strings won't work on Windows, instead the argument strings
by flohofwoe 5y ago
Btw, I'm almost certain that the first code example of expecting the command line arguments as UTF-8 strings won't work on Windows, instead the argument strings will be 8-bit-ASCII encoded with the current system code page. The only way I got this working reliably with the standard main() entry point was to call the Win32 function GetCommandLineW() and then split into args and convert to UTF-8 myself.
- flohofwoe 5y agoPS: found some example code in the github project which seems to take care of the special Windows situation: https://github.com/soasis/text/blob/main/examples/documentation/source/runtime_locale_encoding.cpp https://github.com/soasis/text/blob/main/examples/documentat... Unfortunately this doesn't look so simple anymore.
- nly 5y agoThis code looks like a disaster to me. The issue on Windows is simply that code pages are a per-application setting (or state) and not a system setting. Programs are free to run with, and change between, any "active code page" they want and spit bytes out in that code page. This is all fine and dandy when you're talking to a -A Windows API, since Windows knows the ACP of your application, but it's disaster for I/O and making arbitrary programs talk to one another. Nothing you can do in your code can fix this, it's a contract you have to have with the outside world. On Linux calling setlocale() with a non-UTF-8 locale might not even even work because your /etc/locale.gen file will probably only have a couple of enabled entries, and doing so would be mostly pointless because your terminal and all your other programs are likely using UTF-8. The downside to the Linux approach is that if somebody does actually send you a ISO-8859-1 encoded file you will have to do some conversion. #include <boost/locale.hpp> #include <fstream> #include <iostream> #include <string> int main(int argc, char** argv) { std::ifstream ifs ("test.txt"); std::string str; boost::locale::generator gen; // No dependency on what's in /etc/locale.gen auto loc = gen ("en_US.ISO-8859-1"); ifs.imbue (loc); // Should make formatting input functions work std::getline (ifs, str); str = boost::locale::conv::to_utf<char> (str, loc); for (auto wc: str) { std::cout << sizeof(wc) << ": " << static_cast<uint64_t>(static_cast<unsigned char>(wc)) << "\n"; } }
- malkia 5y agoYou can also https://docs.microsoft.com/en-us/cpp/c-language/using-wmain?view=msvc-160 https://docs.microsoft.com/en-us/cpp/c-language/using-wmain?... e.g. #pragma comment(linker, "/ENTRY:wmainCRTStartup") int wmain(int argc, const wchar_t** argv) { ... // You can use this to actually replace WinMain with proper main (if you use /ENTRY:mainCRTStartup then it's just the good ole main).
- Asooka 5y agoThat's how e.g. Python and Go do it. It's not pretty, but it is reliable. Going the other direction - calling programs with non-ASCII arguments is also a pain. I remember having to manually encode everything in the "mbcs" locale when invoking a subprocess with Python 2. I think Python 3 handles it properly now, but it has been a while.