5 ms·

> Work in utf-16 when dealing with file paths. Convert back and forth at the point of entry / exit to / from your app. But that's not compatible with other OSe

> Work in utf-16 when dealing with file paths. Convert back and forth at the point of entry / exit to / from your app. But that's not compatible with other OSes, so you can't write native multiplatform C++ code just because windows keeps insisting on their worst-of-both-worlds UTF-16 (worst of both worlds because it's not single value per unicode character like UTF-32, and not ASCII compatible like UTF-8)

GoblinSlayer 3y ago
How do you write code that you can't write one function that opens a file?
beached_whale 3y ago
Store it as std::path
- deleted 3y ago
  [deleted]
plonk 3y ago
Have a few functions taking UTF-8 paths for the filesystem operations you need, have separate versions for Windows and for other systems, and enable one or the other version with the preprocessor?
- bartwe 3y ago
  linux uses utf8 for display of filenames but, the paths themselves allow non-utf8 byte sequences.
  - mmis1000 3y ago
    linux don't display them, the shell (emulator) do. Linux just send the bytes back to userland and let shell interpret them with a proper format to human. And even then. Tons of distro default the global LANG to c for some reason. So utf8 display isn't even working by default.
    - c4mpute 3y ago
      Even better: Each user can have his own locale and charset, and may even change that per program/shell/session. One may save filenames as UTF-8, one as ASCII, one as ISO8859-13, one as EBCDIC. However, the common denominator nowadays is UTF-8, which has been a blessing overall getting rid of most of the aforementioned mess for international multi-user systems. And there is the C.UTF-8 locale which is slowly gaining traction.
  - loeg 3y ago
    It's an oversimplification to say Linux uses UTF-8 for display. Linux just stores bags of bytes and leaves interpretation to userspace. You could store paths in ISO-8859-1 if you wanted. The only special bytes are '\0' and '/'.
    - jeroenhd 3y ago
      Not only could you, this actually happens in practice. Not necessarily ISO-8859-1, but specifically SHIFT-JIS, a Japanese encoding that you will run into if you run old Japanese software. To make things even worse, SHIFT-JIS is almost entirely incompatible with any form of UTF based encoding, and depending on the attempted normalisation you can quickly end up with paths that have been messed up multiple times in a row. I forgot what Japanese emulator I tried to run when I found all of this out, ut sufficed to say I didn't enjoy the experience.
      - Tigress8780 3y ago
        I buy digital Japanese Doujin music on sites like booth.pm, and their provided zip files extracts "beautifully" on Linux if you simply `unzip` them.
        jeroenhd 3y ago
        Lots of Japanese products are switching or have switched to UTF-8, so I have no doubt that modern ZIP files will extract without a problem.
        charcircuit 3y ago
        Don't you mean `unzip -O shift-jis` them?
    - AshamedCaptain 3y ago
      Except that Linux does support several filesystems that do claim to store the filenames in a specific encoding and therefore the kernel must do conversion. Mostly Windows FSes, but nowadays case-insensitive ext4 also applies.
- shortrounddev2 3y ago
  This is how 99% of cross-platform C++ works. Very few C++ is actually "cross platform" in that it compiles the same functions for different platforms
  - deleted 3y ago
    [deleted]
bruce511 3y ago
Not to be that guy, but utf-32 is fixed length code point, not fixed length character. Characters can consist of multiple code points. I agree that utf-8 is preferable to utf-16 though in most cases where the root script is English/Latin.
- usrusr 3y ago
  Makes me wonder how bad utf-8 in off-latin environments really is: what's the fraction of strings that is "language content", and how much is colons, quotes, decimal numbers and the like? What fraction of coding cultures embraces home locale script for identifiers, and how many continue on the trajectory set in motion by pre-unicode programming languages? My guess would be that utf-8 is still quite bad, but I wouldn't be entirely surprised if actual numbers turned out considerably less bad than one might expect. And how much of the "utf-8 tax" remains if the actual format of most data at rest and in transit isn't json but gzipped json?
  - vlovich123 3y ago
    Experiments I recall from a while back from browsers and JDK indicated the vast majority of memory is consumed by latin1 strings (even for apps in Asia). Of course, those experiments I recall are something like 20 years old and likely biased (websites were smaller, content was still latin1 dominated). It would be interesting to know the results from Chinese apps and websites today. For websites, the markup used the most space and that stuff remains latin1. For apps I don’t know. But also, text size in memory is typically a joke because the expensive bit is all the multimedia that accompanies text these days. Remember, 100mib is 25million characters even in utf32. That’s 1 hour or so of high quality audio or a few seconds of video. In terms of compression, the smaller initial file will typically compress better so not sure what you mean by “utf-8” tax. UTF8 is not slower to process than utf32 as far as I’m aware except for one small corner case of random codepoint access (but then in UTF text processing, you really shouldn’t be doing that I think but I’m not a utf expert ).
- 3PS 3y ago
  > Characters can consist of multiple code points. I enjoy being "that guy", and so, to be unfathomably pedantic, I will point out that the Unicode standard actually uses the word "character" to refer to _encoded characters_, i.e. the mapping from a code point to an abstract character [0]. So 1 code point = 1 character for Unicode. Of course, in real life, character is a silly made-up word that means whatever you think it means and using it to refer to an extended grapheme cluster or glyph is probably closer to how people really think of them anyway. [0]: See section 3.4 of https://www.unicode.org/versions/Unicode15.0.0/UnicodeStandard-15.0.pdf https://www.unicode.org/versions/Unicode15.0.0/UnicodeStanda...
shortrounddev2 3y ago
If you're calling Win32 functions it's already not compatible with other OSes... If you want to use cross platform code with files, use the C or C++ standard library's file handling functions. C++ also supports wstring, but its implementation is dependent on the platform for god knows what reason. utf-16 is also supported in C11: https://en.wikipedia.org/wiki/C11_(C_standard_revision)#Changes_from_C99 https://en.wikipedia.org/wiki/C11_(C_standard_revision)#Chan... > Improved Unicode support based on the C Unicode Technical Report ISO/IEC TR 19769:2004 (char16_t and char32_t types for storing UTF-16/UTF-32 encoded data, including conversion functions in <uchar.h> and the corresponding u and U string literal prefixes, as well as the u8 prefix for UTF-8 encoded literals). You can use uchar.h for converting instead of the Win32 functions. Here's a helper I wrote to wrap these string types in std::string and std::wstring: bool mbStrToWideChar(string const& in, wstring& out) { bool status = true; i32 bufferSize = MultiByteToWideChar( CP_ACP, 0, in.c_str(), -1, nullptr, 0 ); WCHAR* buffer = new WCHAR[bufferSize]; i32 conversionResult = MultiByteToWideChar( CP_ACP, 0, in.c_str(), -1, buffer, bufferSize ); if (!conversionResult) { cerr << "Failed to convert ANSI string [" << in << "] to Wide string!" << endl; status = false; goto Delete; } out = wstring(buffer); Delete: if (buffer) { delete[] buffer; } return status; } and bool wideStrToMbStr(wstring const& in, string& out) { // Get buffer size i32 bufferSize = WideCharToMultiByte( CP_UTF8, 0, in.c_str(), (i32)in.size(), nullptr, 0, nullptr, nullptr ); // Set outstring to new buffer size out = string(bufferSize, '\0'); //Convert i32 res = WideCharToMultiByte( CP_UTF8, 0, in.c_str(), (i32)in.size(), &out[0], (i32)out.size(), nullptr, nullptr ); if (!res) { wcerr << L"Failed to convert Wide string [" << in << L"] to ANSI string!" << endl; return false; } return true; }
fsloth 3y ago
Our native multiplatform c++ code uses wstring (win, mac, android, wasm). It’s not the worst option. AFAIK There is no 100% method to have single codebase tackle the differences in platforms. My recipe for keeping sane is to have single implementation for storage (wstring), and then very explicit points of converting paths from ‘platform’ to program internal. Platform differences unfortunately go way deeper than just using UTF-16. Lots of gnarly details. No platform is worse than another - they just have different kinds of good and bad in different places.
gavinhoward 3y ago
Speaking as someone who did what GP suggested, it can be cross-platform; I just abstracted the string to path and path to string conversions, as well as any path operations that are different across platforms. The first thing to realize to make it work is that paths are not strings, so you actually don't want to treat them like they are.

GoblinSlayer 3y ago

How do you write code that you can't write one function that opens a file?

beached_whale 3y ago

Store it as std::path

deleted 3y ago
[deleted]

plonk 3y ago

Have a few functions taking UTF-8 paths for the filesystem operations you need, have separate versions for Windows and for other systems, and enable one or the other version with the preprocessor?

bartwe 3y ago
linux uses utf8 for display of filenames but, the paths themselves allow non-utf8 byte sequences.
- mmis1000 3y ago
  linux don't display them, the shell (emulator) do. Linux just send the bytes back to userland and let shell interpret them with a proper format to human. And even then. Tons of distro default the global LANG to c for some reason. So utf8 display isn't even working by default.
  - c4mpute 3y ago
    Even better: Each user can have his own locale and charset, and may even change that per program/shell/session. One may save filenames as UTF-8, one as ASCII, one as ISO8859-13, one as EBCDIC. However, the common denominator nowadays is UTF-8, which has been a blessing overall getting rid of most of the aforementioned mess for international multi-user systems. And there is the C.UTF-8 locale which is slowly gaining traction.
- loeg 3y ago
  It's an oversimplification to say Linux uses UTF-8 for display. Linux just stores bags of bytes and leaves interpretation to userspace. You could store paths in ISO-8859-1 if you wanted. The only special bytes are '\0' and '/'.
  - jeroenhd 3y ago
    Not only could you, this actually happens in practice. Not necessarily ISO-8859-1, but specifically SHIFT-JIS, a Japanese encoding that you will run into if you run old Japanese software. To make things even worse, SHIFT-JIS is almost entirely incompatible with any form of UTF based encoding, and depending on the attempted normalisation you can quickly end up with paths that have been messed up multiple times in a row. I forgot what Japanese emulator I tried to run when I found all of this out, ut sufficed to say I didn't enjoy the experience.
    - Tigress8780 3y ago
      I buy digital Japanese Doujin music on sites like booth.pm, and their provided zip files extracts "beautifully" on Linux if you simply `unzip` them.
      - jeroenhd 3y ago
        Lots of Japanese products are switching or have switched to UTF-8, so I have no doubt that modern ZIP files will extract without a problem.
      - charcircuit 3y ago
        Don't you mean `unzip -O shift-jis` them?
  - AshamedCaptain 3y ago
    Except that Linux does support several filesystems that do claim to store the filenames in a specific encoding and therefore the kernel must do conversion. Mostly Windows FSes, but nowadays case-insensitive ext4 also applies.
shortrounddev2 3y ago
This is how 99% of cross-platform C++ works. Very few C++ is actually "cross platform" in that it compiles the same functions for different platforms
- deleted 3y ago
  [deleted]

bruce511 3y ago

Not to be that guy, but utf-32 is fixed length code point, not fixed length character. Characters can consist of multiple code points. I agree that utf-8 is preferable to utf-16 though in most cases where the root script is English/Latin.

usrusr 3y ago
Makes me wonder how bad utf-8 in off-latin environments really is: what's the fraction of strings that is "language content", and how much is colons, quotes, decimal numbers and the like? What fraction of coding cultures embraces home locale script for identifiers, and how many continue on the trajectory set in motion by pre-unicode programming languages? My guess would be that utf-8 is still quite bad, but I wouldn't be entirely surprised if actual numbers turned out considerably less bad than one might expect. And how much of the "utf-8 tax" remains if the actual format of most data at rest and in transit isn't json but gzipped json?
- vlovich123 3y ago
  Experiments I recall from a while back from browsers and JDK indicated the vast majority of memory is consumed by latin1 strings (even for apps in Asia). Of course, those experiments I recall are something like 20 years old and likely biased (websites were smaller, content was still latin1 dominated). It would be interesting to know the results from Chinese apps and websites today. For websites, the markup used the most space and that stuff remains latin1. For apps I don’t know. But also, text size in memory is typically a joke because the expensive bit is all the multimedia that accompanies text these days. Remember, 100mib is 25million characters even in utf32. That’s 1 hour or so of high quality audio or a few seconds of video. In terms of compression, the smaller initial file will typically compress better so not sure what you mean by “utf-8” tax. UTF8 is not slower to process than utf32 as far as I’m aware except for one small corner case of random codepoint access (but then in UTF text processing, you really shouldn’t be doing that I think but I’m not a utf expert ).
3PS 3y ago
> Characters can consist of multiple code points. I enjoy being "that guy", and so, to be unfathomably pedantic, I will point out that the Unicode standard actually uses the word "character" to refer to _encoded characters_, i.e. the mapping from a code point to an abstract character [0]. So 1 code point = 1 character for Unicode. Of course, in real life, character is a silly made-up word that means whatever you think it means and using it to refer to an extended grapheme cluster or glyph is probably closer to how people really think of them anyway. [0]: See section 3.4 of https://www.unicode.org/versions/Unicode15.0.0/UnicodeStandard-15.0.pdf https://www.unicode.org/versions/Unicode15.0.0/UnicodeStanda...

shortrounddev2 3y ago

If you're calling Win32 functions it's already not compatible with other OSes... If you want to use cross platform code with files, use the C or C++ standard library's file handling functions. C++ also supports wstring, but its implementation is dependent on the platform for god knows what reason. utf-16 is also supported in C11: https://en.wikipedia.org/wiki/C11_(C_standard_revision)#Changes_from_C99 https://en.wikipedia.org/wiki/C11_(C_standard_revision)#Chan... > Improved Unicode support based on the C Unicode Technical Report ISO/IEC TR 19769:2004 (char16_t and char32_t types for storing UTF-16/UTF-32 encoded data, including conversion functions in <uchar.h> and the corresponding u and U string literal prefixes, as well as the u8 prefix for UTF-8 encoded literals). You can use uchar.h for converting instead of the Win32 functions. Here's a helper I wrote to wrap these string types in std::string and std::wstring: bool mbStrToWideChar(string const& in, wstring& out) { bool status = true; i32 bufferSize = MultiByteToWideChar( CP_ACP, 0, in.c_str(), -1, nullptr, 0 ); WCHAR* buffer = new WCHAR[bufferSize]; i32 conversionResult = MultiByteToWideChar( CP_ACP, 0, in.c_str(), -1, buffer, bufferSize ); if (!conversionResult) { cerr << "Failed to convert ANSI string [" << in << "] to Wide string!" << endl; status = false; goto Delete; } out = wstring(buffer); Delete: if (buffer) { delete[] buffer; } return status; } and bool wideStrToMbStr(wstring const& in, string& out) { // Get buffer size i32 bufferSize = WideCharToMultiByte( CP_UTF8, 0, in.c_str(), (i32)in.size(), nullptr, 0, nullptr, nullptr ); // Set outstring to new buffer size out = string(bufferSize, '\0'); //Convert i32 res = WideCharToMultiByte( CP_UTF8, 0, in.c_str(), (i32)in.size(), &out[0], (i32)out.size(), nullptr, nullptr ); if (!res) { wcerr << L"Failed to convert Wide string [" << in << L"] to ANSI string!" << endl; return false; } return true; }

fsloth 3y ago

Our native multiplatform c++ code uses wstring (win, mac, android, wasm). It’s not the worst option. AFAIK There is no 100% method to have single codebase tackle the differences in platforms. My recipe for keeping sane is to have single implementation for storage (wstring), and then very explicit points of converting paths from ‘platform’ to program internal. Platform differences unfortunately go way deeper than just using UTF-16. Lots of gnarly details. No platform is worse than another - they just have different kinds of good and bad in different places.

gavinhoward 3y ago

Speaking as someone who did what GP suggested, it can be cross-platform; I just abstracted the string to path and path to string conversions, as well as any path operations that are different across platforms. The first thing to realize to make it work is that paths are not strings, so you actually don't want to treat them like they are.