6 ms·

Alternative sane approach: - Forget about long paths unless you really need to care. You need to care if you're accepting a file path from another app, for exa

• Aardwolf 3y ago
  > Work in utf-16 when dealing with file paths. Convert back and forth at the point of entry / exit to / from your app. But that's not compatible with other OSes, so you can't write native multiplatform C++ code just because windows keeps insisting on their worst-of-both-worlds UTF-16 (worst of both worlds because it's not single value per unicode character like UTF-32, and not ASCII compatible like UTF-8)
  • GoblinSlayer 3y ago
    How do you write code that you can't write one function that opens a file?
  • beached_whale 3y ago
    Store it as std::path
    • deleted 3y ago
      [deleted]
  • plonk 3y ago
    Have a few functions taking UTF-8 paths for the filesystem operations you need, have separate versions for Windows and for other systems, and enable one or the other version with the preprocessor?
    • bartwe 3y ago
      linux uses utf8 for display of filenames but, the paths themselves allow non-utf8 byte sequences.
      • mmis1000 3y ago
        linux don't display them, the shell (emulator) do. Linux just send the bytes back to userland and let shell interpret them with a proper format to human. And even then. Tons of distro default the global LANG to c for some reason. So utf8 display isn't even working by default.
        • c4mpute 3y ago
          Even better: Each user can have his own locale and charset, and may even change that per program/shell/session. One may save filenames as UTF-8, one as ASCII, one as ISO8859-13, one as EBCDIC. However, the common denominator nowadays is UTF-8, which has been a blessing overall getting rid of most of the aforementioned mess for international multi-user systems. And there is the C.UTF-8 locale which is slowly gaining traction.
      • loeg 3y ago
        It's an oversimplification to say Linux uses UTF-8 for display. Linux just stores bags of bytes and leaves interpretation to userspace. You could store paths in ISO-8859-1 if you wanted. The only special bytes are '\0' and '/'.
        • jeroenhd 3y ago
          Not only could you, this actually happens in practice. Not necessarily ISO-8859-1, but specifically SHIFT-JIS, a Japanese encoding that you will run into if you run old Japanese software. To make things even worse, SHIFT-JIS is almost entirely incompatible with any form of UTF based encoding, and depending on the attempted normalisation you can quickly end up with paths that have been messed up multiple times in a row. I forgot what Japanese emulator I tried to run when I found all of this out, ut sufficed to say I didn't enjoy the experience.
  • bruce511 3y ago
    Not to be that guy, but utf-32 is fixed length code point, not fixed length character. Characters can consist of multiple code points. I agree that utf-8 is preferable to utf-16 though in most cases where the root script is English/Latin.
    • usrusr 3y ago
      Makes me wonder how bad utf-8 in off-latin environments really is: what's the fraction of strings that is "language content", and how much is colons, quotes, decimal numbers and the like? What fraction of coding cultures embraces home locale script for identifiers, and how many continue on the trajectory set in motion by pre-unicode programming languages? My guess would be that utf-8 is still quite bad, but I wouldn't be entirely surprised if actual numbers turned out considerably less bad than one might expect. And how much of the "utf-8 tax" remains if the actual format of most data at rest and in transit isn't json but gzipped json?
      • vlovich123 3y ago
        Experiments I recall from a while back from browsers and JDK indicated the vast majority of memory is consumed by latin1 strings (even for apps in Asia). Of course, those experiments I recall are something like 20 years old and likely biased (websites were smaller, content was still latin1 dominated). It would be interesting to know the results from Chinese apps and websites today. For websites, the markup used the most space and that stuff remains latin1. For apps I don’t know. But also, text size in memory is typically a joke because the expensive bit is all the multimedia that accompanies text these days. Remember, 100mib is 25million characters even in utf32. That’s 1 hour or so of high quality audio or a few seconds of video. In terms of compression, the smaller initial file will typically compress better so not sure what you mean by “utf-8” tax. UTF8 is not slower to process than utf32 as far as I’m aware except for one small corner case of random codepoint access (but then in UTF text processing, you really shouldn’t be doing that I think but I’m not a utf expert ).
    • 3PS 3y ago
      > Characters can consist of multiple code points. I enjoy being "that guy", and so, to be unfathomably pedantic, I will point out that the Unicode standard actually uses the word "character" to refer to _encoded characters_, i.e. the mapping from a code point to an abstract character [0]. So 1 code point = 1 character for Unicode. Of course, in real life, character is a silly made-up word that means whatever you think it means and using it to refer to an extended grapheme cluster or glyph is probably closer to how people really think of them anyway. [0]: See section 3.4 of https://www.unicode.org/versions/Unicode15.0.0/UnicodeStandard-15.0.pdf https://www.unicode.org/versions/Unicode15.0.0/UnicodeStanda...
  • shortrounddev2 3y ago
    If you're calling Win32 functions it's already not compatible with other OSes... If you want to use cross platform code with files, use the C or C++ standard library's file handling functions. C++ also supports wstring, but its implementation is dependent on the platform for god knows what reason. utf-16 is also supported in C11: https://en.wikipedia.org/wiki/C11_(C_standard_revision)#Changes_from_C99 https://en.wikipedia.org/wiki/C11_(C_standard_revision)#Chan... > Improved Unicode support based on the C Unicode Technical Report ISO/IEC TR 19769:2004 (char16_t and char32_t types for storing UTF-16/UTF-32 encoded data, including conversion functions in <uchar.h> and the corresponding u and U string literal prefixes, as well as the u8 prefix for UTF-8 encoded literals). You can use uchar.h for converting instead of the Win32 functions. Here's a helper I wrote to wrap these string types in std::string and std::wstring: bool mbStrToWideChar(string const& in, wstring& out) { bool status = true; i32 bufferSize = MultiByteToWideChar( CP_ACP, 0, in.c_str(), -1, nullptr, 0 ); WCHAR* buffer = new WCHAR[bufferSize]; i32 conversionResult = MultiByteToWideChar( CP_ACP, 0, in.c_str(), -1, buffer, bufferSize ); if (!conversionResult) { cerr << "Failed to convert ANSI string [" << in << "] to Wide string!" << endl; status = false; goto Delete; } out = wstring(buffer); Delete: if (buffer) { delete[] buffer; } return status; } and bool wideStrToMbStr(wstring const& in, string& out) { // Get buffer size i32 bufferSize = WideCharToMultiByte( CP_UTF8, 0, in.c_str(), (i32)in.size(), nullptr, 0, nullptr, nullptr ); // Set outstring to new buffer size out = string(bufferSize, '\0'); //Convert i32 res = WideCharToMultiByte( CP_UTF8, 0, in.c_str(), (i32)in.size(), &out[0], (i32)out.size(), nullptr, nullptr ); if (!res) { wcerr << L"Failed to convert Wide string [" << in << L"] to ANSI string!" << endl; return false; } return true; }
  • fsloth 3y ago
    Our native multiplatform c++ code uses wstring (win, mac, android, wasm). It’s not the worst option. AFAIK There is no 100% method to have single codebase tackle the differences in platforms. My recipe for keeping sane is to have single implementation for storage (wstring), and then very explicit points of converting paths from ‘platform’ to program internal. Platform differences unfortunately go way deeper than just using UTF-16. Lots of gnarly details. No platform is worse than another - they just have different kinds of good and bad in different places.
  • gavinhoward 3y ago
    Speaking as someone who did what GP suggested, it can be cross-platform; I just abstracted the string to path and path to string conversions, as well as any path operations that are different across platforms. The first thing to realize to make it work is that paths are not strings, so you actually don't want to treat them like they are.
• actionfromafar 3y ago
  If you need to consider slightly older Windows versions, it's WTF-16, not UTF-16.
  • loeg 3y ago
    Isn't it UCS-2 rather than any form of UTF-16?
    • protomolecule 3y ago
      Not after Windows 2000.
  • chrismorgan 3y ago
    There’s no such thing as WTF-16—or if you want to use that label for “potentially-ill-formed UTF-16”, then everything that says it does UTF-16 is actually WTF-16, because no one validates UTF-16.
    • actionfromafar 3y ago
      Yes and no - some programs or systems of programs regularly transform between UTF-8 and UTF-16. A file path in Windows can be "valid" (or whatever we wanna call it) and still not convert cleanly to UTF-8. Edit to clarify: many systems which deal with non-path UTF-16 strings, nowadays are so intertwined with the UTF-8 world, that they get the validation "for free" so to speak, here and there, wherever they interact with the rest of the world. Paths though, remain a common little corner case, working fine until it suddenly doesn't, for some weird combination.
      • chrismorgan 3y ago
        That’s just potentially-ill-formed UTF-16, which is what all UTF-16 is in practice. The novel thing about UTF-8 environments is that they (or most of ’em, anyway) finally actually validate stuff.
• smarkov 3y ago
  > Don't change random registry settings or code pages on the user's computer, that's mad. A million times this. Multiple Valve games set the microphone volume in Windows to 100% whenever you launch the game, which, for my setup, makes me sound unbearably loud. Unless your app has a valid reason to change my settings, like if I've set it up to do that on a schedule or with some condition, sure, otherwise don't mess with my settings.
  • formerly_proven 3y ago
    Story time When the USB audio device class was created [1], it had support for AVR-like audio controls (volume, balance, tone etc.), and, like would be reasonable, made all that work with decibels. But, as far as I know, that part of the standard got ignored by basically everyone, including some of whom wrote it (Microsoft), by presenting e.g. the on-device volume control as a 0-100 scale, like Windows and ALSA still do. Except... some hardware devices actually implemented the spec, including the PCM29xx family of audio codecs. These, and clones of them, have been used in quite a few USB audio cards and some very low-end audio interface over the decades. And for those, "100" in Windows means something like "+30 dBfs" [2] - so you'll get extreme distortion with basically any signal. [1] 1.0 is actually impressively complex and basically models the entire stack of audio devices someone could have possibly had in their TV cabinet in the 1995, including stuff like dolby and surround decoders. [2] This is defined around 5.2.2.2.3 and/or 5.2.2.4.3.2 in https://www.usb.org/sites/default/files/audio10.pdf https://www.usb.org/sites/default/files/audio10.pdf
• gavinhoward 3y ago
  This is exactly the approach I took in my own cross-platform "standard" library. It works. In particular, paths must be a different type than strings. As a bonus: it becomes easier to implement niceties like path concatenation and realpath.
• cbsmith 3y ago
  You can also just over-allocate to avoid having to call it twice, or do some really basic heuristics to fast path the call (it's very specific UTF-8 sequences that need more than 2 bytes in UTF-16).