12 ms·

What it takes to pass a file path to a Windows API in C++

153 points by AshleysBrain 3y ago

Lockal 3y ago
The article should be called "What it takes to do anything Windows API in C++". For example, this Unicode issue is applicable to almost every API there. And it is not as simple as "call MultiByteToWideChar twice". Microsoft did not support UTF-16, they UCS-2, which they called "Unicode", but in fact back then it was just a wider ASCII. The worst thing about UCS-2 is that it does not support roundtrip. For example, imagine, that you wrote your Enterprise MS Tech Contoso Ltd.(R) authentication system, where user registers in UTF-8 on webpage, then on some layer it checks that user with the same name (in UTF-8 modern DB) does not exist, then it writes user basic information in UCS-2 encoded legacy "Enterprise DB". Aaand... Voila, Evil User overrides data of other user, because UTF-8 can't losslessly represent arbitrary sequences of 16-bit code units (should have used https://simonsapin.github.io/wtf-8/ https://simonsapin.github.io/wtf-8/ instead, but your Enterprise tech was destroyed).
- colejohnson66 3y ago
  They called it “Unicode” because, at the time, thanks to Han Unification, UCS-2 was widely believed to be the one and only encoding that anyone would ever need. Two bytes per codepoint. No more, and no less. C#/.NET even inherit this misnomer and call UTF-16LE “Unicode”: System.Text.Encoding.UnicodeEncoding
Cold_Miserable 3y ago
Windows uses unicode because NTFS uses unicode instead of efficient ascii.
- becurious 3y ago
  When it was designed there was no UTF-8. You had to know the code point; and file systems like FAT did not encode the code point, so file names would be interpreted by whatever the current code point was. So having a file system and other APIs support Unicode natively made a lot of sense at the time; it simplifies many things including deploying into organizations that have global users. This is now over 30 years ago. And the A vs the W suffixed APIs were not simple ascii; they were multi-byte, so to work with strings you still had to know the code page. Obviously UTF-8 is much simpler for that now, but that wasn’t the world when this was designed.
londons_explore 3y ago
Tl:DR: there are many different ways to send file paths to system library functions. That's mostly because of Windows' long legacy support history. Don't worry about it - just pick one and go.
- c4mpute 3y ago
  No. Because the API is fragmented into old, not-so-old and new parts, some higher-, some lower-level[0]. You need things from all of them, the new API isn't complete enough and has warts in places where using one of the older ones is better/easier/safer. So in the end you will have to use all the ways, and know when to use which one. All this can of course be fixed with another layer of indirection. But if you ever wanted to know why Windows file APIs and filesystems are so dog-slow, here is (part of) your answer. [0] e.g. there are lower level APIs that can create and access files and filenames that higher level APIs will filter out or disallow.
  - sn_master 3y ago
    > new API isn't complete enough and has warts in places where using one of the older ones is better/easier/safer Describes Windows Forms vs everything else MS has been trying to do in the past 15 years.
  - HeckFeck 3y ago
    I feel inclined to bypass all that Win32 gubbins and talk to the Kernel directly. Back in my day we just walked into the office of the man in charge and shook hands, we struck deals like real men. If we ask ask the NT kernel politely in a language it understands, we will get our filepath. Right? http://undocumented.ntinternals.net/index.html?page=UserMode%2FUndocumented%20Functions%2FNT%20Objects%2FFile%2FNtOpenFile.html http://undocumented.ntinternals.net/index.html?page=UserMode...
    - huhtenberg 3y ago
      NT API is now documented by Microsoft. The coverage is not complete, but enough to be useful. E.g. https://learn.microsoft.com/en-us/windows/win32/api/winternl/nf-winternl-ntopenfile https://learn.microsoft.com/en-us/windows/win32/api/winternl....
Affric 3y ago
So, a lot of what people think of when they think of OS is UI based (GUI/TUI/CLI) but kernel APIs are the real bread and butter and this is where the UNIX philosophy really shines. Passing a string as a file name in C++ with macOS or with Linux in my experience was simple. The permitted length of using ASCII characters is about 4 times as long (may god have mercy on your soul). I am not here to shit on Windows but the Windows devs clearly have a very different set of priorities (e.g. backwards compatibility) than the other breeds of modern OS devs. I guess to a large extent we all expect to be in the browser (gross) in some number of years but Windows seems so much harder from the perspective of someone that has programmed for Unixen and studied Windows as an OS.
- cerved 3y ago
  It's not all backwards comparability. I'm willing to bet that some (a large part?) is just sloppy software development. SQL Server (2017?) breaks if you update it on a UTF-8 Windows because it runs a TSQL that doesn't work with that code page. That script is a mess. Some of it is indented using tabs, some space. Trailing whitespace. Yuck
  - nikanj 3y ago
    My hot take: Code quality is not measured by formatting issues, but by error resilience and number of actual bugs. Much of modern linting and commit hooking is dedicated to checking whitespace placement, variable naming and function lengths but the well-formatted newly rewritten code is still buggy as hell - it just looks pretty
    - cerved 3y ago
      I've often found that people that don't care about white-space also don't care that much about other aspects of code quality The inverse may not have the same correlation, since it can be automated.
    - edejong 3y ago
      Formatting doesn’t remove bugs, but it’ll help you detect them. Linted code helps you scan the code faster and provides valuable pattern recognition, allowing us to detect common mistakes. There have been numerous bugs caused by incorrect code formatting, most notably an SSL security bug from 2014: https://dwheeler.com/essays/apple-goto-fail.html https://dwheeler.com/essays/apple-goto-fail.html Another reason for formatting is the “minimal diff” paradigm. If a formatting rule would not be followed, in the next commit hitting this code, the format would also be affected, causing a larger diff than necessary. There are other reasons for simple format linting, but the reasons above are the most profound. Lastly, formatting is part of a range of static code analysis tools. Generally, formatting inconsistencies are the easiest to detect and resolve, as opposed to more sophisticated tools.
tester756 3y ago
Which is interesting because for example C#/.NET base class library provides very high quality APIs They even wrote a book about API design "Framework Design Guidelines: Conventions, Idioms, and Patterns for Reusable .Net Libraries"
- toyg 3y ago
  C# was effectively built from scratch with the benefit of hindsight from both C++ and Java. Decent practices should be considered been the bare minimum, surely.
h2odragon 3y ago
It's possible to get a file into a windows filesystem whose name contains "*" or ":" characters. Windows reacts quite poorly when it find those in file names; but only in certain APIs, others pass them without concern. Makes for fun explosions. The "260 character limit" is a good one; mostly seen that one in "I can do anything to this file but delete it!" complaints.
- quickthrower2 3y ago
  I had the can’t delete a file problem the other day. Got around it my moving it’s parent folder to a less nested location
- huhtenberg 3y ago
  It's also possible to create files whose names start (or contain) a NUL character. Doing so will trigger every single antivirus, but it's doable nonetheless and messes a lot of things very thoroughly.
  - Someone1234 3y ago
    Also, possible in the Windows Registry, which no normal UI tools (inc. Reg Edit) could remove.
    - 13of40 3y ago
      A similar quirk of the registry API is that it has a simplified set of methods that don't specify the data type of the value, so you can have an application that expects to read a null terminated string value, but substitute a binary value with no null termination. If the app doesn't handle that right, it's buffer overrun time. Usually that's foot-shooting, but sometimes you can do that in HKCU as a low privileged user to a value that gets read by a higher privileged process and causes it to "misbehave".
  - ale42 3y ago
    You can also create a file (or even better a folder) named CON or COM4. Good luck doing anything with it using the Explorer. (15 years ago I've seen some malware using that to make removal difficult)
wzdd 3y ago
Alternative sane approach: - Forget about long paths unless you really need to care. You need to care if you're accepting a file path from another app, for example. You don't need to care if you're using your own files in your install directory; too much of Windows doesn't support long files (including explorer, as mentioned, not to mention third party apps), so it's very unlikely that you'll need to deal with them if you're just handling your own data. - Work in utf-16 when dealing with file paths. Convert back and forth at the point of entry / exit to / from your app. It's unsurprising that you need to call the conversion function twice in order to find out how much memory to allocate, that's just how that works. Don't change random registry settings or code pages on the user's computer, that's mad.
- Aardwolf 3y ago
  > Work in utf-16 when dealing with file paths. Convert back and forth at the point of entry / exit to / from your app. But that's not compatible with other OSes, so you can't write native multiplatform C++ code just because windows keeps insisting on their worst-of-both-worlds UTF-16 (worst of both worlds because it's not single value per unicode character like UTF-32, and not ASCII compatible like UTF-8)
  - GoblinSlayer 3y ago
    How do you write code that you can't write one function that opens a file?
  - beached_whale 3y ago
    Store it as std::path
    - deleted 3y ago
      [deleted]
  - plonk 3y ago
    Have a few functions taking UTF-8 paths for the filesystem operations you need, have separate versions for Windows and for other systems, and enable one or the other version with the preprocessor?
sn_master 3y ago
Now try doing this for Windows 98 or older, specially the non-ASCII file paths part (which was a common use case outside of the US).
- ormax3 3y ago
  https://en.wikipedia.org/wiki/Microsoft_Layer_for_Unicode https://en.wikipedia.org/wiki/Microsoft_Layer_for_Unicode
  - sn_master 3y ago
    > The MSLU was announced in March 2001, and was first made available as a compatibility layer for Unicode-supporting code written for the then-new Windows XP RC1 in the July 2001 edition of Microsoft's Platform SDK. People had to deal with local language file names since Windows 3.1 at least and it became very common with Windows 95. Good luck if you wanted to deal with files named in more than 1 non-ASCII language (very common in Europe or the Middle East).
ormax3 3y ago
Like explained in https://utf8everywhere.org/#windows https://utf8everywhere.org/#windows , you can write simple wrapper functions `narrow`/`widen` used when you are about to call window api functions. ::SetWindowTextW(widen(someStdString).c_str()); Implementation is straightforward, relying on `WideCharToMultiByte`/`MultiByteToWideChar` to do the conversion: https://github.com/neacsum/utf8/blob/master/src/utf8.cpp https://github.com/neacsum/utf8/blob/master/src/utf8.cpp
- huhtenberg 3y ago
  You are missing OP's point - this still costs you 2 extra calls. If this cost really matters (and practically speaking it never does), then, as the other commenter said, the correct solution is to just use OS-native encoding for all file system paths and names used by the program, hidden behind an abstraction layer if needs be. UTF16 for Windows, UTF8 elsewhere.
  - ormax3 3y ago
    The above manifesto makes the argument to use UTF-8 *everywhere*, even on windows where the internal representation is not native utf8. The conversion overhead is really negligible: https://utf8everywhere.org/#faq.cvt.perf https://utf8everywhere.org/#faq.cvt.perf (note: the two api calls per conversion is because how those specific functions work, first call to get the size to allocate, second to do the actual conversion, but you can always use another library in the implementation for the utf8<->utf16 conversion that might be more optimized than those windows api functions)
    - donatj 3y ago
      Especially negligible versus the trip to the file system you are setting up for.
      - Tempest1981 3y ago
        And AVX-512 can help: https://lemire.me/blog/2023/09/13/transcoding-unicode-strings-at-crazy-speeds-with-avx-512/ https://lemire.me/blog/2023/09/13/transcoding-unicode-string...
Tigress8780 3y ago
Talking about how strange the current state of Windows API is, there are actually 3 different APIs that can move the mouse cursor: SetCursorPos [1], SendInput [2], and mouse_event [3]. Applications might react differently to input events generated by these APIs. For example, in Windows 11's display settings, the position of screens can not be dragged if input events are coming from SetCursorPos, but works fine if using SendInput. Microsoft's own PowerToys uses a mixture of both under certain (complex) conditions, but I never found out the actual difference between them. I was writing an application that sends mouse input from a Linux machine to a Windows one (similar to Synergy), and I originally received mouse movement events that are accelerated (with user or system-defined acceleration factors), and I found out that there are no Windows API (three of them) that accepts relative movements without further acceleration (i.e. Windows will always apply further acceleration, making the mouse hard to use). I ended up directly hook into evdev to get raw mouse movements and let Windows accelerate them. [1] https://learn.microsoft.com/en-us/windows/win32/api/winuser/nf-winuser-setcursorpos https://learn.microsoft.com/en-us/windows/win32/api/winuser/... [2] https://learn.microsoft.com/en-us/windows/win32/api/winuser/nf-winuser-sendinput https://learn.microsoft.com/en-us/windows/win32/api/winuser/... [3] https://learn.microsoft.com/en-us/windows/win32/api/winuser/nf-winuser-mouse_event https://learn.microsoft.com/en-us/windows/win32/api/winuser/...
- EsportToys 3y ago
  Actuall you're missing ClipCursor as another way of moving the cursor. Also, mouse_event is just a wrapper around what's basically SendInput(mouse)
- lights0123 3y ago
  What do remote desktop tools do? TeamViewer allows mouse movement from even a phone touchscreen just fine, and unless they're computing the inverse of Windows's mouse acceleration, they must have another solution.
  - tjoff 3y ago
    Some at least don't even bother to send relative mouse movements, what matters is where you click. So they disable the guest mouse-pointer and rely on the host instead and then only send positions every now and then and when you are actually doing something interactive. Synergy types of applications don't have that freedom because the host mouse cursor don't extend to the other devices display.
deleted 3y ago
[deleted]
BazookaMusic 3y ago
Ah yes, the summoner of outages if you have production systems with windows server and allow user input in paths.
_8j50 3y ago
Most windows software uses utf16. This guy is used to web and *nix software where UTF-8 is popular. All your strings should default to UTF-16 on Windows so you don't have to deal with this against specific APIs. The longpath thing shouldn't be something you have to deal with each time you call an API. You should do what Git for windows and other software do which is enable long paths (assuming you actually need it) during setup/install.
- n2d4 3y ago
  A lot of people would disagree with you: >This document also recommends choosing UTF-8 for internal string representation in Windows applications, despite the fact that this standard is less popular there, both due to historical reasons and the lack of native UTF-8 support by the API. We believe that, even on this platform, the following arguments outweigh the lack of native support. https://utf8everywhere.org/ https://utf8everywhere.org/
  - jeroenhd 3y ago
    Linux and MacOS use UTF-8. Qt, Windows itself, Java, Javascript, and (on Windows) C# will all use UTF-16. Most computers run Windows, by a rather big margin. Even more computers use Javascript. Are you running KDE? Don't be surprised if half the text in your screen is secretly UTF-16! UTF-8 is definitely better, but if you're on Windows, you're making your own life harder by using UTF-8. Microsoft is migrating APIs to UTF-8 but it'll take years before that's finished. Just look at all the conversion code you need according to the page you linked. Now, if you're writing a cross platform library, you'll have to decide what encoding(s) you use. I personally don't really see why you wouldn't make your library encoding agnostic, but if you want to stick to one single encoding, you'll have to decide. If you're writing a program for Windows, technical superiority is a mediocre argument for making your own life that much harder. There are tons of formats and design decisions that may be technologically superior, but just end up wasting programmers' time, and this is one of them.
  - _8j50 3y ago
    Windows is not open source, I don't prefer either encoding I am just telling you that as that document acknowledges, UTF-16 is the default whether "everyone" (tech bubble) likes it or not. If you write windows software assume that default, not your favorite utf8. Just like how you assume endianess on a system because of the default.
laurent123456 3y ago
Every time I created software for Windows in recent years I used Qt, WxWidgets or even Delphi, which all have wrappers to handle system-specific quirks. Is it common these days to code directly to the Windows API? It feels like it would lead to code that's harder to maintain and certainly less portable.
hgs3 3y ago
It gets worse. Windows doesn't use UTF-16 for file paths it uses UCS-2. Example: Windows paths can contain unpaired UTF-16 surrogates which is illegal in UTF-16. Unix paths are no better; they are "bags of bytes" and not necessarily UTF-8. File paths are not guaranteed to be Unicode normalized. Even if they are canonically equivalent (equal graphemes) it doesn't mean they are binary equivalent (equal code points). You need to treat file paths as their own special thing and not strings.
- qsdf38100 3y ago
  Wrong. Windows uses UTF-16. It used to use UCS-2 a long time ago.
  - pornel 3y ago
    Unpaired surrogates are not allowed in UTF-16, so whatever Windows uses is not quite it.
    - jeroenhd 3y ago
      Windows doesn't bother checking every string after every modification, but neither does Linux with UTF-8. You can pass invalid UTF-8 to tons of APIs and almost all of them will just work, just like you can with UTF-16 in Windows. Doing a string validation check in every single API call would waste cycles for no good reason.
      - Karellen 3y ago
        Linux (the kernel) doesn't claim (AFAIK) to use UTF-8. It takes NUL-terminated strings in any charset you choose (or none) and either compares them with other NUL-terminated strings, or spits them back out again later. Interpreting a string as encoding text in particular character set is, as far as the kernel is concerned, a problem for userspace. Windows does make claims of "supporting" UTF-16.
        jeroenhd 3y ago
thehias 3y ago
Whats even worse: Trying to figure out if the current user/process is allowed to write to a file in a specific path.
- Someone1234 3y ago
  Because you shouldn't. Either try and write, and see if it fails, or if need-be ask for an exclusive lock on the file and see if the OS gives it to you. That isn't just permissions either, it actually picks up a lot of potential problems (e.g. device disconnected, lock conflict, et al).
  - colejohnson66 3y ago
    This. Seeing if you can do something before actually doing it falls victim to TOCTOU[0] bugs. Until one actually tries to perform the security-sensitive operation, there's no guaranteeing that you will succeed. [0]: https://en.wikipedia.org/wiki/Time-of-check_to_time-of-use https://en.wikipedia.org/wiki/Time-of-check_to_time-of-use
herf 3y ago
In addition to maxpath (which can be ignored in most regular apps), I remember short filenames used to have some severe performance implications. For instance, with Picasa we tried never to stat a file over Samba that didn't exist, because it would make Windows enumerate all the other files in the folder (presumably to see if they were a SFN match). That was a while ago (XP), but I just strace'd and it still seems to do it.
ilrwbwrkhv 3y ago
I don't know why but Microsoft programmers seem to make bizarre code choices. I worked with a lot of principal developers in my life but the ones from Microsoft wrote the most convoluted code I have ever seen. Even though it's anecdotal I wonder what is going on in the culture that Microsoft keeps producing these shabby programs all over the place. Casey has also spoken out about the horrible state of affairs in Microsoft in the past https://news.ycombinator.com/item?id=27728177 https://news.ycombinator.com/item?id=27728177 so I'm sure it's not a one off phenomena.
- patmorgan23 3y ago
  I think I read on another hacker news post that some windows API where intentionally made convoluted/hard to work with AND poorly documented in the official documentation. So that the API author's could go off and write a "win32 explained" book, and become fabulously wealthy. Allegedly.
  - ilrwbwrkhv 3y ago
    Yes! I remember that too. I wouldn't be surprised.
LargeTomato 3y ago
One time I installed a buggy version of Arduino and it installed recursively, creating file paths far larger than 260 char. I was unable to delete the files using explorer, cmd line, or power shell. I had to use cygwin. That was the most disappointed I've ever been with Windows.
- kmeisthax 3y ago
  Half a decade ago anything that needed Node packages - even just a Gulp build system - would wind up constructing extremely deep directory trees that Explorer would choke on. So you would npm install the project, realize you needed a different version of something, try to delete node_modules, and fail horribly with a bunch of long path errors.
- jeroenhd 3y ago
  You can remove these files using the command line by using the \\?\ syntax to bypass all normalisation and compatibility tricks Windows employs. Definitely works in CMD, never checked in Powershell (but I assume it works). I've had a similar issue with a recursive folder and transferring it to another host in Linux. A megabyte on my machine filled up the server pretty quick.
- MatmaRex 3y ago
  The built-in "robocopy" tool (normally used for making backups / mirrors) is capable of deleting any messed up directories like that, by mirroring an empty directory over it.
civilized 3y ago
Why are people obsessing about the overhead of two function calls? What application are you writing that needs to optimize passing millions of long file names in real time? Is this some kind of file path based video game?
- gpderetta 3y ago
  Git, for example, used by millions of developers every day.
  - GoblinSlayer 3y ago
    Windows kernel uses UTF-16, so reencoding happens anyway, even if you don't know about it.
  - im3w1l 3y ago
    Crazy idea... but what if you stored the files in a fast purpose-built database instead of the normal file system. You would have to mount it I suppose, to be able edit the files. But I think it could be very performant.
    - gpderetta 3y ago
      The filesystems is a database for storing files already. There is no particular reason a different approach be faster in principle.
      - kmeisthax 3y ago
        Git actually does have a separate disk format that's way more database-like, but it's only used for older data that doesn't need to be shuffled around as often. Git was written assuming the filesystem is decently fast at metadata mutation, which is a good assumption on Linux and less so on Windows.
      - im3w1l 3y ago
        Git needs some queries that filesystems arent very good at natively, like "get all modified files". There are workarounds but yeah.
grishka 3y ago
> most modern software uses UTF-8 [citation needed] UTF-8 is used for storage and serialization, yes, but most mainstream programming languages store unicode strings in memory as UTF-16. The notable exception is Rust, which does indeed use UTF-8 even in memory.
- loeg 3y ago
  > most mainstream programming languages store unicode strings in memory as UTF-16 Uh. What languages do you have in mind? C/C++ don't have any preference for UTF-16. Python3 doesn't use UTF-16. As you mentioned, Rust doesn't.
  - steeleduncan 3y ago
    java, .net, apple ecosystem (i.e. swift/objc),...
    - Longhanks 3y ago
      Swift is UTF-8 since 2019 https://www.swift.org/blog/utf8-string/ https://www.swift.org/blog/utf8-string/
  - grishka 3y ago
    Java, C# and the rest of .net, JavaScript, Swift/ObjC. C/C++ don't have a preference but, for example, if you're writing for Windows, you'd probably want to use wide characters. I'm not sure what kind of strings Linux GUIs use but I suspect it's wide characters as well.
    - 7jjjjjjj 3y ago
      Qt uses UTF-16, Gtk uses UTF-8.
    - Longhanks 3y ago
      Swift is UTF-8 since 2019 https://www.swift.org/blog/utf8-string/ https://www.swift.org/blog/utf8-string/
gumby 3y ago
C++'s filesystem package (part of the standard library) should take care of all that nonsense for you. It's whole point is so that you don't have to pay attention to format of the underlying filenames, directories, etc.
- LegionMammal978 3y ago
  I think some people might be wary of the std::filesystem APIs because the standard allows implementations to completely disregard TOCTOU issues internally, to the point of breaking memory safety [0]: > A file system race is the condition that occurs when multiple threads, processes, or computers interleave access and modification of the same object within a file system. Behavior is undefined if calls to functions provided by subclause [filesystems] introduce a file system race. It's not just implementation-defined behavior, but full UB! You're utterly at the mercy of your implementation to do something reasonable when it encounters a TOCTOU issue, or, for that matter, any kind of concurrent modification to a file or directory. And C++ has a long history of implementations being unreliable in their behavior when UB is encountered. [0] https://wg21.link/fs.race.behavior#1 https://wg21.link/fs.race.behavior#1
  - gumby 3y ago
    The more limited C API (system calls) is also full of race opportunities. The windows file APIs are better, but not a lot better. Anyway, the problem described in the post (scanning the string twice when converting) is as race prone, or not, as the the filesystem API.
    - LegionMammal978 3y ago
      > The more limited C API (system calls) is also full of race opportunities. The windows file APIs are better, but not a lot better. Sure, there are opportunities for races, but POSIX and the Windows API limit the possible outcomes of concurrent modification, and they also provide tools (fixed file handles, etc.) for well-written programs to prevent TOCTOU bugs. Meanwhile, the std::filesystem API just throws its hands in the air and says that filesystem races are UB, period, making it unusable if the program isn't in complete control of the directory tree it operates on. > Anyway, the problem described in the post (scanning the string twice when converting) is as race prone, or not, as the the filesystem API. I don't see what you mean? The typical scenario here is, you receive some absolute or relative path from an external source, encoded in ASCII or UTF-8 or some other non-UTF-16 encoding, and you want to operate on the file or directory at that path. Your internal path string is never going to change, only the filesystem can change. So there aren't any races from scanning your internal string twice to re-encode it; races can only come from using the re-encoded path in multiple calls to the filesystem API.
forrestthewoods 3y ago
Ehhhhhh. Passing ascii file paths to a Windows API is really straight forward and trivial. No conversion needed. MaxPath is real though. Exceeding it not recommended.
shortrounddev2 3y ago
> Alternatively you can set the process code page to UTF-8 and call the 'A' variant API directly, but only sometimes, and only with Windows 10 v1903+, and you might still have to change the system locale setting and reboot There is no need to do this; you can just call the A variant by name. Wide char and ascii functions are distinguished with W or A at the end, such as CreateFileA or CreateFileW. CreateFile is a macro.
- daemin 3y ago
  I always directly call the A or W versions of these functions. It makes things unambigious. Additionally I have my own Windows header file which undefines all of the macros so my function names do not get mangled with A and W suffixes.
- justsomehnguy 3y ago
  It really caught me by surprise. Sure, I'm not a dev but in my mind A variant was for 9x compatibility.
WirelessGigabit 3y ago
1607 has reached End of Servicing. So unless you need to support software (= $$$) on older versions that shouldn't be a concern. Second, I don't see a risk in setting `LongPathsEnabled` on a machine as it's only 1 piece of the puzzle. You still need to opt-in at the application level: https://learn.microsoft.com/en-us/windows/win32/fileio/maximum-file-path-limitation?tabs=registry https://learn.microsoft.com/en-us/windows/win32/fileio/maxim...
daemin 3y ago
For game development this should only matter on the game developers' machine, where there is more interaction with loose files on disk. For any shipped game being played on a player's machine the data files should be some form of packed files, which are close to the executable and only a handful need to be opened.
hohoharvey 3y ago
[dead]
searealist 3y ago
This article doesn't mention std::filesystem::path once, which is odd for a C++ article. Just set your executable code page to UTF-8, and let the filesystem::path constructor convert for you. It's pretty easy actually.

153 points by AshleysBrain 3y ago

The article should be called "What it takes to do anything Windows API in C++". For example, this Unicode issue is applicable to almost every API there. And it is not as simple as "call MultiByteToWideChar twice". Microsoft did not support UTF-16, they UCS-2, which they called "Unicode", but in fact back then it was just a wider ASCII. The worst thing about UCS-2 is that it does not support roundtrip. For example, imagine, that you wrote your Enterprise MS Tech Contoso Ltd.(R) authentication system, where user registers in UTF-8 on webpage, then on some layer it checks that user with the same name (in UTF-8 modern DB) does not exist, then it writes user basic information in UCS-2 encoded legacy "Enterprise DB". Aaand... Voila, Evil User overrides data of other user, because UTF-8 can't losslessly represent arbitrary sequences of 16-bit code units (should have used https://simonsapin.github.io/wtf-8/ https://simonsapin.github.io/wtf-8/ instead, but your Enterprise tech was destroyed).

colejohnson66 3y ago
They called it “Unicode” because, at the time, thanks to Han Unification, UCS-2 was widely believed to be the one and only encoding that anyone would ever need. Two bytes per codepoint. No more, and no less. C#/.NET even inherit this misnomer and call UTF-16LE “Unicode”: System.Text.Encoding.UnicodeEncoding

Cold_Miserable 3y ago

Windows uses unicode because NTFS uses unicode instead of efficient ascii.

becurious 3y ago
When it was designed there was no UTF-8. You had to know the code point; and file systems like FAT did not encode the code point, so file names would be interpreted by whatever the current code point was. So having a file system and other APIs support Unicode natively made a lot of sense at the time; it simplifies many things including deploying into organizations that have global users. This is now over 30 years ago. And the A vs the W suffixed APIs were not simple ascii; they were multi-byte, so to work with strings you still had to know the code page. Obviously UTF-8 is much simpler for that now, but that wasn’t the world when this was designed.

londons_explore 3y ago

Tl:DR: there are many different ways to send file paths to system library functions. That's mostly because of Windows' long legacy support history. Don't worry about it - just pick one and go.

c4mpute 3y ago
No. Because the API is fragmented into old, not-so-old and new parts, some higher-, some lower-level[0]. You need things from all of them, the new API isn't complete enough and has warts in places where using one of the older ones is better/easier/safer. So in the end you will have to use all the ways, and know when to use which one. All this can of course be fixed with another layer of indirection. But if you ever wanted to know why Windows file APIs and filesystems are so dog-slow, here is (part of) your answer. [0] e.g. there are lower level APIs that can create and access files and filenames that higher level APIs will filter out or disallow.
- sn_master 3y ago
  > new API isn't complete enough and has warts in places where using one of the older ones is better/easier/safer Describes Windows Forms vs everything else MS has been trying to do in the past 15 years.
- HeckFeck 3y ago
  I feel inclined to bypass all that Win32 gubbins and talk to the Kernel directly. Back in my day we just walked into the office of the man in charge and shook hands, we struck deals like real men. If we ask ask the NT kernel politely in a language it understands, we will get our filepath. Right? http://undocumented.ntinternals.net/index.html?page=UserMode%2FUndocumented%20Functions%2FNT%20Objects%2FFile%2FNtOpenFile.html http://undocumented.ntinternals.net/index.html?page=UserMode...
  - huhtenberg 3y ago
    NT API is now documented by Microsoft. The coverage is not complete, but enough to be useful. E.g. https://learn.microsoft.com/en-us/windows/win32/api/winternl/nf-winternl-ntopenfile https://learn.microsoft.com/en-us/windows/win32/api/winternl....

Affric 3y ago

So, a lot of what people think of when they think of OS is UI based (GUI/TUI/CLI) but kernel APIs are the real bread and butter and this is where the UNIX philosophy really shines. Passing a string as a file name in C++ with macOS or with Linux in my experience was simple. The permitted length of using ASCII characters is about 4 times as long (may god have mercy on your soul). I am not here to shit on Windows but the Windows devs clearly have a very different set of priorities (e.g. backwards compatibility) than the other breeds of modern OS devs. I guess to a large extent we all expect to be in the browser (gross) in some number of years but Windows seems so much harder from the perspective of someone that has programmed for Unixen and studied Windows as an OS.

cerved 3y ago
It's not all backwards comparability. I'm willing to bet that some (a large part?) is just sloppy software development. SQL Server (2017?) breaks if you update it on a UTF-8 Windows because it runs a TSQL that doesn't work with that code page. That script is a mess. Some of it is indented using tabs, some space. Trailing whitespace. Yuck
- nikanj 3y ago
  My hot take: Code quality is not measured by formatting issues, but by error resilience and number of actual bugs. Much of modern linting and commit hooking is dedicated to checking whitespace placement, variable naming and function lengths but the well-formatted newly rewritten code is still buggy as hell - it just looks pretty
  - cerved 3y ago
    I've often found that people that don't care about white-space also don't care that much about other aspects of code quality The inverse may not have the same correlation, since it can be automated.
  - edejong 3y ago
    Formatting doesn’t remove bugs, but it’ll help you detect them. Linted code helps you scan the code faster and provides valuable pattern recognition, allowing us to detect common mistakes. There have been numerous bugs caused by incorrect code formatting, most notably an SSL security bug from 2014: https://dwheeler.com/essays/apple-goto-fail.html https://dwheeler.com/essays/apple-goto-fail.html Another reason for formatting is the “minimal diff” paradigm. If a formatting rule would not be followed, in the next commit hitting this code, the format would also be affected, causing a larger diff than necessary. There are other reasons for simple format linting, but the reasons above are the most profound. Lastly, formatting is part of a range of static code analysis tools. Generally, formatting inconsistencies are the easiest to detect and resolve, as opposed to more sophisticated tools.

tester756 3y ago

Which is interesting because for example C#/.NET base class library provides very high quality APIs They even wrote a book about API design "Framework Design Guidelines: Conventions, Idioms, and Patterns for Reusable .Net Libraries"

toyg 3y ago
C# was effectively built from scratch with the benefit of hindsight from both C++ and Java. Decent practices should be considered been the bare minimum, surely.

h2odragon 3y ago

It's possible to get a file into a windows filesystem whose name contains "*" or ":" characters. Windows reacts quite poorly when it find those in file names; but only in certain APIs, others pass them without concern. Makes for fun explosions. The "260 character limit" is a good one; mostly seen that one in "I can do anything to this file but delete it!" complaints.

quickthrower2 3y ago
I had the can’t delete a file problem the other day. Got around it my moving it’s parent folder to a less nested location
huhtenberg 3y ago
It's also possible to create files whose names start (or contain) a NUL character. Doing so will trigger every single antivirus, but it's doable nonetheless and messes a lot of things very thoroughly.
- Someone1234 3y ago
  Also, possible in the Windows Registry, which no normal UI tools (inc. Reg Edit) could remove.
  - 13of40 3y ago
    A similar quirk of the registry API is that it has a simplified set of methods that don't specify the data type of the value, so you can have an application that expects to read a null terminated string value, but substitute a binary value with no null termination. If the app doesn't handle that right, it's buffer overrun time. Usually that's foot-shooting, but sometimes you can do that in HKCU as a low privileged user to a value that gets read by a higher privileged process and causes it to "misbehave".
- ale42 3y ago
  You can also create a file (or even better a folder) named CON or COM4. Good luck doing anything with it using the Explorer. (15 years ago I've seen some malware using that to make removal difficult)

wzdd 3y ago

Alternative sane approach: - Forget about long paths unless you really need to care. You need to care if you're accepting a file path from another app, for example. You don't need to care if you're using your own files in your install directory; too much of Windows doesn't support long files (including explorer, as mentioned, not to mention third party apps), so it's very unlikely that you'll need to deal with them if you're just handling your own data. - Work in utf-16 when dealing with file paths. Convert back and forth at the point of entry / exit to / from your app. It's unsurprising that you need to call the conversion function twice in order to find out how much memory to allocate, that's just how that works. Don't change random registry settings or code pages on the user's computer, that's mad.

Aardwolf 3y ago
> Work in utf-16 when dealing with file paths. Convert back and forth at the point of entry / exit to / from your app. But that's not compatible with other OSes, so you can't write native multiplatform C++ code just because windows keeps insisting on their worst-of-both-worlds UTF-16 (worst of both worlds because it's not single value per unicode character like UTF-32, and not ASCII compatible like UTF-8)
- GoblinSlayer 3y ago
  How do you write code that you can't write one function that opens a file?
- beached_whale 3y ago
  Store it as std::path
  - deleted 3y ago
    [deleted]
- plonk 3y ago
  Have a few functions taking UTF-8 paths for the filesystem operations you need, have separate versions for Windows and for other systems, and enable one or the other version with the preprocessor?

sn_master 3y ago

Now try doing this for Windows 98 or older, specially the non-ASCII file paths part (which was a common use case outside of the US).

ormax3 3y ago
https://en.wikipedia.org/wiki/Microsoft_Layer_for_Unicode https://en.wikipedia.org/wiki/Microsoft_Layer_for_Unicode
- sn_master 3y ago
  > The MSLU was announced in March 2001, and was first made available as a compatibility layer for Unicode-supporting code written for the then-new Windows XP RC1 in the July 2001 edition of Microsoft's Platform SDK. People had to deal with local language file names since Windows 3.1 at least and it became very common with Windows 95. Good luck if you wanted to deal with files named in more than 1 non-ASCII language (very common in Europe or the Middle East).

ormax3 3y ago

Like explained in https://utf8everywhere.org/#windows https://utf8everywhere.org/#windows , you can write simple wrapper functions `narrow`/`widen` used when you are about to call window api functions. ::SetWindowTextW(widen(someStdString).c_str()); Implementation is straightforward, relying on `WideCharToMultiByte`/`MultiByteToWideChar` to do the conversion: https://github.com/neacsum/utf8/blob/master/src/utf8.cpp https://github.com/neacsum/utf8/blob/master/src/utf8.cpp

huhtenberg 3y ago
You are missing OP's point - this still costs you 2 extra calls. If this cost really matters (and practically speaking it never does), then, as the other commenter said, the correct solution is to just use OS-native encoding for all file system paths and names used by the program, hidden behind an abstraction layer if needs be. UTF16 for Windows, UTF8 elsewhere.
- ormax3 3y ago
  The above manifesto makes the argument to use UTF-8 *everywhere*, even on windows where the internal representation is not native utf8. The conversion overhead is really negligible: https://utf8everywhere.org/#faq.cvt.perf https://utf8everywhere.org/#faq.cvt.perf (note: the two api calls per conversion is because how those specific functions work, first call to get the size to allocate, second to do the actual conversion, but you can always use another library in the implementation for the utf8<->utf16 conversion that might be more optimized than those windows api functions)
  - donatj 3y ago
    Especially negligible versus the trip to the file system you are setting up for.
    - Tempest1981 3y ago
      And AVX-512 can help: https://lemire.me/blog/2023/09/13/transcoding-unicode-strings-at-crazy-speeds-with-avx-512/ https://lemire.me/blog/2023/09/13/transcoding-unicode-string...

Tigress8780 3y ago

Talking about how strange the current state of Windows API is, there are actually 3 different APIs that can move the mouse cursor: SetCursorPos [1], SendInput [2], and mouse_event [3]. Applications might react differently to input events generated by these APIs. For example, in Windows 11's display settings, the position of screens can not be dragged if input events are coming from SetCursorPos, but works fine if using SendInput. Microsoft's own PowerToys uses a mixture of both under certain (complex) conditions, but I never found out the actual difference between them. I was writing an application that sends mouse input from a Linux machine to a Windows one (similar to Synergy), and I originally received mouse movement events that are accelerated (with user or system-defined acceleration factors), and I found out that there are no Windows API (three of them) that accepts relative movements without further acceleration (i.e. Windows will always apply further acceleration, making the mouse hard to use). I ended up directly hook into evdev to get raw mouse movements and let Windows accelerate them. [1] https://learn.microsoft.com/en-us/windows/win32/api/winuser/nf-winuser-setcursorpos https://learn.microsoft.com/en-us/windows/win32/api/winuser/... [2] https://learn.microsoft.com/en-us/windows/win32/api/winuser/nf-winuser-sendinput https://learn.microsoft.com/en-us/windows/win32/api/winuser/... [3] https://learn.microsoft.com/en-us/windows/win32/api/winuser/nf-winuser-mouse_event https://learn.microsoft.com/en-us/windows/win32/api/winuser/...

EsportToys 3y ago
Actuall you're missing ClipCursor as another way of moving the cursor. Also, mouse_event is just a wrapper around what's basically SendInput(mouse)
lights0123 3y ago
What do remote desktop tools do? TeamViewer allows mouse movement from even a phone touchscreen just fine, and unless they're computing the inverse of Windows's mouse acceleration, they must have another solution.
- tjoff 3y ago
  Some at least don't even bother to send relative mouse movements, what matters is where you click. So they disable the guest mouse-pointer and rely on the host instead and then only send positions every now and then and when you are actually doing something interactive. Synergy types of applications don't have that freedom because the host mouse cursor don't extend to the other devices display.

deleted 3y ago

[deleted]

BazookaMusic 3y ago

Ah yes, the summoner of outages if you have production systems with windows server and allow user input in paths.

_8j50 3y ago

Most windows software uses utf16. This guy is used to web and *nix software where UTF-8 is popular. All your strings should default to UTF-16 on Windows so you don't have to deal with this against specific APIs. The longpath thing shouldn't be something you have to deal with each time you call an API. You should do what Git for windows and other software do which is enable long paths (assuming you actually need it) during setup/install.

n2d4 3y ago
A lot of people would disagree with you: >This document also recommends choosing UTF-8 for internal string representation in Windows applications, despite the fact that this standard is less popular there, both due to historical reasons and the lack of native UTF-8 support by the API. We believe that, even on this platform, the following arguments outweigh the lack of native support. https://utf8everywhere.org/ https://utf8everywhere.org/
- jeroenhd 3y ago
  Linux and MacOS use UTF-8. Qt, Windows itself, Java, Javascript, and (on Windows) C# will all use UTF-16. Most computers run Windows, by a rather big margin. Even more computers use Javascript. Are you running KDE? Don't be surprised if half the text in your screen is secretly UTF-16! UTF-8 is definitely better, but if you're on Windows, you're making your own life harder by using UTF-8. Microsoft is migrating APIs to UTF-8 but it'll take years before that's finished. Just look at all the conversion code you need according to the page you linked. Now, if you're writing a cross platform library, you'll have to decide what encoding(s) you use. I personally don't really see why you wouldn't make your library encoding agnostic, but if you want to stick to one single encoding, you'll have to decide. If you're writing a program for Windows, technical superiority is a mediocre argument for making your own life that much harder. There are tons of formats and design decisions that may be technologically superior, but just end up wasting programmers' time, and this is one of them.
- _8j50 3y ago
  Windows is not open source, I don't prefer either encoding I am just telling you that as that document acknowledges, UTF-16 is the default whether "everyone" (tech bubble) likes it or not. If you write windows software assume that default, not your favorite utf8. Just like how you assume endianess on a system because of the default.

laurent123456 3y ago

Every time I created software for Windows in recent years I used Qt, WxWidgets or even Delphi, which all have wrappers to handle system-specific quirks. Is it common these days to code directly to the Windows API? It feels like it would lead to code that's harder to maintain and certainly less portable.

hgs3 3y ago

It gets worse. Windows doesn't use UTF-16 for file paths it uses UCS-2. Example: Windows paths can contain unpaired UTF-16 surrogates which is illegal in UTF-16. Unix paths are no better; they are "bags of bytes" and not necessarily UTF-8. File paths are not guaranteed to be Unicode normalized. Even if they are canonically equivalent (equal graphemes) it doesn't mean they are binary equivalent (equal code points). You need to treat file paths as their own special thing and not strings.

qsdf38100 3y ago
Wrong. Windows uses UTF-16. It used to use UCS-2 a long time ago.
- pornel 3y ago
  Unpaired surrogates are not allowed in UTF-16, so whatever Windows uses is not quite it.
  - jeroenhd 3y ago
    Windows doesn't bother checking every string after every modification, but neither does Linux with UTF-8. You can pass invalid UTF-8 to tons of APIs and almost all of them will just work, just like you can with UTF-16 in Windows. Doing a string validation check in every single API call would waste cycles for no good reason.
    - Karellen 3y ago
      Linux (the kernel) doesn't claim (AFAIK) to use UTF-8. It takes NUL-terminated strings in any charset you choose (or none) and either compares them with other NUL-terminated strings, or spits them back out again later. Interpreting a string as encoding text in particular character set is, as far as the kernel is concerned, a problem for userspace. Windows does make claims of "supporting" UTF-16.
      - jeroenhd 3y ago

thehias 3y ago

Whats even worse: Trying to figure out if the current user/process is allowed to write to a file in a specific path.

Someone1234 3y ago
Because you shouldn't. Either try and write, and see if it fails, or if need-be ask for an exclusive lock on the file and see if the OS gives it to you. That isn't just permissions either, it actually picks up a lot of potential problems (e.g. device disconnected, lock conflict, et al).
- colejohnson66 3y ago
  This. Seeing if you can do something before actually doing it falls victim to TOCTOU[0] bugs. Until one actually tries to perform the security-sensitive operation, there's no guaranteeing that you will succeed. [0]: https://en.wikipedia.org/wiki/Time-of-check_to_time-of-use https://en.wikipedia.org/wiki/Time-of-check_to_time-of-use

herf 3y ago

In addition to maxpath (which can be ignored in most regular apps), I remember short filenames used to have some severe performance implications. For instance, with Picasa we tried never to stat a file over Samba that didn't exist, because it would make Windows enumerate all the other files in the folder (presumably to see if they were a SFN match). That was a while ago (XP), but I just strace'd and it still seems to do it.

ilrwbwrkhv 3y ago

I don't know why but Microsoft programmers seem to make bizarre code choices. I worked with a lot of principal developers in my life but the ones from Microsoft wrote the most convoluted code I have ever seen. Even though it's anecdotal I wonder what is going on in the culture that Microsoft keeps producing these shabby programs all over the place. Casey has also spoken out about the horrible state of affairs in Microsoft in the past https://news.ycombinator.com/item?id=27728177 https://news.ycombinator.com/item?id=27728177 so I'm sure it's not a one off phenomena.

patmorgan23 3y ago
I think I read on another hacker news post that some windows API where intentionally made convoluted/hard to work with AND poorly documented in the official documentation. So that the API author's could go off and write a "win32 explained" book, and become fabulously wealthy. Allegedly.
- ilrwbwrkhv 3y ago
  Yes! I remember that too. I wouldn't be surprised.

LargeTomato 3y ago

One time I installed a buggy version of Arduino and it installed recursively, creating file paths far larger than 260 char. I was unable to delete the files using explorer, cmd line, or power shell. I had to use cygwin. That was the most disappointed I've ever been with Windows.

kmeisthax 3y ago
Half a decade ago anything that needed Node packages - even just a Gulp build system - would wind up constructing extremely deep directory trees that Explorer would choke on. So you would npm install the project, realize you needed a different version of something, try to delete node_modules, and fail horribly with a bunch of long path errors.
jeroenhd 3y ago
You can remove these files using the command line by using the \\?\ syntax to bypass all normalisation and compatibility tricks Windows employs. Definitely works in CMD, never checked in Powershell (but I assume it works). I've had a similar issue with a recursive folder and transferring it to another host in Linux. A megabyte on my machine filled up the server pretty quick.
MatmaRex 3y ago
The built-in "robocopy" tool (normally used for making backups / mirrors) is capable of deleting any messed up directories like that, by mirroring an empty directory over it.

civilized 3y ago

Why are people obsessing about the overhead of two function calls? What application are you writing that needs to optimize passing millions of long file names in real time? Is this some kind of file path based video game?

gpderetta 3y ago
Git, for example, used by millions of developers every day.
- GoblinSlayer 3y ago
  Windows kernel uses UTF-16, so reencoding happens anyway, even if you don't know about it.
- im3w1l 3y ago
  Crazy idea... but what if you stored the files in a fast purpose-built database instead of the normal file system. You would have to mount it I suppose, to be able edit the files. But I think it could be very performant.
  - gpderetta 3y ago
    The filesystems is a database for storing files already. There is no particular reason a different approach be faster in principle.
    - kmeisthax 3y ago
      Git actually does have a separate disk format that's way more database-like, but it's only used for older data that doesn't need to be shuffled around as often. Git was written assuming the filesystem is decently fast at metadata mutation, which is a good assumption on Linux and less so on Windows.
    - im3w1l 3y ago
      Git needs some queries that filesystems arent very good at natively, like "get all modified files". There are workarounds but yeah.

grishka 3y ago

> most modern software uses UTF-8 [citation needed] UTF-8 is used for storage and serialization, yes, but most mainstream programming languages store unicode strings in memory as UTF-16. The notable exception is Rust, which does indeed use UTF-8 even in memory.

loeg 3y ago
> most mainstream programming languages store unicode strings in memory as UTF-16 Uh. What languages do you have in mind? C/C++ don't have any preference for UTF-16. Python3 doesn't use UTF-16. As you mentioned, Rust doesn't.
- steeleduncan 3y ago
  java, .net, apple ecosystem (i.e. swift/objc),...
  - Longhanks 3y ago
    Swift is UTF-8 since 2019 https://www.swift.org/blog/utf8-string/ https://www.swift.org/blog/utf8-string/
- grishka 3y ago
  Java, C# and the rest of .net, JavaScript, Swift/ObjC. C/C++ don't have a preference but, for example, if you're writing for Windows, you'd probably want to use wide characters. I'm not sure what kind of strings Linux GUIs use but I suspect it's wide characters as well.
  - 7jjjjjjj 3y ago
    Qt uses UTF-16, Gtk uses UTF-8.
  - Longhanks 3y ago
    Swift is UTF-8 since 2019 https://www.swift.org/blog/utf8-string/ https://www.swift.org/blog/utf8-string/

gumby 3y ago

C++'s filesystem package (part of the standard library) should take care of all that nonsense for you. It's whole point is so that you don't have to pay attention to format of the underlying filenames, directories, etc.

LegionMammal978 3y ago
I think some people might be wary of the std::filesystem APIs because the standard allows implementations to completely disregard TOCTOU issues internally, to the point of breaking memory safety [0]: > A file system race is the condition that occurs when multiple threads, processes, or computers interleave access and modification of the same object within a file system. Behavior is undefined if calls to functions provided by subclause [filesystems] introduce a file system race. It's not just implementation-defined behavior, but full UB! You're utterly at the mercy of your implementation to do something reasonable when it encounters a TOCTOU issue, or, for that matter, any kind of concurrent modification to a file or directory. And C++ has a long history of implementations being unreliable in their behavior when UB is encountered. [0] https://wg21.link/fs.race.behavior#1 https://wg21.link/fs.race.behavior#1
- gumby 3y ago
  The more limited C API (system calls) is also full of race opportunities. The windows file APIs are better, but not a lot better. Anyway, the problem described in the post (scanning the string twice when converting) is as race prone, or not, as the the filesystem API.
  - LegionMammal978 3y ago
    > The more limited C API (system calls) is also full of race opportunities. The windows file APIs are better, but not a lot better. Sure, there are opportunities for races, but POSIX and the Windows API limit the possible outcomes of concurrent modification, and they also provide tools (fixed file handles, etc.) for well-written programs to prevent TOCTOU bugs. Meanwhile, the std::filesystem API just throws its hands in the air and says that filesystem races are UB, period, making it unusable if the program isn't in complete control of the directory tree it operates on. > Anyway, the problem described in the post (scanning the string twice when converting) is as race prone, or not, as the the filesystem API. I don't see what you mean? The typical scenario here is, you receive some absolute or relative path from an external source, encoded in ASCII or UTF-8 or some other non-UTF-16 encoding, and you want to operate on the file or directory at that path. Your internal path string is never going to change, only the filesystem can change. So there aren't any races from scanning your internal string twice to re-encode it; races can only come from using the re-encoded path in multiple calls to the filesystem API.

forrestthewoods 3y ago

Ehhhhhh. Passing ascii file paths to a Windows API is really straight forward and trivial. No conversion needed. MaxPath is real though. Exceeding it not recommended.

shortrounddev2 3y ago

> Alternatively you can set the process code page to UTF-8 and call the 'A' variant API directly, but only sometimes, and only with Windows 10 v1903+, and you might still have to change the system locale setting and reboot There is no need to do this; you can just call the A variant by name. Wide char and ascii functions are distinguished with W or A at the end, such as CreateFileA or CreateFileW. CreateFile is a macro.

daemin 3y ago
I always directly call the A or W versions of these functions. It makes things unambigious. Additionally I have my own Windows header file which undefines all of the macros so my function names do not get mangled with A and W suffixes.
justsomehnguy 3y ago
It really caught me by surprise. Sure, I'm not a dev but in my mind A variant was for 9x compatibility.

WirelessGigabit 3y ago

1607 has reached End of Servicing. So unless you need to support software (= $$$) on older versions that shouldn't be a concern. Second, I don't see a risk in setting `LongPathsEnabled` on a machine as it's only 1 piece of the puzzle. You still need to opt-in at the application level: https://learn.microsoft.com/en-us/windows/win32/fileio/maximum-file-path-limitation?tabs=registry https://learn.microsoft.com/en-us/windows/win32/fileio/maxim...

daemin 3y ago

For game development this should only matter on the game developers' machine, where there is more interaction with loose files on disk. For any shipped game being played on a player's machine the data files should be some form of packed files, which are close to the executable and only a handful need to be opened.

hohoharvey 3y ago

[dead]

searealist 3y ago

This article doesn't mention std::filesystem::path once, which is odd for a C++ article. Just set your executable code page to UTF-8, and let the filesystem::path constructor convert for you. It's pretty easy actually.