5 ms·
The Wild West of Windows command line parsing
- hulitu 5y agoParsing ? On windows even when programs accept command line arguments, good luck finding who they are.
- iqanq 5y agoThe Wild West of Trying to Shove UTF-8 onto a UTF-16 Operating System
- dgellow 5y agoSince 2019 Windows has support for a UTF-8 codepage. You can enable it via the application manifest. https://docs.microsoft.com/en-us/windows/apps/design/globalizing/use-utf8-code-page https://docs.microsoft.com/en-us/windows/apps/design/globali... That's just for the application though, the operation system itself is still using UTF-16. Edit: ah, nevermind, that's mentioned by the author in the article
- ChrisSD 5y agoThe issue there is that the Windows kernel does not validate Unicode. Linux has the same essential problem. Or more so, in that it does not specify an encoding at all.
- tialaramex 5y agoThey're essentially identical problems. In Linux these strings are arbitrary bytes with a rule forbidding byte 0x00, in Windows they're arbitrary unsigned 16-bit integers, I don't know if 0x0000 is forbidden. Not a significant difference. Rust's OsString provides a structure that has the potentially nonsensical raw data inside it, and offers ways to ask for: * The Unicode text if that's what is actually encoded (or else None) * The text after applying Unicode decoding rules and substituting U+FFFD (Unicode's "Replacement Character" �) where errors occur * The actual raw bytes / 16-bit unsigned integers If all your program cared about was a filename, this is not coincidentally also how these operating systems spell filenames, so you can just hand the OsString to an OS-level file API, to open it, rename it or whatever without caring if it is Unicode or not. The U+FFFD option does expose one potential surprise on Unix flavour systems which doesn't exist on Windows because U+FFFD is one UTF-16 code unit, but it is three UTF-8 code units, so such a "decoded" input could exceed the size of some internal storage you'd assumed would never need to be larger than the command line maximum length.
- deleted 5y ago[deleted]
- zokier 5y ago> Today, processes are nearly always launched by other programs, but despite this, must still serialize the argument array into a string as though a human had typed it out. Author seems to make assumption that an argument array is the fundamental form of a command-line, leading to this weird thinking that on Windows you need extra layer of serialization. But I'm not convinced that is actually the case, I'd view it more like you have a command-line expression and on unixy systems you serialize it as an array and on windows as a string, but neither is really more fundamental form.
- roelschroeven 5y agoMore fundamental or not, an array of arguments is much more practical than a string, hopefully well-formed, thrown at you. I can't think of any case where you would not want to split up that string in separate arguments. (I'm not talking about the case where you control both parent and child program; then you can do whatever you want and even use e.g. JSON. But in the general case, parents expect children to accept argument lists and children expect to be passes argument lists.)
- zokier 5y ago> I can't think of any case where you would not want to split up that string in separate arguments Random example, but jq takes its input expression non-tokenized (i.e. as a single argument) while in contrast find takes its expression as a sequence of tokens (i.e. an array of arguments).
- formerly_proven 5y agojq actually has a bunch of flags but they're not commonly used.
- xg15 5y agoTrue in principle: On Windows, the fundamental data structure for command line args is an array, while on linux it's an array of strings. I think you could argue that an array is a better-suited data structure than a single string though, as most programs take more than a single argument on the command line. Also note than on unixy systems, the command line is only "serialized as an array" when you run a command from the shell. (And then the shell does the serialisation, not the OS). If a program is launched by another process, the command line is always an array, never a string.
- xg15 5y agoWhat always confused me on Windows is how shells (e.g. CMD) fit into all of this. You can use pipes, variables, semicolon delimiters etc inside a line, which have to be parsed by the shell before launching any process - and those interact with quoting! So does that mean the shell first parses a line using its own quoting rules, then launches the processes which parse the line again, possibly using a different set of quoting rules? This sounds plainly insane.
- formerly_proven 5y agoIIRC Batch interprets ^ escapes unless it's double quoted - so, yes.
- mananaysiempre 5y agoIt also has a different idea of what’s actually double-quoted: it ignores backslashes completely. (Well, the Microsoft C runtime and SHELL32.DLL also have different ideas in that regard, as mentioned in the links in the post, but CMD.EXE’s different different idea is ... more different. I still don’t understand whetehr SHELL32.DLL and pre-2008 C runtime are the same or not. An MSDN blog post discussing this is—concisely if not in detail—here: https://docs.microsoft.com/en-gb/archive/blogs/twistylittlepassagesallalike/everyone-quotes-command-line-arguments-the-wrong-way https://docs.microsoft.com/en-gb/archive/blogs/twistylittlep...; I think I came upon it by following links from the head post but I can’t seem to recover the path.)
- ChrisSD 5y agoYes the shell interprets it first. `cmd.exe` does very little processing and mostly passes on the literal string. But what it does do can be confusing (e.g. `^` is an escape character which is annoying if you were trying to pass in a regex). In powershell you can use `--%` to pass a literal string as the command line. This is useful for applications that don't use the standard C/C++ parsing rules. See: https://docs.microsoft.com/en-us/powershell/module/microsoft.powershell.core/about/about_parsing?view=powershell-7.2#the-stop-parsing-token https://docs.microsoft.com/en-us/powershell/module/microsoft...
- iforgotpassword 5y ago> Many runtimes, including Microsoft’s own CRTs, don’t call GetCommandLineW and instead do their own parsing. I think that's supposed to be CommandLineToArgv here. As many others, I started on dos, then Windows and finally Linux. When I switched to Linux I found it really odd that the parsing is done by the shell, and how you can't know whether the user typed in a lot of jpg files or the shell expanded *.jpg. but the more I used it, the more reasonable it appeared. You almost never really need to know whether the shell expanded something or not. There is no ambiguity between programs, as the article also mentions. This has bitten me in Windows before, but was somehow just a fact of life. Quoting on Linux is consistent. On Windows, the first time you want to enter a command line that contains multiple literal " and spaces, you want to jump off the nearest cliff. Except you don't if you never used anything else, because it's just normal that every program uses somewhat different parsing rules.
- Someone 5y agoI don’t think that’s a strong argument. Counterarguments: - it’s easy to hit a Unix command line length limit when a single * argument expands into many arguments. xargs solves that, but it shouldn’t be necessary. - Unix tools cannot discriminate between * and the list of arguments it expands to. Consequently, rm cannot know whether you typed a single * or the specific list it expands into, so it cannot silently accept the latter and ask ‘are you sure’ for the former. - “cat foo* > foos” or “grep -r foo * > foos” fill up your disk. - tools that don’t want to accept file names as input have to do all kinds of escaping to prevent it. For example, you can’t have a calculator that can be invoked with calc 3 * 4 + 5 (sed, perl or python one-liners suffer the same fate) I think a system library function to do the globbing, etc. would have been the better choice. I also guess Unix doesn’t have that only because, back in the time globbing was invented, it would make all the executables so much larger.
- iforgotpassword 5y agoGood points, some more or less relevant. > - it’s easy to hit a Unix command line length limit when a single * argument expands into many arguments. xargs solves that, but it shouldn’t be necessary. Not on Linux. It used to be a single page (so in practice 4k), but on my system for example, it's 128k. Never ran into this problem. > Unix tools cannot discriminate between * and the list of arguments it expands to. Consequently, rm cannot know whether you typed a single * or the specific list it expands into, so it cannot ask silently accept the latter and ask ‘are you sure’ for the former. Fair point. I like to alias rm to rm -i for that reason, which is sometimes inconvenient too of you actually want to delete a lot of files, so I have to pass -f again to undo it... > “cat paper* > paper.all” or “grep foo > foos” fill up your disk Hmmm, does it create the output file first and then expand the arguments? Otherwise the output file has to already exist, which would create the same problem on Windows, or Linux with a glob syscall, unless you explicitly check if stdout is redirected to a file, and that file is one of your input files. I guess in windows this cannot happen in practice as you cannot open a file twice for reading and writing, but then this has nothing to do with command line handling anymore. > calc 3 * 4 + 5 You'd probably make it so that it has its own parsing too in that case, so you can do calc '3 * 4 + 5'. There are ugly situations with the Linux or really shell way, and after having switched to Linux a long time ago I might be biased towards it, but I really think I had greater wtf moments with command line handling on Windows.
- _0w8t 5y agoAs the author pointed out, Windows does not have API to convert a list of arguments into a string to launch a process. Here is how Chromium does that: https://chromium.googlesource.com/chromium/src/+/refs/heads/main/base/command_line.cc#106 https://chromium.googlesource.com/chromium/src/+/refs/heads/...
- ChrisSD 5y ago> Rust (LLVM) pragmatically call[s] CommandLineToArgv This is not correct. It calls `GetCommandLine` to get the raw command line string and then attempts to parse it in the same way as modern C/C++. https://github.com/rust-lang/rust/blob/3b186511f62b0ce20e72ede0e8e13f8787155f02/library/std/src/sys/windows/args.rs#L35 https://github.com/rust-lang/rust/blob/3b186511f62b0ce20e72e... The rules themselves are officially documented here: https://docs.microsoft.com/en-us/cpp/cpp/main-function-command-line-args?view=msvc-160#parsing-c-command-line-arguments https://docs.microsoft.com/en-us/cpp/cpp/main-function-comma... Though the linked David Deley investigation is more in-depth: https://daviddeley.com/autohotkey/parameters/parameters.htm#WIN https://daviddeley.com/autohotkey/parameters/parameters.htm#...
- ryukoposting 5y agoWow. I would have never known. For the last couple years, I have been running Windows every day on all of my computers. I used Debian Testing for many years before that, though- that's where I learned to code. When I'm not using WSL2 (which is rare), it's almost always little CLI tools using Python, Rust, Nim, or C with MinGW. I'm a professional C programmer, I use Windows every day at home and at work, and I couldn't tell you how to use the Windows API to get command line args. All I can say is that I'm all the more grateful for the MinGW people for letting me pretend this insanity isn't there.
- unnouinceput 5y agoQuote: "From Windows XP through Windows 11, this returns exactly the same address as GetCommandLineW. There’s little reason to do it this way other than to annoy Raymond Chen, but it’s still neat and maybe has some super niche use" Hihi, this made me lol. I bet plenty enough will start to hardcode the address, leading to Ray having an entry saying "this is not in the contract" and after many will do it regardless, in about 10 years and 2 Windows versions later Ray will write another grump-ish entry in his blog of how he had to patch old software that was failing due to Microsoft finally making that address dynamic.
- t43562 5y agoThe way CreateProcess takes a string as an argument instead of argv is what made me lose patience with Windows overall. I wasn't a big fan of Windows to start with but I had to support it as a potential platform in a build syste. We wrote a kind of miniature shell that worked on Windows and Linux to capture and keep output from being interleaved (something GNU make does itself now). It was a great misery to try to quote various bits of a string so that windows cmdline parsing would interpret internationalised filenames with spaces, single or double quotes etc as separate arguments. The command was potentially a call to exec bash or some other shell with its own quoting rules going on in the arguments.... This may seem like a small thing but it made me really reject the idea of wasting time on the platform. It's really the most crucial API in the OS and it's awful and there was a very simple pre-existing example of how to do it right (char *argv[]) - what excuse was there for it?