5 ms·

> Many runtimes, including Microsoft’s own CRTs, don’t call GetCommandLineW and instead do their own parsing. I think that's supposed to be CommandLineToArgv h

> Many runtimes, including Microsoft’s own CRTs, don’t call GetCommandLineW and instead do their own parsing. I think that's supposed to be CommandLineToArgv here. As many others, I started on dos, then Windows and finally Linux. When I switched to Linux I found it really odd that the parsing is done by the shell, and how you can't know whether the user typed in a lot of jpg files or the shell expanded *.jpg. but the more I used it, the more reasonable it appeared. You almost never really need to know whether the shell expanded something or not. There is no ambiguity between programs, as the article also mentions. This has bitten me in Windows before, but was somehow just a fact of life. Quoting on Linux is consistent. On Windows, the first time you want to enter a command line that contains multiple literal " and spaces, you want to jump off the nearest cliff. Except you don't if you never used anything else, because it's just normal that every program uses somewhat different parsing rules.

Someone 5y ago
I don’t think that’s a strong argument. Counterarguments: - it’s easy to hit a Unix command line length limit when a single * argument expands into many arguments. xargs solves that, but it shouldn’t be necessary. - Unix tools cannot discriminate between * and the list of arguments it expands to. Consequently, rm cannot know whether you typed a single * or the specific list it expands into, so it cannot silently accept the latter and ask ‘are you sure’ for the former. - “cat foo* > foos” or “grep -r foo * > foos” fill up your disk. - tools that don’t want to accept file names as input have to do all kinds of escaping to prevent it. For example, you can’t have a calculator that can be invoked with calc 3 * 4 + 5 (sed, perl or python one-liners suffer the same fate) I think a system library function to do the globbing, etc. would have been the better choice. I also guess Unix doesn’t have that only because, back in the time globbing was invented, it would make all the executables so much larger.
- iforgotpassword 5y ago
  Good points, some more or less relevant. > - it’s easy to hit a Unix command line length limit when a single * argument expands into many arguments. xargs solves that, but it shouldn’t be necessary. Not on Linux. It used to be a single page (so in practice 4k), but on my system for example, it's 128k. Never ran into this problem. > Unix tools cannot discriminate between * and the list of arguments it expands to. Consequently, rm cannot know whether you typed a single * or the specific list it expands into, so it cannot ask silently accept the latter and ask ‘are you sure’ for the former. Fair point. I like to alias rm to rm -i for that reason, which is sometimes inconvenient too of you actually want to delete a lot of files, so I have to pass -f again to undo it... > “cat paper* > paper.all” or “grep foo > foos” fill up your disk Hmmm, does it create the output file first and then expand the arguments? Otherwise the output file has to already exist, which would create the same problem on Windows, or Linux with a glob syscall, unless you explicitly check if stdout is redirected to a file, and that file is one of your input files. I guess in windows this cannot happen in practice as you cannot open a file twice for reading and writing, but then this has nothing to do with command line handling anymore. > calc 3 * 4 + 5 You'd probably make it so that it has its own parsing too in that case, so you can do calc '3 * 4 + 5'. There are ugly situations with the Linux or really shell way, and after having switched to Linux a long time ago I might be biased towards it, but I really think I had greater wtf moments with command line handling on Windows.
  - jcelerier 5y ago
    > It used to be a single page (so in practice 4k), but on my system for example, it's 128k. Never ran into this problem. that's fairly reachable with some complicated compiler command lines where things are 8-levels deep in some random embedded SDK
    - Hello71 5y ago
      but that's not relevant to the original question of *
  - Someone 5y ago
    > but on my system for example, it's 128k. Never ran into this problem. That may be because of decades of defence against it. For example, the naive way to run foo on all your .c files in one go is: find . -type f -name "*.c" -exec foo {} \+ (aside: also note that I had to escape the + at the end of the command. RANT: of course, man find doesn’t mention that, as that’s a shell issue that has nothing to do with find. Technically correct, but beginning users will be baffled) In my home directory, that’s about a megabyte of arguments, so I have to fall back to find . -type f -name "*.c" -exec foo {} \; which calls foo thousands of times, to invoking xargs, or to some other workaround. Either way, foo is called more than once, so the result may not be what I expected. For real-world examples, see - https://stackoverflow.com/questions/55453993/argument-list-too-long-error-for-find-command https://stackoverflow.com/questions/55453993/argument-list-t... - https://www.linuxquestions.org/questions/linux-software-2/argument-list-too-long-why-am-i-getting-this-with-the-find-command-585622/ https://www.linuxquestions.org/questions/linux-software-2/ar... - https://www.baeldung.com/linux/argument-list-too-long-error https://www.baeldung.com/linux/argument-list-too-long-error > You'd probably make it so that it has its own parsing too in that case, so you can do calc '3 * 4 + 5’ That’s what you would do, yes, but that’s not ideal, and won’t work well if you want to write a one-liner for a tool whose language can contain single quotes, such as sed/awk/perl/…. > Hmmm, does it create the output file first and then expand the arguments? Yes, it does. It all works, but isn’t ideal, and I think using a library to do argument parsing would help here (variable expansion may be a problem when doing that, though, as postponing that till after the tool runs may introduce a race condition when running a tool in the background. If that is a problem at all, I think that’s solvable, though)
- drdec 5y ago
  I think that rm in particular is usually a shell built-in, not a separate executable, so it does have access to the exact arguments. But your larger point still stands.
  - Karellen 5y ago
    Really? It doesn't appear to be on bash, csh, ksh, tcsh, or zsh. Which shells do have `rm` builtin? https://manpages.debian.org/bullseye/bash/bash.1.en.html#SHELL_BUILTIN_COMMANDS https://manpages.debian.org/bullseye/bash/bash.1.en.html#SHE... https://manpages.debian.org/bullseye/csh/csh.1.en.html#Built-in_commands https://manpages.debian.org/bullseye/csh/csh.1.en.html#Built... <https://manpages.debian.org/bullseye/ksh/ksh.1.en.html#Built-in_Commands. https://manpages.debian.org/bullseye/ksh/ksh.1.en.html#Built...> https://manpages.debian.org/bullseye/tcsh/tcsh.1.en.html#Builtin_commands https://manpages.debian.org/bullseye/tcsh/tcsh.1.en.html#Bui... https://manpages.debian.org/bullseye/zsh-common/zshbuiltins.1.en.html https://manpages.debian.org/bullseye/zsh-common/zshbuiltins....
- Hello71 5y ago
  > - it’s easy to hit a Unix command line length limit when a single * argument expands into many arguments. xargs solves that, but it shouldn’t be necessary. this isn't true in the vast majority of cases on Linux and most other Unix-like operating systems; Linux has ARG_MAX=2097152, macOS =262144, and Solaris 10 =1048320. this problem does often arise with globstar, but! that is actually an argument towards having the shell expand arguments, not each command; if each command had to implement globstar, it would be effectively useless. > - Unix tools cannot discriminate between * and the list of arguments it expands to. Consequently, rm cannot know whether you typed a single * or the specific list it expands into, so it cannot silently accept the latter and ask ‘are you sure’ for the former. zsh comes with a function to check "rm" arguments and issue a warning if rm * is supplied. this doesn't work for arbitrary commands, but few commands actually require such treatment. a comparison could be made to tab completion functions, which are also shell-specific. > - “cat foo* > foos” or “grep -r foo * > foos” fill up your disk. I am almost certain that the implied premise is incorrect. According to https://pubs.opengroup.org/onlinepubs/9699919799/utilities/V3_chap02.html https://pubs.opengroup.org/onlinepubs/9699919799/utilities/V..., the shell performs word expansion before redirection. > - tools that don’t want to accept file names as input have to do all kinds of escaping to prevent it. For example, you can’t have a calculator that can be invoked with this is a bad example; it is simple to write "calc '3 * 4 + 5'", or of course more reasonably "calc '3*4+5'". in fact, this program already exists, and is called expr. > I think a system library function to do the globbing, etc. would have been the better choice. that would be terrible. it would be impossible to innovate globbing extensions like ksh or zsh; the system administrator would dictate for all users a major component of their experience. in modern terms, it would be like only being able to install Gnome or KDE, not both at the same time.
  - JNRowe 5y ago
    > zsh comes with a function to check "rm" arguments and issue a warning if rm * is supplied. For those of us following along at home that is the RM_STAR_SILENT option, and there is also a RM_STAR_WAIT option should you wish to just have a little wait first. > it is simple to write "calc '3 * 4 + 5'" You can use the GP's syntax without quoting on zsh by using the noglob builtin: `noglob calc 3 * 4 + 5`. There are a few commands I like to alias with `noglob =$cmd` for nicer interactive usage. -- GNU coreutils' cat and GNU grep will also error out in the basic case of IO loops with `{cat,grep}: foos: input file is output file`. I largely agree with the premise that there are a heap of problems though, and having some environments that fix the some of the problems isn't all that useful.
- Karellen 5y ago
  > I also guess Unix doesn’t have that only because, back in the time globbing was invented, it would make all the executables so much larger. I think it was more that, doing it in the shell meant it would be possible to write another shell with totally different syntax and globbing rules. That way, all the commands would still work, no matter which shell any particular user chose to use. The bourne shell (and its descendants) won out as a "standard" in the end, and the system shell at `/bin/sh` is now required by POSIX to be bourne-compatible, but Unix was designed to have pluggable shells, and it might not have turned out that way.
- chlorion 5y ago
  I don't think programs should be aware of globbing at all except in special circumstances, like with the `find` command. It could be helpful in a few rare circumstances I guess, like with 'rm' as you showed, but you can have the shell itself handle those cases, which is what some shells such as zsh do today. With regards to trying to pass a literal '*' to commands like calc, I think the user is responsible for escaping the star and not calc itself. Since the shell is what handles glob expansion, calc has no way of even doing so that I can think of. Escaping a glob just requires putting it in single quotes like so calc '3 * 4 + 5'
azalemeth 5y ago
> Quoting on Linux is consistent. Out of curiosity, what do you use to do this? Gnu getopt and genopt are brilliant in my experience and handle all of the edge cases really well. The fact that they are consistent just means that it works, as long as you don't hit the stack limit.
- iforgotpassword 5y ago
  I mean, from the apps perspective, everything has already been either parsed properly by the shell, or passed directly as intended in an array by another program. From the user's perspective, the shell does it the same for every program you invoke. Not all shells have exactly the same quoting rules necessarily, but it's the component that's deciding how to do it. If you're in doubt you can even ask it to print how it parses the command line you're currently writing. On Windows I can compile the same C program with two different compilers and get two different comment line parsing behaviors, which I'd say, is much less obvious to the user.

Someone 5y ago

I don’t think that’s a strong argument. Counterarguments: - it’s easy to hit a Unix command line length limit when a single * argument expands into many arguments. xargs solves that, but it shouldn’t be necessary. - Unix tools cannot discriminate between * and the list of arguments it expands to. Consequently, rm cannot know whether you typed a single * or the specific list it expands into, so it cannot silently accept the latter and ask ‘are you sure’ for the former. - “cat foo* > foos” or “grep -r foo * > foos” fill up your disk. - tools that don’t want to accept file names as input have to do all kinds of escaping to prevent it. For example, you can’t have a calculator that can be invoked with calc 3 * 4 + 5 (sed, perl or python one-liners suffer the same fate) I think a system library function to do the globbing, etc. would have been the better choice. I also guess Unix doesn’t have that only because, back in the time globbing was invented, it would make all the executables so much larger.

iforgotpassword 5y ago
Good points, some more or less relevant. > - it’s easy to hit a Unix command line length limit when a single * argument expands into many arguments. xargs solves that, but it shouldn’t be necessary. Not on Linux. It used to be a single page (so in practice 4k), but on my system for example, it's 128k. Never ran into this problem. > Unix tools cannot discriminate between * and the list of arguments it expands to. Consequently, rm cannot know whether you typed a single * or the specific list it expands into, so it cannot ask silently accept the latter and ask ‘are you sure’ for the former. Fair point. I like to alias rm to rm -i for that reason, which is sometimes inconvenient too of you actually want to delete a lot of files, so I have to pass -f again to undo it... > “cat paper* > paper.all” or “grep foo > foos” fill up your disk Hmmm, does it create the output file first and then expand the arguments? Otherwise the output file has to already exist, which would create the same problem on Windows, or Linux with a glob syscall, unless you explicitly check if stdout is redirected to a file, and that file is one of your input files. I guess in windows this cannot happen in practice as you cannot open a file twice for reading and writing, but then this has nothing to do with command line handling anymore. > calc 3 * 4 + 5 You'd probably make it so that it has its own parsing too in that case, so you can do calc '3 * 4 + 5'. There are ugly situations with the Linux or really shell way, and after having switched to Linux a long time ago I might be biased towards it, but I really think I had greater wtf moments with command line handling on Windows.
- jcelerier 5y ago
  > It used to be a single page (so in practice 4k), but on my system for example, it's 128k. Never ran into this problem. that's fairly reachable with some complicated compiler command lines where things are 8-levels deep in some random embedded SDK
  - Hello71 5y ago
    but that's not relevant to the original question of *
- Someone 5y ago
  > but on my system for example, it's 128k. Never ran into this problem. That may be because of decades of defence against it. For example, the naive way to run foo on all your .c files in one go is: find . -type f -name "*.c" -exec foo {} \+ (aside: also note that I had to escape the + at the end of the command. RANT: of course, man find doesn’t mention that, as that’s a shell issue that has nothing to do with find. Technically correct, but beginning users will be baffled) In my home directory, that’s about a megabyte of arguments, so I have to fall back to find . -type f -name "*.c" -exec foo {} \; which calls foo thousands of times, to invoking xargs, or to some other workaround. Either way, foo is called more than once, so the result may not be what I expected. For real-world examples, see - https://stackoverflow.com/questions/55453993/argument-list-too-long-error-for-find-command https://stackoverflow.com/questions/55453993/argument-list-t... - https://www.linuxquestions.org/questions/linux-software-2/argument-list-too-long-why-am-i-getting-this-with-the-find-command-585622/ https://www.linuxquestions.org/questions/linux-software-2/ar... - https://www.baeldung.com/linux/argument-list-too-long-error https://www.baeldung.com/linux/argument-list-too-long-error > You'd probably make it so that it has its own parsing too in that case, so you can do calc '3 * 4 + 5’ That’s what you would do, yes, but that’s not ideal, and won’t work well if you want to write a one-liner for a tool whose language can contain single quotes, such as sed/awk/perl/…. > Hmmm, does it create the output file first and then expand the arguments? Yes, it does. It all works, but isn’t ideal, and I think using a library to do argument parsing would help here (variable expansion may be a problem when doing that, though, as postponing that till after the tool runs may introduce a race condition when running a tool in the background. If that is a problem at all, I think that’s solvable, though)
drdec 5y ago
I think that rm in particular is usually a shell built-in, not a separate executable, so it does have access to the exact arguments. But your larger point still stands.
- Karellen 5y ago
  Really? It doesn't appear to be on bash, csh, ksh, tcsh, or zsh. Which shells do have `rm` builtin? https://manpages.debian.org/bullseye/bash/bash.1.en.html#SHELL_BUILTIN_COMMANDS https://manpages.debian.org/bullseye/bash/bash.1.en.html#SHE... https://manpages.debian.org/bullseye/csh/csh.1.en.html#Built-in_commands https://manpages.debian.org/bullseye/csh/csh.1.en.html#Built... <https://manpages.debian.org/bullseye/ksh/ksh.1.en.html#Built-in_Commands. https://manpages.debian.org/bullseye/ksh/ksh.1.en.html#Built...> https://manpages.debian.org/bullseye/tcsh/tcsh.1.en.html#Builtin_commands https://manpages.debian.org/bullseye/tcsh/tcsh.1.en.html#Bui... https://manpages.debian.org/bullseye/zsh-common/zshbuiltins.1.en.html https://manpages.debian.org/bullseye/zsh-common/zshbuiltins....
Hello71 5y ago
> - it’s easy to hit a Unix command line length limit when a single * argument expands into many arguments. xargs solves that, but it shouldn’t be necessary. this isn't true in the vast majority of cases on Linux and most other Unix-like operating systems; Linux has ARG_MAX=2097152, macOS =262144, and Solaris 10 =1048320. this problem does often arise with globstar, but! that is actually an argument towards having the shell expand arguments, not each command; if each command had to implement globstar, it would be effectively useless. > - Unix tools cannot discriminate between * and the list of arguments it expands to. Consequently, rm cannot know whether you typed a single * or the specific list it expands into, so it cannot silently accept the latter and ask ‘are you sure’ for the former. zsh comes with a function to check "rm" arguments and issue a warning if rm * is supplied. this doesn't work for arbitrary commands, but few commands actually require such treatment. a comparison could be made to tab completion functions, which are also shell-specific. > - “cat foo* > foos” or “grep -r foo * > foos” fill up your disk. I am almost certain that the implied premise is incorrect. According to https://pubs.opengroup.org/onlinepubs/9699919799/utilities/V3_chap02.html https://pubs.opengroup.org/onlinepubs/9699919799/utilities/V..., the shell performs word expansion before redirection. > - tools that don’t want to accept file names as input have to do all kinds of escaping to prevent it. For example, you can’t have a calculator that can be invoked with this is a bad example; it is simple to write "calc '3 * 4 + 5'", or of course more reasonably "calc '3*4+5'". in fact, this program already exists, and is called expr. > I think a system library function to do the globbing, etc. would have been the better choice. that would be terrible. it would be impossible to innovate globbing extensions like ksh or zsh; the system administrator would dictate for all users a major component of their experience. in modern terms, it would be like only being able to install Gnome or KDE, not both at the same time.
- JNRowe 5y ago
  > zsh comes with a function to check "rm" arguments and issue a warning if rm * is supplied. For those of us following along at home that is the RM_STAR_SILENT option, and there is also a RM_STAR_WAIT option should you wish to just have a little wait first. > it is simple to write "calc '3 * 4 + 5'" You can use the GP's syntax without quoting on zsh by using the noglob builtin: `noglob calc 3 * 4 + 5`. There are a few commands I like to alias with `noglob =$cmd` for nicer interactive usage. -- GNU coreutils' cat and GNU grep will also error out in the basic case of IO loops with `{cat,grep}: foos: input file is output file`. I largely agree with the premise that there are a heap of problems though, and having some environments that fix the some of the problems isn't all that useful.
Karellen 5y ago
> I also guess Unix doesn’t have that only because, back in the time globbing was invented, it would make all the executables so much larger. I think it was more that, doing it in the shell meant it would be possible to write another shell with totally different syntax and globbing rules. That way, all the commands would still work, no matter which shell any particular user chose to use. The bourne shell (and its descendants) won out as a "standard" in the end, and the system shell at `/bin/sh` is now required by POSIX to be bourne-compatible, but Unix was designed to have pluggable shells, and it might not have turned out that way.
chlorion 5y ago
I don't think programs should be aware of globbing at all except in special circumstances, like with the `find` command. It could be helpful in a few rare circumstances I guess, like with 'rm' as you showed, but you can have the shell itself handle those cases, which is what some shells such as zsh do today. With regards to trying to pass a literal '*' to commands like calc, I think the user is responsible for escaping the star and not calc itself. Since the shell is what handles glob expansion, calc has no way of even doing so that I can think of. Escaping a glob just requires putting it in single quotes like so calc '3 * 4 + 5'

azalemeth 5y ago

> Quoting on Linux is consistent. Out of curiosity, what do you use to do this? Gnu getopt and genopt are brilliant in my experience and handle all of the edge cases really well. The fact that they are consistent just means that it works, as long as you don't hit the stack limit.

iforgotpassword 5y ago
I mean, from the apps perspective, everything has already been either parsed properly by the shell, or passed directly as intended in an array by another program. From the user's perspective, the shell does it the same for every program you invoke. Not all shells have exactly the same quoting rules necessarily, but it's the component that's deciding how to do it. If you're in doubt you can even ask it to print how it parses the command line you're currently writing. On Windows I can compile the same C program with two different compilers and get two different comment line parsing behaviors, which I'd say, is much less obvious to the user.