13 ms·

Quoting command line arguments the wrong way (2011)

127 points by moopling 10y ago

ajross 10y ago
Everyone does this "wrong" because every app does this differently. The core reason windows command line options aren't (or at least shouldn't be) used to pass complicated data is that way back when DOS simply provided a single command line string to the executed program and let it parse it itself. So no two command line parsers are the same. The glitch with escaping here is merely one symptom of a broader problem. Unix got this right by forcing the shell to provide the kernel a pre-parsed list of strings, so the only insanity the tool integrator needs to understand is the shell's quoting syntax. Which is still insane. But it's only insane in one particular way.
- comex 10y ago
  On Unix, most tools don't really need to use the shell at all; it's enough to treat argument lists as lists internally and pass them to exec or posix_spawn (which of course, unlike the Windows _exec and _spawn, aren't broken for arguments with spaces!). However, at minimum it is still useful to shell-escape arguments when displaying them to the user (for ease of copy+paste), so it's unfortunate that many languages don't have any standard library function to do this - including C on POSIX, and Python before version 3.
  - iokanuon 10y ago
    But in comparasion with MS systems, escaping strings for unix shells is very simple - just prepend and append an ' and change every ' into '\''
    - ajross 10y ago
      Alas, no. Bourne shell syntax allows for double quotes with variable interpolation and some other fancy syntax (including backslash escaping of literal double-quote characters), and a single-quote syntax for "raw" strings with no fancy syntax INCLUDING backslash escaping. So your rule won't work. You can't single-quote a string that itself contains single quotes, which makes for some fun when you have arbitrary strings (file names are the big frustration) that need to be substituted into a parseable command line. But like I said above: Bourne syntax[1] is only one kind of insanity, which is still much better than the DOS/Windows world of a separate parser for every app. [1] We shall not speak of the C shell here.
      - skissane 10y ago
        A method I've used: replace all occurrences of ' with '"'"' and then surround resulting string with ' It's a bit ugly but it works.
      - ufo 10y ago
        Their rule does work. The single quotes are transformed into '\''. The first ' closes the current single quotes string. The \' adds a single quote character outside a quoted string (and outside single quotes you can use backslash escapes). Finally, the last ' reopens the single quoted string that we closed before.
  - d0mine 10y ago
    pipes.quote() is in Python's stdlib since forever. Though it was not documented until it became shlex.quote() in Python 3.
  - kazinator 10y ago
    _exec an _spawn are not Windows; they are functions in the MS Visual C redistributable run-time. (Well, they are also in a system DLL called MSVCRT.DLL. That is an internal library which is undocumented and considered by Microsoft to be off-limits to applications.)
- pwdisswordfish 10y ago
  Unix got something right in that you can unambiguously pass a list of separate strings to launched processes. However, it does nothing to ensure unambiguous meaning of those strings. This is for example why you should avoid giving your files such cute names as '-rf'.
  - sukilot 10y ago
    What does it mean to say that an argument's meaning could be unamibigious regardless of the program it is passed into ? that's a logical impossibility Still, it would be right and proper if Unix programs a little type-safety in their arguments, for example by requiring that ALL arguments be flags, as in this hypothetical smart_rm: "/bin/smart_rm -rf --pattern foo/bar"
  - tuukkah 10y ago
    You can escape file names that look like options with '--'.
    - ufo 10y ago
      This depends on the application recognizing the -- convention and also depends on having all the little scripts in your system remembering to use the --.
      - kazinator 10y ago
        Even if the OS kernel provided a process launching API with separated options and arguments, that would not remove the need for the -- syntax to remove the ambiguity at the shell level, and hence your need to use that in scripts. It would remove the problem of programs all being required to implement --.
  - quotemstr 10y ago
    > This is for example why you should avoid giving your files such cute names as '-rf'. The kernel should ban these names. I'm a big fan of dwheeler's proposal for fixing filenames: see http://www.dwheeler.com/essays/fixing-unix-linux-filenames.html http://www.dwheeler.com/essays/fixing-unix-linux-filenames.h... These is no god damn reason why a filename should be able to contain, say, LF, DEL, or BEL. None whatsoever.
- kevin_thibedeau 10y ago
  MS does provide an alternate startup routine that parses the arguments before entering main().
- kazinator 10y ago
  > the only insanity the tool integrator needs to understand is the shell's quoting syntax Only if the designer is using command-based functions like the ISO C system or POSIX popen; not when forking and exec'ing programs.
zkhalique 10y ago
Oh brother, this is one of the reasons UNIX was much more developer-friendly.
- agentgt 10y ago
  Actually Bash does have some issues with command line arguments if doing variable expansion. For example in Java apps it is actually fairly difficult to pass -DsomeParameter="something with a space in it" if doing variable expansion.
  - kbp 10y ago
    Could you clarify what you mean? What's wrong with -DsomeParameter="$var"? Or am I misunderstanding?
    - agentgt 10y ago
      I'm trying to find the exact situation but basically if you have something like: PROPS="-Dprop1=foo bar -Dprop2=bar foo" java $PROPS SomeClass.class You can try to escape with backslashes and what not but it becomes fairly hard if not impossible to expand multiple parameters correctly as a single variable. I believe one solution is to use arrays and another is just not use arguments and instead rely on other configuration mechanisms (env variables, files, etc). You see this often rear its ugly head with daemon scripts.
      - colemickens 10y ago
        A bit hard to understand your exact scenario here (basically the same shell lexing problem...), but I think this is what you're looking for. I left an example of an argument with a space in there (the last one). PROPS=("-Dprop1=foo" "bar" "-Dprop2=bar foo") java "${PROPS[@]}" SomeClass.class (The quotes around `${PROPS[@]}` is important.) Do note, there is an unfortunate edge case here if PROPS is empty, you'll get a false `""` arg passed to java in that case. There's a less pleasant syntax that avoids that issue but I don't recall it off-hand. (edit: Please read the replies to my post, I didn't think about the fact that this syntax is bash specific. Thanks to those who pointed it out)
pwdisswordfish 10y ago
This is a quite good example of the kind of problems you run into when you follow the philosophy of representing all data in informally-specified ad-hoc text formats. Everyone thinks they can just roll their own parser/serialiser, which they then neglect to test thoroughly enough, creating subtle bugs when the serialisation side forgets to escape data somewhere, or the parsing side doesn't even provide any way to escape grammar-significant characters.
- tbrownaw 10y ago
  No, the problem is the lack of a standard encoder function to go with the standard decoder function. It's not "everyone thinks they can", it's "everyone has to". The problem is also YAGNI and validation-thru-testing, instead of up-front design. The "ad-hoc" and "text" parts aren't what's important, it's the whole approach of not doing any more than the bare minimum of up-front work. Which seems to historically give overall better results, even if it does come with interesting bugs that need fixing later.
  - pwdisswordfish 10y ago
    Well, the real problem here is that there's not really a 'standard' you can rely on in the first place. And while it's true that text isn't a necessary part of the general problem, in my experience text-based formats seem especially prone to it. How many times have you seen people attempt to use regular expressions to parse HTML/validate e-mail addresses/whatever?
- hk__2 10y ago
  I don't see how the problems you're listing have anything to do with text formats; you have these kind of issues with any "informally-specified" format.
- grymoire1 10y ago
  I think this indicates a huge difference between Microsoft/UNIX mindsets. Microsoft allowed rename *.txt *.bak To do this, the "rename" command had to understand how to parse the asterisk character while being familiar with the contents of the directory. However, creating a new replacement "rename" command is difficult, as well as creating new commands that can parse wildcards. In the Unix environment, the shell expands the asterisk to all files that matches that pattern, and then passes these files to the "rename" command, who never sees the asterisk. Therefore it's trivial to create a new "rename' utility because it doesn't need to parse wildcards. However, renaming all .txt to .bak is awkward in a UNIX system.
makecheck 10y ago
If an API has a problem, you fix the API. If necessary, you release a free library to back-port the improved API to older OS versions as well. You come up with a fixed version, make it as convenient as possible, call it the “new standard”, officially deprecate the alternatives, and then write a blog post. Except the blog post would only need two lines of sample code showing how easy it is to work around the problem now. Not this. Frankly, few developers will even know about the need for careful coding such as this, and even fewer will actually do it because it will muck up each and every program with dozens of lines of extra stuff to work around a deficient part of the PLATFORM.
- JoshTriplett 10y ago
  Exactly. While they can't fix the old functions without potentially breaking software that relied on the old behavior, they can introduce a new set of process-launching functions that do the right thing: handle arguments exactly as written, no splitting, no quoting, no metacharacter interpretation. Then, for new OSes, write a compatibility layer to implement the old functions on top of the new ones, and for old OSes, write a compatibility library to implement the new functions by quoting and passing to the old ones. Mark the old functions as "deprecated, do not use in new code", and point to the new ones.
  - gravypod 10y ago
    This is something that is important. The steps to go about. 1. Fix old solution 2. Implement alternative that is feature complete and document 3. Write comparability layers If people followed this life as a software developer would be easier on bloodpressure. I remember fondly working with many APIs thst were deprecated but had no alternative and the devs admitted it.
- tetrep 10y ago
  I'm absolutely amazed that Windows doesn't offer a process spawning API that takes an array of strings as arguments[0], if only because that's exactly how a C program expects them anyway. [0]: https://linux.die.net/man/3/exec https://linux.die.net/man/3/exec
nilved 10y ago
I'm going to use this opportunity to ask a question I've been thinking about for a long time. Why do we have both environment variables and command line arguments? They are the same thing, except one is key-to-value and one is positional and often needs to be parsed by hand in an ad-hoc fashion. I don't think that people should use command line arguments when environment variables are an option, and I'm not aware of any use cases where they are not an option.
- hk__2 10y ago
  Environment variable affect all instances; command-line arguments affect only one program instance. Set your defaults with environment variables (or a config file), then override them as needed with command-line flags.
  - icebraining 10y ago
    No, you can definitively pass an environment variable to a single instance.
    - grymoire1 10y ago
      Yes, that is true but very misleading. If the single instance does not create any new processes, then there is no practical difference. The difference occurs when a process A created process B and process B spawns one or more processes.
- makecheck 10y ago
  Some reasons: — It is possible to “see” a command line (in "ps" output, etc.) in ways that you won’t see environment settings. This can be useful when passing information to a sub-process that you need to keep private. — It may be that you are configuring your sub-process in a place that is “far away” from the point that actually executes the command. Rather than have to thread an extra command-line argument through your code to make sure it is part of the final command invocation, it can be quite convenient to just set a variable. I have often used this to enable debugging features or test experimental features, or even to disable entire features when unexpected problems arise. — In a similar way, your program may use multiple languages or otherwise be difficult to manage in any common way without environment variables. — In a cross-platform scenario, environment variable names might be far easier to keep constant across UNIX, Windows, etc. than command-line syntax. I’m sure there are other reasons.
jdright 10y ago
I dont really know what to say from this without risking being downvoted. But, this coming from a Microsoft blog is a little... awkward. DOS heritage + really bad shell implementations, well.. I avoid the hell out of using command line on any Windows. Luckily there is mintty.
- marcosdumay 10y ago
  > well.. I avoid the hell out of using command line on any Windows How do you avoid it if it's the only API available for spawning processes on Windows?
  - alkonaut 10y ago
    High level Api's could easily provide this, e.g. the C# Process.Start(executable, args); Takes a single string as args, and has no overload taking an array as args, which formats it safely and correctly so that the receiving process would see the same string in its args vector. So while it seems to be pretty easily fixable, it hasn't been.
    - marcosdumay 10y ago
      The right way to do it would be to change the OS API into accepting an array of strings, and turn this function into an overlay that will have its parameters parsed and broken apart in user-space. But yes, if MS ever fixes this it's more likely that they'll go into the route you described. I can't wait to see how many bug reports with crazy descriptions it will create, and how many more overlays will be written to fix the bugs preserving backward compatibility.
- gaius 10y ago
  DOS's command line conventions come from the PDP-11 which pre-dates Unix.
  - Piskvorrr 10y ago
    The barest core does. Quoting and escaping...is not a part of that, IIRC.
joeyh 10y ago
That's about Windows, but many uses of system() that involve non-static strings also probably get quoting wrong. Of course, avoiding the shell is the best way to avoid the problem. Sometimes, you can't avoid the shell. My preferred shell quoting method, for unix, is to wrap each parameter in single quotes. Then only single quotes inside a parameter are a problem. They can be replaced with '"'"' Probably a lot of things use double quotes and perhaps try to escape $ and ' and " but miss details to do with \ and perhaps other characters that some shells treat specially. Another way is to pass the filename in the environment: system("rm -rf \"$DIR\"")
- pwdisswordfish 10y ago
  > Sometimes, you can't avoid the shell. What circumstances have you got in mind?
  - tene 10y ago
    My favorite awkward environment to cite is running a command remotely over ssh. As far as I've been able to tell from casual testing, without having read the source code yet, ssh does something very similar to what Windows does here and just glues everything together with spaces and passes it to the remote shell for interpretation, so you have to deal with the shell and provide your own quoting.
    - dozzie 10y ago
      >>> Sometimes, you can't avoid the shell. >> What circumstances have you got in mind? > My favorite awkward environment to cite is running a command remotely over ssh. Dude, if you run remote commands by calling them through SSH, you didn't just got things backward: you fucked things up heavily. SSH was never ever designed as a batch, unsupervised tool, despite many people using it as such. Remote code that is parametrized should be run exactly as that: as a remote procedure call, a technique known for over thirty years now. One of the reasons is quoting (because for non-interactive SSH call the command needs to be quoted exactly twice if in shell, and exactly once when run from exec()), but there are problems with distributing keys, maintaining usable home directories, and disabling unnecessary things that are enabled by default (port forwarding, among the others), and that doesn't exhaust the list of issues. Proper RPC protocol, like XML-RPC (which was released twenty years ago and is still usable while being quite simple), covers quoting -- or actually, serializing data -- without programmers worrying if they got their list of metacharacters right and did enough passes for things to work correctly. On the other hand, I'm not surprised that people do this through SSH (and a variant of this stupidity: adding apache user to sudoers, so a web panel can add firewall rules). After all, I've never seen an easy to use RPC server that has all the procedures passed in its configuration. I needed to write such thing myself (once in Perl, as xmlrpcd, and recently in Python, with custom protocol that can do a little more, as harpd of HarpCaller project).
subfnsnx 10y ago
Como hackear
rusanu 10y ago
Looking now at how Windows handles console application arguments, it sure looks broken. But you have to put your mindset in cca. 1990 and think what Windows applications looked like back then, and what was the model Microsoft was betting on. Arguments were passed via DDE[0], and then later all the bets were on OLE[1] and finally COM[2]. System components were all the time accessed via in-process DLLs communicating with services over LRPC[3]. In this world, the command line, the pipe philosophy and the 'less is more' mindset were not only not welcome, they were the adversary. Even when finally it was aknowledged that the command shell needs some love too, the answer was PowerShell, which yet again defined an object interface between cmdlets[4]. [0] https://en.wikipedia.org/wiki/Dynamic_Data_Exchange https://en.wikipedia.org/wiki/Dynamic_Data_Exchange [1] https://en.wikipedia.org/wiki/Object_Linking_and_Embedding https://en.wikipedia.org/wiki/Object_Linking_and_Embedding [2] https://en.wikipedia.org/wiki/Component_Object_Model https://en.wikipedia.org/wiki/Component_Object_Model [3] https://en.wikipedia.org/wiki/Local_Procedure_Call https://en.wikipedia.org/wiki/Local_Procedure_Call [4] https://en.wikipedia.org/wiki/PowerShell#Pipeline https://en.wikipedia.org/wiki/PowerShell#Pipeline
- sukilot 10y ago
  That's a long way of saying that Microsoft spent many years promoting a overcomplicated bad ideas over simple correct ideas.
  - rdslw 10y ago
    So true. After 25yrs using pcs/unix/macs, I've come to conclusion that it's deliberate on Microsoft side. They create much more closed, difficult and obscure technology which needs, training, certification and lot of care (services provided by Microsoft).
    - quotemstr 10y ago
      Trust me: it's not deliberate. There is no conspiracy. Everyone I met on Windows is at least as well-intentioned as anyone working in the POSIX world. The reason Windows has some bad APIs is the same reason Unix has bad APIs: someone bootstraps a system quickly and doesn't see the problems that can arise from their choice of APIs; the system becomes wildly successful; and now everyone has to support these ill-conceived APIs. Sure, Windows command-line argument passing is bad, but have you ever tried using wait/waitpid/wait4/waitid/etc.? That's a nightmare in the POSIX world; Windows has nice, clean process handles, not the garbage /proc stuff that makes it fundamentally impossible to write a safe pkill(1). If you're writing a brand-new system, for the love of God, do a good job of designing the APIs. You will not have a chance to go back and fix the APIs later.
jaclaz 10y ago
[2011]
grymoire1 10y ago
Oh. I was confused until I realized the publishing site was Microsoft. Apparently "Everyone" only refers to Windows programmers, and Unix/Mac/whatever programmers do not exist in this universe.
- colemickens 10y ago
  Looks like the title has been fixed but I was also confused before this thread filled up with comments confirming my suspicions that it was a Windows-specific issue. It does seem rather misleading to say "everyone" does it wrong when it's specifically a problem with Windows APIs (though not terribly surprising from the Windows team).
- chc 10y ago
  It's an MSDN article. Most of the articles on Apple's developer site are equally inapplicable to Windows development without explicitly saying so.
wfunction 10y ago
I recall seeing different behavior between the C runtime and CommandLineToArgvW. I don't remember what the difference was, but I remember it driving me nuts.
saurik 10y ago
A friend of mine had to fix this recently for PowerShell, which had a regression that caused arguments you passed to programs to be incorrectly escaped to executed commands. https://github.com/PowerShell/PowerShell/pull/2182 https://github.com/PowerShell/PowerShell/pull/2182 It is extremely common to get this wrong. Apache Portable Runtime even gets this wrong :/. (I haven't submitted a patch for this yet, but I intend to: I ran into it a couple months ago and then got distracted after working around it in my program by predicting what incorrect escaping might be performed by APR and compensating by adding quotes and escape characters to my input to their open process function... my build is statically linked so I don't feel bad about this temporary hack ;P.)
- quotemstr 10y ago
  Wow. Microsoft really has changed for the better since I left. Not a coincidence, I'm sure. ;-)
vram22 10y ago
IIRC, Microsoft C (quite some years ago - had worked on a product using it) had different variants of functions to spawn or execute a process from another one - like the exec family - execlp, execvp, execvpe, execlpe, etc., which varied in things like fixed vs. variable number of args, checking environment vs. not, etc. I also remember reading about it in a Waite Publications book, The Microsoft C Bible, by Nabajyoti Barkakati. Not sure if the issues mentioned in the OP could be solved if those functions were present - need to check. Edit: and the DOS / Windows exec family of functions was likely derived from Unix's exec().
quotemstr 10y ago
Oh, hi. This is my article. I'm a bit sad that I lost edit rights when I left Microsoft.
- ronsor 10y ago
  You should always host your own blog and then post a summary on the MSDN blogs -- always have control over content
Pxtl 10y ago
What I find frustrating is how many MS tools freak at the sight of a quoted path when quoted paths are what "copy filename with path" (or whatever the context command is called) gives.
deleted 10y ago
[deleted]
ihaveahadron 10y ago
Unless it's all in my head , there are a lot of things that, everyone, is doing the wrong way. Maybe read a book from someone that did it the right way and is dead now, that doesn't mean you can't read new things . Pussies
kazinator 10y ago
This should be "quoting command line arguments the right way for passage into applications developed using Microsoft Visual C, and linked to its C run-time library that parses the command line string and calls main or wmain". There is no general correct way to quote command line arguments in Windows, because every application receives just a character string which it parses however it wants. There is no single specification for the syntax by which arguments are delimited within the command string.

127 points by moopling 10y ago

Everyone does this "wrong" because every app does this differently. The core reason windows command line options aren't (or at least shouldn't be) used to pass complicated data is that way back when DOS simply provided a single command line string to the executed program and let it parse it itself. So no two command line parsers are the same. The glitch with escaping here is merely one symptom of a broader problem. Unix got this right by forcing the shell to provide the kernel a pre-parsed list of strings, so the only insanity the tool integrator needs to understand is the shell's quoting syntax. Which is still insane. But it's only insane in one particular way.

comex 10y ago
On Unix, most tools don't really need to use the shell at all; it's enough to treat argument lists as lists internally and pass them to exec or posix_spawn (which of course, unlike the Windows _exec and _spawn, aren't broken for arguments with spaces!). However, at minimum it is still useful to shell-escape arguments when displaying them to the user (for ease of copy+paste), so it's unfortunate that many languages don't have any standard library function to do this - including C on POSIX, and Python before version 3.
- iokanuon 10y ago
  But in comparasion with MS systems, escaping strings for unix shells is very simple - just prepend and append an ' and change every ' into '\''
  - ajross 10y ago
    Alas, no. Bourne shell syntax allows for double quotes with variable interpolation and some other fancy syntax (including backslash escaping of literal double-quote characters), and a single-quote syntax for "raw" strings with no fancy syntax INCLUDING backslash escaping. So your rule won't work. You can't single-quote a string that itself contains single quotes, which makes for some fun when you have arbitrary strings (file names are the big frustration) that need to be substituted into a parseable command line. But like I said above: Bourne syntax[1] is only one kind of insanity, which is still much better than the DOS/Windows world of a separate parser for every app. [1] We shall not speak of the C shell here.
    - skissane 10y ago
      A method I've used: replace all occurrences of ' with '"'"' and then surround resulting string with ' It's a bit ugly but it works.
    - ufo 10y ago
      Their rule does work. The single quotes are transformed into '\''. The first ' closes the current single quotes string. The \' adds a single quote character outside a quoted string (and outside single quotes you can use backslash escapes). Finally, the last ' reopens the single quoted string that we closed before.
- d0mine 10y ago
  pipes.quote() is in Python's stdlib since forever. Though it was not documented until it became shlex.quote() in Python 3.
- kazinator 10y ago
  _exec an _spawn are not Windows; they are functions in the MS Visual C redistributable run-time. (Well, they are also in a system DLL called MSVCRT.DLL. That is an internal library which is undocumented and considered by Microsoft to be off-limits to applications.)
pwdisswordfish 10y ago
Unix got something right in that you can unambiguously pass a list of separate strings to launched processes. However, it does nothing to ensure unambiguous meaning of those strings. This is for example why you should avoid giving your files such cute names as '-rf'.
- sukilot 10y ago
  What does it mean to say that an argument's meaning could be unamibigious regardless of the program it is passed into ? that's a logical impossibility Still, it would be right and proper if Unix programs a little type-safety in their arguments, for example by requiring that ALL arguments be flags, as in this hypothetical smart_rm: "/bin/smart_rm -rf --pattern foo/bar"
- tuukkah 10y ago
  You can escape file names that look like options with '--'.
  - ufo 10y ago
    This depends on the application recognizing the -- convention and also depends on having all the little scripts in your system remembering to use the --.
    - kazinator 10y ago
      Even if the OS kernel provided a process launching API with separated options and arguments, that would not remove the need for the -- syntax to remove the ambiguity at the shell level, and hence your need to use that in scripts. It would remove the problem of programs all being required to implement --.
- quotemstr 10y ago
  > This is for example why you should avoid giving your files such cute names as '-rf'. The kernel should ban these names. I'm a big fan of dwheeler's proposal for fixing filenames: see http://www.dwheeler.com/essays/fixing-unix-linux-filenames.html http://www.dwheeler.com/essays/fixing-unix-linux-filenames.h... These is no god damn reason why a filename should be able to contain, say, LF, DEL, or BEL. None whatsoever.
kevin_thibedeau 10y ago
MS does provide an alternate startup routine that parses the arguments before entering main().
kazinator 10y ago
> the only insanity the tool integrator needs to understand is the shell's quoting syntax Only if the designer is using command-based functions like the ISO C system or POSIX popen; not when forking and exec'ing programs.

zkhalique 10y ago

Oh brother, this is one of the reasons UNIX was much more developer-friendly.

agentgt 10y ago
Actually Bash does have some issues with command line arguments if doing variable expansion. For example in Java apps it is actually fairly difficult to pass -DsomeParameter="something with a space in it" if doing variable expansion.
- kbp 10y ago
  Could you clarify what you mean? What's wrong with -DsomeParameter="$var"? Or am I misunderstanding?
  - agentgt 10y ago
    I'm trying to find the exact situation but basically if you have something like: PROPS="-Dprop1=foo bar -Dprop2=bar foo" java $PROPS SomeClass.class You can try to escape with backslashes and what not but it becomes fairly hard if not impossible to expand multiple parameters correctly as a single variable. I believe one solution is to use arrays and another is just not use arguments and instead rely on other configuration mechanisms (env variables, files, etc). You see this often rear its ugly head with daemon scripts.
    - colemickens 10y ago
      A bit hard to understand your exact scenario here (basically the same shell lexing problem...), but I think this is what you're looking for. I left an example of an argument with a space in there (the last one). PROPS=("-Dprop1=foo" "bar" "-Dprop2=bar foo") java "${PROPS[@]}" SomeClass.class (The quotes around `${PROPS[@]}` is important.) Do note, there is an unfortunate edge case here if PROPS is empty, you'll get a false `""` arg passed to java in that case. There's a less pleasant syntax that avoids that issue but I don't recall it off-hand. (edit: Please read the replies to my post, I didn't think about the fact that this syntax is bash specific. Thanks to those who pointed it out)

pwdisswordfish 10y ago

This is a quite good example of the kind of problems you run into when you follow the philosophy of representing all data in informally-specified ad-hoc text formats. Everyone thinks they can just roll their own parser/serialiser, which they then neglect to test thoroughly enough, creating subtle bugs when the serialisation side forgets to escape data somewhere, or the parsing side doesn't even provide any way to escape grammar-significant characters.

tbrownaw 10y ago
No, the problem is the lack of a standard encoder function to go with the standard decoder function. It's not "everyone thinks they can", it's "everyone has to". The problem is also YAGNI and validation-thru-testing, instead of up-front design. The "ad-hoc" and "text" parts aren't what's important, it's the whole approach of not doing any more than the bare minimum of up-front work. Which seems to historically give overall better results, even if it does come with interesting bugs that need fixing later.
- pwdisswordfish 10y ago
  Well, the real problem here is that there's not really a 'standard' you can rely on in the first place. And while it's true that text isn't a necessary part of the general problem, in my experience text-based formats seem especially prone to it. How many times have you seen people attempt to use regular expressions to parse HTML/validate e-mail addresses/whatever?
hk__2 10y ago
I don't see how the problems you're listing have anything to do with text formats; you have these kind of issues with any "informally-specified" format.
grymoire1 10y ago
I think this indicates a huge difference between Microsoft/UNIX mindsets. Microsoft allowed rename *.txt *.bak To do this, the "rename" command had to understand how to parse the asterisk character while being familiar with the contents of the directory. However, creating a new replacement "rename" command is difficult, as well as creating new commands that can parse wildcards. In the Unix environment, the shell expands the asterisk to all files that matches that pattern, and then passes these files to the "rename" command, who never sees the asterisk. Therefore it's trivial to create a new "rename' utility because it doesn't need to parse wildcards. However, renaming all .txt to .bak is awkward in a UNIX system.

makecheck 10y ago

If an API has a problem, you fix the API. If necessary, you release a free library to back-port the improved API to older OS versions as well. You come up with a fixed version, make it as convenient as possible, call it the “new standard”, officially deprecate the alternatives, and then write a blog post. Except the blog post would only need two lines of sample code showing how easy it is to work around the problem now. Not this. Frankly, few developers will even know about the need for careful coding such as this, and even fewer will actually do it because it will muck up each and every program with dozens of lines of extra stuff to work around a deficient part of the PLATFORM.

JoshTriplett 10y ago
Exactly. While they can't fix the old functions without potentially breaking software that relied on the old behavior, they can introduce a new set of process-launching functions that do the right thing: handle arguments exactly as written, no splitting, no quoting, no metacharacter interpretation. Then, for new OSes, write a compatibility layer to implement the old functions on top of the new ones, and for old OSes, write a compatibility library to implement the new functions by quoting and passing to the old ones. Mark the old functions as "deprecated, do not use in new code", and point to the new ones.
- gravypod 10y ago
  This is something that is important. The steps to go about. 1. Fix old solution 2. Implement alternative that is feature complete and document 3. Write comparability layers If people followed this life as a software developer would be easier on bloodpressure. I remember fondly working with many APIs thst were deprecated but had no alternative and the devs admitted it.
tetrep 10y ago
I'm absolutely amazed that Windows doesn't offer a process spawning API that takes an array of strings as arguments[0], if only because that's exactly how a C program expects them anyway. [0]: https://linux.die.net/man/3/exec https://linux.die.net/man/3/exec

nilved 10y ago

I'm going to use this opportunity to ask a question I've been thinking about for a long time. Why do we have both environment variables and command line arguments? They are the same thing, except one is key-to-value and one is positional and often needs to be parsed by hand in an ad-hoc fashion. I don't think that people should use command line arguments when environment variables are an option, and I'm not aware of any use cases where they are not an option.

hk__2 10y ago
Environment variable affect all instances; command-line arguments affect only one program instance. Set your defaults with environment variables (or a config file), then override them as needed with command-line flags.
- icebraining 10y ago
  No, you can definitively pass an environment variable to a single instance.
  - grymoire1 10y ago
    Yes, that is true but very misleading. If the single instance does not create any new processes, then there is no practical difference. The difference occurs when a process A created process B and process B spawns one or more processes.
makecheck 10y ago
Some reasons: — It is possible to “see” a command line (in "ps" output, etc.) in ways that you won’t see environment settings. This can be useful when passing information to a sub-process that you need to keep private. — It may be that you are configuring your sub-process in a place that is “far away” from the point that actually executes the command. Rather than have to thread an extra command-line argument through your code to make sure it is part of the final command invocation, it can be quite convenient to just set a variable. I have often used this to enable debugging features or test experimental features, or even to disable entire features when unexpected problems arise. — In a similar way, your program may use multiple languages or otherwise be difficult to manage in any common way without environment variables. — In a cross-platform scenario, environment variable names might be far easier to keep constant across UNIX, Windows, etc. than command-line syntax. I’m sure there are other reasons.

jdright 10y ago

I dont really know what to say from this without risking being downvoted. But, this coming from a Microsoft blog is a little... awkward. DOS heritage + really bad shell implementations, well.. I avoid the hell out of using command line on any Windows. Luckily there is mintty.

marcosdumay 10y ago
> well.. I avoid the hell out of using command line on any Windows How do you avoid it if it's the only API available for spawning processes on Windows?
- alkonaut 10y ago
  High level Api's could easily provide this, e.g. the C# Process.Start(executable, args); Takes a single string as args, and has no overload taking an array as args, which formats it safely and correctly so that the receiving process would see the same string in its args vector. So while it seems to be pretty easily fixable, it hasn't been.
  - marcosdumay 10y ago
    The right way to do it would be to change the OS API into accepting an array of strings, and turn this function into an overlay that will have its parameters parsed and broken apart in user-space. But yes, if MS ever fixes this it's more likely that they'll go into the route you described. I can't wait to see how many bug reports with crazy descriptions it will create, and how many more overlays will be written to fix the bugs preserving backward compatibility.
gaius 10y ago
DOS's command line conventions come from the PDP-11 which pre-dates Unix.
- Piskvorrr 10y ago
  The barest core does. Quoting and escaping...is not a part of that, IIRC.

joeyh 10y ago

That's about Windows, but many uses of system() that involve non-static strings also probably get quoting wrong. Of course, avoiding the shell is the best way to avoid the problem. Sometimes, you can't avoid the shell. My preferred shell quoting method, for unix, is to wrap each parameter in single quotes. Then only single quotes inside a parameter are a problem. They can be replaced with '"'"' Probably a lot of things use double quotes and perhaps try to escape $ and ' and " but miss details to do with \ and perhaps other characters that some shells treat specially. Another way is to pass the filename in the environment: system("rm -rf \"$DIR\"")

pwdisswordfish 10y ago
> Sometimes, you can't avoid the shell. What circumstances have you got in mind?
- tene 10y ago
  My favorite awkward environment to cite is running a command remotely over ssh. As far as I've been able to tell from casual testing, without having read the source code yet, ssh does something very similar to what Windows does here and just glues everything together with spaces and passes it to the remote shell for interpretation, so you have to deal with the shell and provide your own quoting.
  - dozzie 10y ago
    >>> Sometimes, you can't avoid the shell. >> What circumstances have you got in mind? > My favorite awkward environment to cite is running a command remotely over ssh. Dude, if you run remote commands by calling them through SSH, you didn't just got things backward: you fucked things up heavily. SSH was never ever designed as a batch, unsupervised tool, despite many people using it as such. Remote code that is parametrized should be run exactly as that: as a remote procedure call, a technique known for over thirty years now. One of the reasons is quoting (because for non-interactive SSH call the command needs to be quoted exactly twice if in shell, and exactly once when run from exec()), but there are problems with distributing keys, maintaining usable home directories, and disabling unnecessary things that are enabled by default (port forwarding, among the others), and that doesn't exhaust the list of issues. Proper RPC protocol, like XML-RPC (which was released twenty years ago and is still usable while being quite simple), covers quoting -- or actually, serializing data -- without programmers worrying if they got their list of metacharacters right and did enough passes for things to work correctly. On the other hand, I'm not surprised that people do this through SSH (and a variant of this stupidity: adding apache user to sudoers, so a web panel can add firewall rules). After all, I've never seen an easy to use RPC server that has all the procedures passed in its configuration. I needed to write such thing myself (once in Perl, as xmlrpcd, and recently in Python, with custom protocol that can do a little more, as harpd of HarpCaller project).

subfnsnx 10y ago

Como hackear

rusanu 10y ago

Looking now at how Windows handles console application arguments, it sure looks broken. But you have to put your mindset in cca. 1990 and think what Windows applications looked like back then, and what was the model Microsoft was betting on. Arguments were passed via DDE[0], and then later all the bets were on OLE[1] and finally COM[2]. System components were all the time accessed via in-process DLLs communicating with services over LRPC[3]. In this world, the command line, the pipe philosophy and the 'less is more' mindset were not only not welcome, they were the adversary. Even when finally it was aknowledged that the command shell needs some love too, the answer was PowerShell, which yet again defined an object interface between cmdlets[4]. [0] https://en.wikipedia.org/wiki/Dynamic_Data_Exchange https://en.wikipedia.org/wiki/Dynamic_Data_Exchange [1] https://en.wikipedia.org/wiki/Object_Linking_and_Embedding https://en.wikipedia.org/wiki/Object_Linking_and_Embedding [2] https://en.wikipedia.org/wiki/Component_Object_Model https://en.wikipedia.org/wiki/Component_Object_Model [3] https://en.wikipedia.org/wiki/Local_Procedure_Call https://en.wikipedia.org/wiki/Local_Procedure_Call [4] https://en.wikipedia.org/wiki/PowerShell#Pipeline https://en.wikipedia.org/wiki/PowerShell#Pipeline

sukilot 10y ago
That's a long way of saying that Microsoft spent many years promoting a overcomplicated bad ideas over simple correct ideas.
- rdslw 10y ago
  So true. After 25yrs using pcs/unix/macs, I've come to conclusion that it's deliberate on Microsoft side. They create much more closed, difficult and obscure technology which needs, training, certification and lot of care (services provided by Microsoft).
  - quotemstr 10y ago
    Trust me: it's not deliberate. There is no conspiracy. Everyone I met on Windows is at least as well-intentioned as anyone working in the POSIX world. The reason Windows has some bad APIs is the same reason Unix has bad APIs: someone bootstraps a system quickly and doesn't see the problems that can arise from their choice of APIs; the system becomes wildly successful; and now everyone has to support these ill-conceived APIs. Sure, Windows command-line argument passing is bad, but have you ever tried using wait/waitpid/wait4/waitid/etc.? That's a nightmare in the POSIX world; Windows has nice, clean process handles, not the garbage /proc stuff that makes it fundamentally impossible to write a safe pkill(1). If you're writing a brand-new system, for the love of God, do a good job of designing the APIs. You will not have a chance to go back and fix the APIs later.

jaclaz 10y ago

[2011]

grymoire1 10y ago

Oh. I was confused until I realized the publishing site was Microsoft. Apparently "Everyone" only refers to Windows programmers, and Unix/Mac/whatever programmers do not exist in this universe.

colemickens 10y ago
Looks like the title has been fixed but I was also confused before this thread filled up with comments confirming my suspicions that it was a Windows-specific issue. It does seem rather misleading to say "everyone" does it wrong when it's specifically a problem with Windows APIs (though not terribly surprising from the Windows team).
chc 10y ago
It's an MSDN article. Most of the articles on Apple's developer site are equally inapplicable to Windows development without explicitly saying so.

wfunction 10y ago

I recall seeing different behavior between the C runtime and CommandLineToArgvW. I don't remember what the difference was, but I remember it driving me nuts.

saurik 10y ago

A friend of mine had to fix this recently for PowerShell, which had a regression that caused arguments you passed to programs to be incorrectly escaped to executed commands. https://github.com/PowerShell/PowerShell/pull/2182 https://github.com/PowerShell/PowerShell/pull/2182 It is extremely common to get this wrong. Apache Portable Runtime even gets this wrong :/. (I haven't submitted a patch for this yet, but I intend to: I ran into it a couple months ago and then got distracted after working around it in my program by predicting what incorrect escaping might be performed by APR and compensating by adding quotes and escape characters to my input to their open process function... my build is statically linked so I don't feel bad about this temporary hack ;P.)

quotemstr 10y ago
Wow. Microsoft really has changed for the better since I left. Not a coincidence, I'm sure. ;-)

vram22 10y ago

IIRC, Microsoft C (quite some years ago - had worked on a product using it) had different variants of functions to spawn or execute a process from another one - like the exec family - execlp, execvp, execvpe, execlpe, etc., which varied in things like fixed vs. variable number of args, checking environment vs. not, etc. I also remember reading about it in a Waite Publications book, The Microsoft C Bible, by Nabajyoti Barkakati. Not sure if the issues mentioned in the OP could be solved if those functions were present - need to check. Edit: and the DOS / Windows exec family of functions was likely derived from Unix's exec().

quotemstr 10y ago

Oh, hi. This is my article. I'm a bit sad that I lost edit rights when I left Microsoft.

ronsor 10y ago
You should always host your own blog and then post a summary on the MSDN blogs -- always have control over content

Pxtl 10y ago

What I find frustrating is how many MS tools freak at the sight of a quoted path when quoted paths are what "copy filename with path" (or whatever the context command is called) gives.

deleted 10y ago

[deleted]

ihaveahadron 10y ago

Unless it's all in my head , there are a lot of things that, everyone, is doing the wrong way. Maybe read a book from someone that did it the right way and is dead now, that doesn't mean you can't read new things . Pussies

kazinator 10y ago

This should be "quoting command line arguments the right way for passage into applications developed using Microsoft Visual C, and linked to its C run-time library that parses the command line string and calls main or wmain". There is no general correct way to quote command line arguments in Windows, because every application receives just a character string which it parses however it wants. There is no single specification for the syntax by which arguments are delimited within the command string.