4 ms·

Unless your app doesn't generate output,then grepping the output will not work if it's stderr and it will be confusing

bombcar 2y ago
You can learn the 2>&1 thing, but it would be nice if there was a feature of one of the greps to "slurp up" the error output. Of course, the real problem is there's no standard, the standards that do exist are ignored, and each new command-line tool generates a new standard.
- sudobash1 2y ago
  It generally isn't possible for grep to "slurp up" the error output because stderr does not get passed through pipes (by default). This is shell behavior and grep cannot do anything about it. As a side note, some shells have started implementing a shorthand for `foo 2>&1 | grep` which is `foo |& grep`.
  - ElevenLathe 2y ago
    Probably you could do a hack like finding the program at the other end of the pipe and looking at its open file descriptors, but lots of corner cases to consider.
  - kelnos 2y ago
    Right, the problem is that apparently shells haven't seen user need for selectively redirecting one or the other. Like I might want to see foo's stdout on the terminal, but pipe stderr (and only stderr) to grep, perhaps so I only see some specific error messages that I care about. `2>&1` will send stderr to stdout, and then the pipe will send everything to grep (and I believe `|&` does the same). (To be fair, that does seem like a pretty rare use case.)
    - sudobash1 2y ago
      Actually, you can. If you run: foo 2>&1 >&- | grep ... then you will grep only the stderr. The stdout will be discarded. Alternatively, if you only want to grep to see the stderr, but you still want to see stdout, you can swap stderr and stdout like this: foo 3>&2 2>&1 1>&3 | grep ... And in many shells, you can easily split the stdout and stderr into two separate pipelines like this: foo > >(grep stdout) 2> >(grep stderr)
    - wahern 2y ago
      > Like I might want to see foo's stdout on the terminal, but pipe stderr (and only stderr) to grep, perhaps so I only see some specific error messages that I care about You mean this: `foo 3>&1 1>&2 2>&3` The above swaps stdout and stderr, saving + restoring the original stdout through descriptor 3. Shell redirection expressions map directly to dup2 syscalls, except the operands are reversed: 3>&1 is dup2(1, 3). Pipes map fairly simply to fork + exec. At its core the shell is a rather thin wrapper around the core Unix syscalls fork, exec, dup2, open, and close. If you look at the original shell implementation, the command parser more-or-less executes these syscalls as it goes along, left to right.
matheusmoreira 2y ago
How can you grep output if there's no output? Can you provide a concrete example of such an application?
- d3VwsX 2y ago
  Some applications print usage information (e.g. with --help flag) to stderr. That is a bit annoying when there is a lot of output and I just want to grep for some flag, since the first attempt will fail and then I have to try again after adding 2>&1.
  - quectophoton 2y ago
    Then there's `ssh-keyscan`, printing the "real" output to stdout, but also printing "comments" to stderr. Nothing wrong with that, but it was a bit confusing the first time I saw that behavior.
  - matheusmoreira 2y ago
    If I pass the --help flag, I'm explicitly asking the program to print the help text. In this case it's appropriate for the help text to be written to standard output since the help text is the output. If the user passes invalid options and inputs, then the program should produce no output and any error or help messages should be written to the error stream. It's important that these messages end up on the terminal when one gets the command invocation wrong while trying to pipe data into another program. If they're written to standard output, the program on the other end of the pipe will slurp all the error messages up and try to parse them as though they were valid inputs.
- quectophoton 2y ago
  I think what they meant was something like: > Unless your app doesn't generate [stdout] output,then grepping the [visible] output will not work if it's stderr and it will be confusing With the first "output" meaning "specifically stdout", and the second meaning "what you see on the terminal". As in, you run a command, see it printed a lot of stuff, so then you do it again but piping it to `grep` or `less`, and realize none of that output was actually going to stdout.

You can learn the 2>&1 thing, but it would be nice if there was a feature of one of the greps to "slurp up" the error output. Of course, the real problem is there's no standard, the standards that do exist are ignored, and each new command-line tool generates a new standard.

sudobash1 2y ago
It generally isn't possible for grep to "slurp up" the error output because stderr does not get passed through pipes (by default). This is shell behavior and grep cannot do anything about it. As a side note, some shells have started implementing a shorthand for `foo 2>&1 | grep` which is `foo |& grep`.
- ElevenLathe 2y ago
  Probably you could do a hack like finding the program at the other end of the pipe and looking at its open file descriptors, but lots of corner cases to consider.
- kelnos 2y ago
  Right, the problem is that apparently shells haven't seen user need for selectively redirecting one or the other. Like I might want to see foo's stdout on the terminal, but pipe stderr (and only stderr) to grep, perhaps so I only see some specific error messages that I care about. `2>&1` will send stderr to stdout, and then the pipe will send everything to grep (and I believe `|&` does the same). (To be fair, that does seem like a pretty rare use case.)
  - sudobash1 2y ago
    Actually, you can. If you run: foo 2>&1 >&- | grep ... then you will grep only the stderr. The stdout will be discarded. Alternatively, if you only want to grep to see the stderr, but you still want to see stdout, you can swap stderr and stdout like this: foo 3>&2 2>&1 1>&3 | grep ... And in many shells, you can easily split the stdout and stderr into two separate pipelines like this: foo > >(grep stdout) 2> >(grep stderr)
  - wahern 2y ago
    > Like I might want to see foo's stdout on the terminal, but pipe stderr (and only stderr) to grep, perhaps so I only see some specific error messages that I care about You mean this: `foo 3>&1 1>&2 2>&3` The above swaps stdout and stderr, saving + restoring the original stdout through descriptor 3. Shell redirection expressions map directly to dup2 syscalls, except the operands are reversed: 3>&1 is dup2(1, 3). Pipes map fairly simply to fork + exec. At its core the shell is a rather thin wrapper around the core Unix syscalls fork, exec, dup2, open, and close. If you look at the original shell implementation, the command parser more-or-less executes these syscalls as it goes along, left to right.

matheusmoreira 2y ago

How can you grep output if there's no output? Can you provide a concrete example of such an application?

d3VwsX 2y ago
Some applications print usage information (e.g. with --help flag) to stderr. That is a bit annoying when there is a lot of output and I just want to grep for some flag, since the first attempt will fail and then I have to try again after adding 2>&1.
- quectophoton 2y ago
  Then there's `ssh-keyscan`, printing the "real" output to stdout, but also printing "comments" to stderr. Nothing wrong with that, but it was a bit confusing the first time I saw that behavior.
- matheusmoreira 2y ago
  If I pass the --help flag, I'm explicitly asking the program to print the help text. In this case it's appropriate for the help text to be written to standard output since the help text is the output. If the user passes invalid options and inputs, then the program should produce no output and any error or help messages should be written to the error stream. It's important that these messages end up on the terminal when one gets the command invocation wrong while trying to pipe data into another program. If they're written to standard output, the program on the other end of the pipe will slurp all the error messages up and try to parse them as though they were valid inputs.
quectophoton 2y ago
I think what they meant was something like: > Unless your app doesn't generate [stdout] output,then grepping the [visible] output will not work if it's stderr and it will be confusing With the first "output" meaning "specifically stdout", and the second meaning "what you see on the terminal". As in, you run a command, see it printed a lot of stuff, so then you do it again but piping it to `grep` or `less`, and realize none of that output was actually going to stdout.