5 ms·

The result of `cat a b > a` couldn't possibly be the result that the user intends, so it seems like poor design by definition. I can see how it would be tricky

The result of `cat a b > a` couldn't possibly be the result that the user intends, so it seems like poor design by definition. I can see how it would be tricky to handle it properly, but I do think it's possible to handle it properly without compromising the expressiveness of the shell.

marvy 11y ago
At first I was going to say that handling this properly would require shell to have magic powers. After trying to come up with a proof of why it's impossible, I think I found a simple solution. There are two problems the author was complaining about. The first one is: cat paper.* > paper.all This has the potential to use up the whole disk. The fix here is easy: do globbing before redirection. But this does not solve the other problem: cat a b > a This has a simple workaround which is even easier to type: cat a >> b. But the trap remains. How could the shell fix it? By pretending the user wrote this instead: cat a b > temp; mv temp a Let's look at the trade offs. We do a bit of extra work: one rename operation. This should be pretty cheap, and solves this problem as well as paper.all. The real trouble I think is coming up with a good name and location for tmpfile that is guaranteed not to mess up the operation of the command we're running. If we put it in the current directory, we get the wrong answer from ls. If we put it in /tmp, it will mess things up when we're working in /tmp. However, after doing a few minutes of research, it appears that this problem is already solvable on Linux: by using the O_TMPFILE flag, there is no need to give the file a name at all, until after the command finishes running.
- voyou 11y ago
  Sometimes I start a long-running process, redirecting it's output, and periodically look at the output file while this process is running. Your proposed solution wouldn't allow for that.
  - marvy 11y ago
    It would be possible, but not convenient: /proc/pid/fd/1. Ick. Ok, maybe the cure is worse than the disease. I'm out of clever ideas for now.
- hackmiester 11y ago
  What if 'a' is a device?
  - marvy 11y ago
    Ouch. Don't do that for devices. Or named pipes I guess. This idea has more problems than I thought at first.
pdkl95 11y ago
In modern bash, this case is already protected with the "noclobber" option (enabled with "set -C" or "set -o noclobber"). It is described in the "REDIRECTION" section of bash(1), subsection "Redirecting Output". It fails ">" redirections with an error message when it would clobber an existing file (you probably wanted to append to the file wiht ">>"). If you really intended this behavior, the new redirection operator '>|' has the old behavior. $ set -o noclobber $ echo foo > bar $ cat bar foo $ cat bar > bar bash: bar: cannot overwrite existing file $ echo baz > bar bash: bar: cannot overwrite existing file $ echo baz >| bar $ cat bar baz edit: fixed typo
- girvo 11y ago
  That's a nice feature, but it should definitely be turned on as a default rather than hidden behind an option.
  - pdkl95 11y ago
    That would break a very large number of existing scripts that depend on being able to replace a file with a traditional ">" redirection. You can set it as the default for your own interactive shell if you want by adding set -o noclobber to your ~/.bashrc file. It is probably a good idea to add it after the line that check if it is being used non-interactively, which probably looks something like this if [[ $- != *i* ]] ; then # Shell is non-interactive return fi # add interactive-shell-only settings here While you're editing .bashrc, you may want to also add these options that are available in modern bash that I find make the experience a lot nicer. # support new glob patterns like !(foo|bar|...) # for example, this move everything EXCPET the .h files # mv !(*.h) /some/dir/ shopt -s extglob # support ** in globs to match zero-or-more-recursive-directories shopt -s globstar # if you're annoyed at having to type "nohup" to prevent # programs from closing when you close the shell window shopt -u huponexit
    - girvo 11y ago
      Right, it definitely would break a lot of existing scripts, so I understand why its not a default. The thing is though, that really proves the argument that some parts of Unix have a bad out of the box user experience! I personally love Unix and find it amazingly powerful, but I've made the same mistakes as everyone and killed entire boxes, and really with some simple UX changes that could be fixed in the bulk of cases. Edit: I forgot to add, thanks for the detailed answer!
weland 11y ago
> The result of `cat a b > a` couldn't possibly be the result that the user intends The result of cat a b > a is precisely the result that the user intends, provided that the user has actually read the documentation.
- pskocik 11y ago
  If the user knew what they were doing they wouldn't have typed that. `cat a b > a` is the same as `cat b > a`. There's no reason to add the a before b except to confuse.
  - weland 11y ago
    It's in the context of why cat paper* > paper.all blows everything up. Of course it's meaningless.

marvy 11y ago

At first I was going to say that handling this properly would require shell to have magic powers. After trying to come up with a proof of why it's impossible, I think I found a simple solution. There are two problems the author was complaining about. The first one is: cat paper.* > paper.all This has the potential to use up the whole disk. The fix here is easy: do globbing before redirection. But this does not solve the other problem: cat a b > a This has a simple workaround which is even easier to type: cat a >> b. But the trap remains. How could the shell fix it? By pretending the user wrote this instead: cat a b > temp; mv temp a Let's look at the trade offs. We do a bit of extra work: one rename operation. This should be pretty cheap, and solves this problem as well as paper.all. The real trouble I think is coming up with a good name and location for tmpfile that is guaranteed not to mess up the operation of the command we're running. If we put it in the current directory, we get the wrong answer from ls. If we put it in /tmp, it will mess things up when we're working in /tmp. However, after doing a few minutes of research, it appears that this problem is already solvable on Linux: by using the O_TMPFILE flag, there is no need to give the file a name at all, until after the command finishes running.

voyou 11y ago
Sometimes I start a long-running process, redirecting it's output, and periodically look at the output file while this process is running. Your proposed solution wouldn't allow for that.
- marvy 11y ago
  It would be possible, but not convenient: /proc/pid/fd/1. Ick. Ok, maybe the cure is worse than the disease. I'm out of clever ideas for now.
hackmiester 11y ago
What if 'a' is a device?
- marvy 11y ago
  Ouch. Don't do that for devices. Or named pipes I guess. This idea has more problems than I thought at first.

pdkl95 11y ago

In modern bash, this case is already protected with the "noclobber" option (enabled with "set -C" or "set -o noclobber"). It is described in the "REDIRECTION" section of bash(1), subsection "Redirecting Output". It fails ">" redirections with an error message when it would clobber an existing file (you probably wanted to append to the file wiht ">>"). If you really intended this behavior, the new redirection operator '>|' has the old behavior. $ set -o noclobber $ echo foo > bar $ cat bar foo $ cat bar > bar bash: bar: cannot overwrite existing file $ echo baz > bar bash: bar: cannot overwrite existing file $ echo baz >| bar $ cat bar baz edit: fixed typo

girvo 11y ago
That's a nice feature, but it should definitely be turned on as a default rather than hidden behind an option.
- pdkl95 11y ago
  That would break a very large number of existing scripts that depend on being able to replace a file with a traditional ">" redirection. You can set it as the default for your own interactive shell if you want by adding set -o noclobber to your ~/.bashrc file. It is probably a good idea to add it after the line that check if it is being used non-interactively, which probably looks something like this if [[ $- != *i* ]] ; then # Shell is non-interactive return fi # add interactive-shell-only settings here While you're editing .bashrc, you may want to also add these options that are available in modern bash that I find make the experience a lot nicer. # support new glob patterns like !(foo|bar|...) # for example, this move everything EXCPET the .h files # mv !(*.h) /some/dir/ shopt -s extglob # support ** in globs to match zero-or-more-recursive-directories shopt -s globstar # if you're annoyed at having to type "nohup" to prevent # programs from closing when you close the shell window shopt -u huponexit
  - girvo 11y ago
    Right, it definitely would break a lot of existing scripts, so I understand why its not a default. The thing is though, that really proves the argument that some parts of Unix have a bad out of the box user experience! I personally love Unix and find it amazingly powerful, but I've made the same mistakes as everyone and killed entire boxes, and really with some simple UX changes that could be fixed in the bulk of cases. Edit: I forgot to add, thanks for the detailed answer!

weland 11y ago

> The result of `cat a b > a` couldn't possibly be the result that the user intends The result of cat a b > a is precisely the result that the user intends, provided that the user has actually read the documentation.

pskocik 11y ago
If the user knew what they were doing they wouldn't have typed that. `cat a b > a` is the same as `cat b > a`. There's no reason to add the a before b except to confuse.
- weland 11y ago
  It's in the context of why cat paper* > paper.all blows everything up. Of course it's meaningless.