4 ms·

> sponge Why not use > file? > mispipe In bash there's PIPESTATUS for that.

• charliesome 6y ago
  Re sponge: the shell will open the output file for writing before invoking the command so in the example joey provides, /etc/passwd will be an empty file by the time sed opens it.
  • samatman 6y ago
    heh, yeah that's a mistake everyone makes once, and hopefully not more than once.
  • prakashdanish 6y ago
    That makes sense but in the example provided, why can't we use sed with the in-line flag?
    • gpanders 6y ago
      Because the -i flag works differently on GNU sed vs non-GNU sed and there is no way to use it portably.
• gravypod 6y ago
  > Why not use > file? I'm assuming sponge buffers everything in memory so you don't get concurrent modification bugs.
  • masklinn 6y ago
    Yes, if you try to sed a file back to itself for instance you'll get a truncated file (or maybe a no-op?). In fact, that's exactly the example sponge's manpage uses.
    • lloeki 6y ago
      That redirection will make the shell truncate the file long before sed has a chance to open it.
    • moonchild 6y ago
      It's a truncated file: $ cat test moo $ cat < test > test $ cat test $
      • Izkata 6y ago
        I was wondering about tee for that: $ cat test moo $ cat < test | tee test > /dev/null $ cat test moo ...seems to work. Am I just getting lucky with a race condition?
        • notimetorelax 6y ago
          Possible, since you don’t pipe into the file but instead ask tee to write into it.
        • Erwin 6y ago
          Yes, check this, with f.txt containing 99999 numbers: $ seq 1 99999 > f.txt $ cat < f.txt | tee f.txt > /dev/null $ wc -l f.txt 23696 f.tx
        • moonchild 6y ago
          > Am I just getting lucky with a race condition? Probably this is buffering, either from dd or from the kernel. $ dd if=/dev/urandom bs=1M count=1 of=test.bin 2>/dev/null $ cp test.bin test2.bin $ cat < test2.bin | tee test2.bin >/dev/null $ diff test.bin test2.bin Binary files test.bin and test2.bin differ $ wc -c test{2,}.bin 131072 test2.bin 1048576 test.bin Notice the file got truncated at exactly 128k; a nice round size for a write cache.
        • kasabali 6y ago
          I think using tac | tac is a safe bet since must be reading all file before by definition.
• wodenokoto 6y ago
  I think this is a good question as moreutils doesn’t do a very clear job of explaining it. Sponge lets you do a series of piped transformations on 1 or more files and overwrite the result back to the original files. Using > things will just break. I’m guessing this also means you can’t pipe something bigger than what your memory can hold.
• semicolonandson 6y ago
  Great question. There's a good Stack Overflow answer here: https://unix.stackexchange.com/questions/207919/sponge-from-moreutils-whats-the-difference-to-shell-redirect-useful-examples https://unix.stackexchange.com/questions/207919/sponge-from-...
• GuB-42 6y ago
  > Why not use > file? This, I learned the hard way, and I am not the only one. So in order for you not to make that mistake: the first thing that happens is that "file" is truncated, that happens before any command is run, so sed s/foo/bar/ file > file will always result in an empty "file", because it will be truncated before "sed" is run. I lost a couple of hours of work like that, once, and then I learned. From the manpage of sponge sponge reads standard input and writes it out to the specified file. Unlike a shell redirect, sponge soaks up all its input before opening the output file. This allows constricting pipelines that read from and write to the same file. So, the command sed s/foo/bar/ file | sponge file will do what you expect
  • atsaloli 6y ago
    Try sed s/foo/bar/ -i file to edit file.
    • GuB-42 6y ago
      That what I do now, one thing to note though is that the "-i" option is not always available. Usually, now, it is, but we still have a couple of Solaris servers at work that don't support it. This, btw, is why I hate shell scripts. There are so many variants of bourne shells and UNIX tools that writing a portable script is a minefield, as if properly dealing with spaces wasn't tricky enough...
      • atsaloli 6y ago
        Right you are. The -i option is not standard (https://pubs.opengroup.org/onlinepubs/9699919799/utilities/sed.html https://pubs.opengroup.org/onlinepubs/9699919799/utilities/s...)