4 ms·

The Tar Pipe (2010)

10 points by callum85 12y ago

gwern 12y ago
A nice writeup, but I'm curious what the tar pipe gains over a more normal way of copying directories like 'cp -a'. (Or is that what it means by 'mostly obsolete'?)
- callum85 12y ago
  For some reason, 'tar|tar' is many times faster than 'cp -a' (at least, when you're copying thousands of small files in deeply nested directories, on OS X). I noticed this once through experimenation, tried to find out why, and never got a good answer. The other thing is that hard-linked files are apparently handled differently: http://stackoverflow.com/questions/316078/interesting-usage-of-tar-but-what-is-happening http://stackoverflow.com/questions/316078/interesting-usage-...
- rakoo 12y ago
  I think the "pipe" part is more important than the "tar" part. It illustrates how processes can cooperate through the use of a standard communication tool and very simple interfaces (the more I read the article, the more I saw the parallel with Go's io.Copy, io.Reader and io.Writer). Once you got how useful piping is, you understand why you can do something as powerful as: machineA: tar -cf - dir | gz | ncat -l -p 6789 machineB: ncat machineA 6789 | gunzip | tar -xf - And here you go, a dir is sent compressed through network with the power of little tools and a good inter-communication.
JadeNB 12y ago
> stdin, stdout, and stderr are files 0, 1, and 2. Always. Shouldn't this be something closer to "stdin, stdout, and stderr are always initially fds 0, 1, and 2"? I thought that one could play some trickery to change this. (I'm not being intentionally snarky, and might be quite wrong.)
- falcolas 12y ago
  They can, yes. If you close all of them, the next files you open will get the 0, 1, and 2 file descriptors. It's a fairly common idiom to close everything, open a handle to /dev/null, and duplicate it twice to get 0, 1, and 2 pointing at /dev/null.
LeoPanthera 12y ago
What's the point of "-f -"? Why not just omit -f entirely?
- crest 12y ago
  Most tar implementations default to the first streamer (e.g. sa0) instead of stdout.

The Tar Pipe (2010)

10 points by callum85 12y ago

gwern 12y ago

A nice writeup, but I'm curious what the tar pipe gains over a more normal way of copying directories like 'cp -a'. (Or is that what it means by 'mostly obsolete'?)

callum85 12y ago
For some reason, 'tar|tar' is many times faster than 'cp -a' (at least, when you're copying thousands of small files in deeply nested directories, on OS X). I noticed this once through experimenation, tried to find out why, and never got a good answer. The other thing is that hard-linked files are apparently handled differently: http://stackoverflow.com/questions/316078/interesting-usage-of-tar-but-what-is-happening http://stackoverflow.com/questions/316078/interesting-usage-...
rakoo 12y ago
I think the "pipe" part is more important than the "tar" part. It illustrates how processes can cooperate through the use of a standard communication tool and very simple interfaces (the more I read the article, the more I saw the parallel with Go's io.Copy, io.Reader and io.Writer). Once you got how useful piping is, you understand why you can do something as powerful as: machineA: tar -cf - dir | gz | ncat -l -p 6789 machineB: ncat machineA 6789 | gunzip | tar -xf - And here you go, a dir is sent compressed through network with the power of little tools and a good inter-communication.

JadeNB 12y ago

> stdin, stdout, and stderr are files 0, 1, and 2. Always. Shouldn't this be something closer to "stdin, stdout, and stderr are always initially fds 0, 1, and 2"? I thought that one could play some trickery to change this. (I'm not being intentionally snarky, and might be quite wrong.)

falcolas 12y ago
They can, yes. If you close all of them, the next files you open will get the 0, 1, and 2 file descriptors. It's a fairly common idiom to close everything, open a handle to /dev/null, and duplicate it twice to get 0, 1, and 2 pointing at /dev/null.

LeoPanthera 12y ago

What's the point of "-f -"? Why not just omit -f entirely?

crest 12y ago
Most tar implementations default to the first streamer (e.g. sa0) instead of stdout.