5 ms·
What's the benefit of using a tarpipe locally?
by hnha 12y ago
What's the benefit of using a tarpipe locally?
- brazzy 12y agoNo giant bookkeeping datastructures that end up with the process thrashing in swap, because tar is designed to work with so many files that you don't want to keep the metadata in memory.
- makmanalp 12y agoThis is true if the bookkeeping was irrelevant and unnecessary. I imagine the scenario where your copy gets stopped in the middle is worse when you use the tar method.
- brazzy 12y agoIt's not as if cp gives you anything more useful in that case. The bookkeeping that caused the problem was a hashtable of files that have been copied, necessary to handle hard links. Would be interesting how tar does that; if it can't do any better then the root comment and mine above are wrong.
- tedunangst 12y agoHow does tar keep track of every hardlink without a giant hash table? It doesn't. It's not magic.
- thaumaturgy 12y agotarpipes can handle gnarly piles of hardlinked files without sweating. BackupPC is free software for network backups that builds a massively hardlinked file pool; tarpipes are the only thing that can reliably copy BackupPC pools from one location to another.
- rbh42 12y agoNot so. Both the sending and the receiving tar processes will need a data structure keeping track of which inodes they've already processed. They can skip inodes with a link count of 1, but if all inodes have multiple links (as in this case), the overhead will be twice that of a single cp process.