4 ms·

I would usually use the tarpipe mentioned already by others for this sort of thing (although I probably wouldn't do 432 million files in one shot): (cd $SOUR

by calvins 12y ago

I would usually use the tarpipe mentioned already by others for this sort of thing (although I probably wouldn't do 432 million files in one shot): (cd $SOURCE && tar cf - .) | (mkdir -p $DEST && cd $DEST && tar xf -) Another option which I just learned about through reading some links from this thread is pax (http://en.wikipedia.org/wiki/Pax_%28Unix%29 http://en.wikipedia.org/wiki/Pax_%28Unix%29), which can do it with just a single process: (mkdir -p $DEST && cd $SOURCE && pax -rw . $DEST) Both will handle hard links fine, but pax may have some advantages in terms of resource usage when processing huge numbers of files and tons of hard links.

tedunangst 12y ago
You know how tar handles hardlinks, right? By creating a giant hash table of every file.
- dredmorbius 12y ago
  How's that going to scale with memory? In-memory hash tables were the downfall of cp here.
  - tedunangst 12y ago
    It's going to scale just like you'd imagine it would. All the people saying "oh, tar was built for this" obviously haven't actually tried replicating the experiment using tar.
    - dredmorbius 12y ago
      Pretty much as I'd suspected.
- calvins 12y ago
  I wasn't suggesting to blindly use a tar pipe for as many files as the OP was trying to address. My main point was that the tar pipe (locally or over ssh) is superior to cp/scp for the sorts of things that many of us do regularly (as discussed elsewhere in this thread), such as copying tens of thousands to low millions of mostly smallish files locally or remotely and wanting to preserve the structure of links and file metadata, and that pax looks like a simpler alternative to the tar pipe that accomplishes the same thing in a single command, although it does still use a hash table internally. I wonder if using something other than a hash table would be worthwhile for cases like the OP though, or if there is a tool that has that option -- the memory overhead of empty buckets and resizing seem like serious downsides when you're trying to handle as many files as the OP.

by calvins 12y ago

tedunangst 12y ago

You know how tar handles hardlinks, right? By creating a giant hash table of every file.

dredmorbius 12y ago
How's that going to scale with memory? In-memory hash tables were the downfall of cp here.
- tedunangst 12y ago
  It's going to scale just like you'd imagine it would. All the people saying "oh, tar was built for this" obviously haven't actually tried replicating the experiment using tar.
  - dredmorbius 12y ago
    Pretty much as I'd suspected.
calvins 12y ago
I wasn't suggesting to blindly use a tar pipe for as many files as the OP was trying to address. My main point was that the tar pipe (locally or over ssh) is superior to cp/scp for the sorts of things that many of us do regularly (as discussed elsewhere in this thread), such as copying tens of thousands to low millions of mostly smallish files locally or remotely and wanting to preserve the structure of links and file metadata, and that pax looks like a simpler alternative to the tar pipe that accomplishes the same thing in a single command, although it does still use a hash table internally. I wonder if using something other than a hash table would be worthwhile for cases like the OP though, or if there is a tool that has that option -- the memory overhead of empty buckets and resizing seem like serious downsides when you're trying to handle as many files as the OP.