I wasn't suggesting to blindly use a tar pipe for as many files as the OP was trying to address. My main point was that the tar pipe (locally or over ssh) is su

I wasn't suggesting to blindly use a tar pipe for as many files as the OP was trying to address. My main point was that the tar pipe (locally or over ssh) is superior to cp/scp for the sorts of things that many of us do regularly (as discussed elsewhere in this thread), such as copying tens of thousands to low millions of mostly smallish files locally or remotely and wanting to preserve the structure of links and file metadata, and that pax looks like a simpler alternative to the tar pipe that accomplishes the same thing in a single command, although it does still use a hash table internally. I wonder if using something other than a hash table would be worthwhile for cases like the OP though, or if there is a tool that has that option -- the memory overhead of empty buckets and resizing seem like serious downsides when you're trying to handle as many files as the OP.