5 ms·
Efficient File Copying on Linux
- ars 10y agoExplanation concludes readahead is the reason that 128KB buffer is fastest on the benchmark, while the benchmark uses /dev/zero and /dev/null which don't have readahead. You need to redo this article using actual reads and writes. Try it with both a quiet machine and a semi-busy one.
- Scaevolus 10y agoI suspect this has more to do with the balance between L2 cache sizes and syscall overhead than anything else. Dumping nothing into nothing at 40GB/s is unacceptable, I need it to be 60GB/s! With actual I/O devices the difference should be negligible. Sequential reads and writes are well optimized at every level, from the CPU to the HDD.
- jaimex2 10y agoGood explanation, thank you. So when using dd or other copy tools setting a block size of 128 kb should also be the best choice?
- heinrich5991 10y agoWhat about `copy_file_range(2)`?
- kevinoid 10y agoIt looks like the FICLONE ioctl was chosen over the copy_file_range syscall[1] for use in coreutils due to concerns about copy_file_range not preserving holes.[2] I agree, it would be interesting to know how these compare, both in filesystem implementation coverage and performance. 1. https://lwn.net/Articles/659523/ https://lwn.net/Articles/659523/ 2. https://debbugs.gnu.org/cgi/bugreport.cgi?bug=24399 https://debbugs.gnu.org/cgi/bugreport.cgi?bug=24399
- privacyfornow 10y agoor sendfile() for that matter.
- JoshTriplett 10y agoI'd be interested to see how this compares to 1) mmapping both files and using memcpy, 2) mmaping the source and making a single call to write passing the whole buffer, and 3) copy_file_range.
- pskocik 10y agoI once benchmarked reads with `mmap` vs `read` (http://stackoverflow.com/a/39196499/1084774 http://stackoverflow.com/a/39196499/1084774) . mmap starts winning big time once the file is >= 16KiB. Copying should have similar performance characteristics.
- dom0 10y agoSo after the last blog post by The Author which mainly showed The Author's lack of understanding, we have another article from The Author highlighting that he does indeed not understand things he writes blog posts about (incorrect rationale and assumptions about 128 KiB block size being optimal, no readahead on virtual device files, and of course not mentioning any of the FD splicing alternatives in a post titled "Efficient ..." or any of the approaches involving memory mappings and explicit prefetching on said mappings). I don't want to be overly extremely dismissive or arrogant here, but this post pretty much boils down to "128 KiB is optimal because that number appears somewhere else, too, and that other spot even has somehow something to do with I/O".
- akanet 10y agoWhile your criticisms about what the author's explanation lacks or does not consider are appreciated, I think this is a wonderful opportunity for you to put forth your own understanding! I know very little about the topic and thus found the article interesting. I would happily learn more.
- ars 10y agoLarger blocksizes are faster, but take more memory and interleave badly with other tasks since they are all-or-nothing. A large block can take a while and nothing else can read or write while it's running. That's about it. You want a blocksize sized so that it takes around 10 ms to write to disk. So around 128K to 1MB, depending on the underlying hardware.
- dredmorbius 10y agoYour 2nd paragraph's characterisation doesn't correspond to the article, which specifically cites empirical testing of multiple blocksizes and a demonstration of optimal performance at 128 KB: "The author of this file did a benchmark using dd if=/dev/zero of=/dev/null with different values of the bs (block size) parameter. On a wide variety of systems, including older Intel CPUs, modern high-end Intel CPUs, and even an IBM POWER7 CPU, a 128 KB block size is fastest. I used gnuplot to graph these results, shown below (higher transfer rate is better). The different symbols represent different system configurations."
- amelius 10y agoFor even faster copying on the same device, use a copy-on-write (COW) filesystem. (I wonder though what API the "cp" command would use to accomplish that).
- dom0 10y ago> (I wonder though what API the "cp" command would use to accomplish that). link(2)
- icebraining 10y agoThat's not copy-on-write, that's just giving an extra name to the same underlying file/inode.
- dom0 10y agoIt isn't (though it is CoW towards atomic file writing), and my comment was a bit tongue-in-cheek, since there really is nothing close to a portable API to do that. Linux recently gained a copy_file_range syscall which is part of the VFS - so a CoW FS or an FS with extent sharing should be able to implement the expected semantics here. Linux also has a bunch of ioctls that directly manipulate extents, like the FICLONERANGE mentioned above or FIDEDUPERANGE. For these to work reliably you need a lot of not-easily available information (in the general case)... these interfaces are comparable in application impl complexity to, say, O_DIRECT.
- icebraining 10y agoGNU cp uses ioctl() with the FICLONE flag when you run "cp --reflink" on a COW filesystem like Btrfs. http://man7.org/linux/man-page/man2/ioctl_ficlonerange.2.html http://man7.org/linux/man-page/man2/ioctl_ficlonerange.2.htm...
- simcop2387 10y ago> GNU cp uses ioctl() with the FICLONE flag when you run "cp --reflink" on a COW filesystem like Btrfs. > http://man7.org/linux/man-page/man2/ioctl_ficlonerange.2.htm.. http://man7.org/linux/man-page/man2/ioctl_ficlonerange.2.htm.... This unfortunately doesn't yet work with zfs on Linux. One of my only gripes with it.
- LeoPanthera 10y agoThe most efficient way to copy a large number of small files is often to use a tarpipe. What block size does "tar" use? And for that matter, "nc", as a tarpipe through nc is a super fast way to move data between machines.
- valarauca1 10y agoI doubt these benchmarks are relevant anymore as Linux has a system just dedicated to copying files. So you never even have to leave the page cache, let alone copy into user space. OFC it was implemented post 4.0 so I doubt GLibc supports it therefore the whole world pretends it doesn't exist.
- jquast 10y agoThe statvfs system call indicates the preferred block size of the filesystem. It is a very large value on zfs, for example. https://docs.python.org/2/library/statvfs.html#statvfs.F_BSIZE https://docs.python.org/2/library/statvfs.html#statvfs.F_BSI...