4 ms·

The fastest `cp` is actually doing no data copies at all (relying on copy-on-write), on filesystems with reflink support. Incidentally, coreutils v9.0 cp switch

The fastest `cp` is actually doing no data copies at all (relying on copy-on-write), on filesystems with reflink support. Incidentally, coreutils v9.0 cp switched to doing reflinks by default [1], so there's already a faster cp. [1] https://git.savannah.gnu.org/cgit/coreutils.git/commit/?id=25725f9d41735d176d73a757430739fb71c7d043 https://git.savannah.gnu.org/cgit/coreutils.git/commit/?id=2...

egberts1 5y ago
True. But, but, … BUT when it comes to files found on differing (physical/logical/virtual) disk partitions, ye gonna need that block copying mechanism within ‘cp’.
- kerneltrap 5y ago
  Not necessarily, while reflink works only within the boundaries of the same filesystem there are other optimizations that can operate across filesystems (even networked ones). For example copy_file_range() [1] allows for copy offloading (depending on the underlying fs/storage), which was incidentally also added to coreutils v9.0 [2]. There are also optimizations on other layers, e.g. transparent compression (e.g. btrfs) and online block-level dedup (e.g. dm-vdo), that can aid in minimizing and/or avoiding actual block copies. When all else fails, cp will anyway fallback to buffered IO, so it won't be bound by the rate of IO (as long as writeback can keep up with memory pressure). Of course improvements on all layers are always welcome and there are certainly workloads that can benefit from an optimized raw throughput, low-latency and scalability. Demanding workloads are typically using dedicated IO stacks and bypass the kernel altogether (e.g SPDK), but the amazing efforts of Axboe and everyone else in the kernel community have been continuously bringing linux up to par in terms of IO performance. [1] https://man7.org/linux/man-pages/man2/copy_file_range.2.html https://man7.org/linux/man-pages/man2/copy_file_range.2.html [2] https://git.savannah.gnu.org/cgit/coreutils.git/commit/?id=4b04a0c3b792d27909670a81d21f2a3b3e0ea563 https://git.savannah.gnu.org/cgit/coreutils.git/commit/?id=4...
  - egberts1 5y ago
    Jen Axboe Rocks!

egberts1 5y ago

True. But, but, … BUT when it comes to files found on differing (physical/logical/virtual) disk partitions, ye gonna need that block copying mechanism within ‘cp’.

kerneltrap 5y ago
Not necessarily, while reflink works only within the boundaries of the same filesystem there are other optimizations that can operate across filesystems (even networked ones). For example copy_file_range() [1] allows for copy offloading (depending on the underlying fs/storage), which was incidentally also added to coreutils v9.0 [2]. There are also optimizations on other layers, e.g. transparent compression (e.g. btrfs) and online block-level dedup (e.g. dm-vdo), that can aid in minimizing and/or avoiding actual block copies. When all else fails, cp will anyway fallback to buffered IO, so it won't be bound by the rate of IO (as long as writeback can keep up with memory pressure). Of course improvements on all layers are always welcome and there are certainly workloads that can benefit from an optimized raw throughput, low-latency and scalability. Demanding workloads are typically using dedicated IO stacks and bypass the kernel altogether (e.g SPDK), but the amazing efforts of Axboe and everyone else in the kernel community have been continuously bringing linux up to par in terms of IO performance. [1] https://man7.org/linux/man-pages/man2/copy_file_range.2.html https://man7.org/linux/man-pages/man2/copy_file_range.2.html [2] https://git.savannah.gnu.org/cgit/coreutils.git/commit/?id=4b04a0c3b792d27909670a81d21f2a3b3e0ea563 https://git.savannah.gnu.org/cgit/coreutils.git/commit/?id=4...
- egberts1 5y ago
  Jen Axboe Rocks!