53 ms·
Show HN: fcp – A significantly faster alternative to cp(1), written in Rust
- cratermoon 5y agoWhat filesystem on Linux? Show us some benchmarks on other filesystems.
- Svetlitski 5y agoGood question. I've already shut down the EC2 instance I used for testing, but I'll fire up another one later today and get back on this. Great point though, performance characteristics will almost certainly vary quite a bit between different filesystems.
- beebeepka 5y agoGenuine question: Is running EC2 instances easier than spawning a local VM?
- Svetlitski 5y agoNo, it isn't. My motivation for running on EC2 was that benchmarking on a laptop is somewhat noisy due to thermal constraints [1]. I specifically used a bare-metal EC2 instance in an attempt to have a less noisy environment (since VM performance is less predictable and depends on the other workloads running on the same hardware) with proper cooling. [1] https://lemire.me/blog/my-sayings/ https://lemire.me/blog/my-sayings/ Don't take my word for it, Prof. Lemire is the expert on performance.
- beebeepka 5y agoI totally buy your explanation. Consumer devices are extremely noisy. Back when I worked on embedded, the only tests that I could trust were running on the boxes
- cratermoon 5y agoI lost the link, but I remember reading an article in the last year or so about tricks for benchmarking on a laptop. Various things you could do to avoid the thermal throttling. The catch was that it was only possible to do approximate, comparative benchmarks, but never get a result that showed the "true" unconstrained performance.
- the8472 5y agoThe most important thing to do would be disabling turbo boost. On linux that's echo 1 > /sys/devices/system/cpu/intel_pstate/no_turbo # intel echo 0 > /sys/devices/system/cpu/cpufreq/boost # amd together with disabling noisy background services that's the most significant reduction in noise. There are further steps that can be taken but for CPU-bound microbenchmarks they only provide incremental improvements in my experience. IO benchmarks are trickier. SSDs have thermal throttling too, there's the OS write cache, the SLC cache performance cliff and other factors.
- cratermoon 5y agoThis discussion is starting to make me think that the industry needs benchmarks that take into account modern hardware behavior better, somehow. The only benchmarking I've done is on deployed software for comparative performance tuning, determining if a given change runs faster or slower as deployed. I've long been a skeptic of benchmarks that try to test across heterogenous systems. My dad, who was a systems programmer, helped teach me how benchmarks could be deceptive.
- pizza234 5y agoWhile that's correct, it's misleading in the big picture. On full load, any machine hits thermal constraints, so, considering clock changes a problem, a desktop is problematic as well. A desktop will certainly have a higher ceiling, and/or a laptop may have too much of a low one, but it's incorrect to assume that the problem is inherent to laptops. The solution is therefore, which is canonical in serious benchmarks and it's not mentioned, is to ensure that the machine runs on a fixed clock rate. This is easily accomplished, at least on desktops, by changing the BIOS. I wasn't personally able to do this from the O/S (Linux). Using a machine without control on the clock (which a cloud one can be, even if bare metal) is subject to generate inaccurate benchmark results.
- celrod 5y agoDepending on the scaling governor you're using, you should be able to adjust behavior from the command line: https://www.kernel.org/doc/html/v4.12/admin-guide/pm/cpufreq.html https://www.kernel.org/doc/html/v4.12/admin-guide/pm/cpufreq... https://www.kernel.org/doc/html/v4.12/admin-guide/pm/intel_pstate.html https://www.kernel.org/doc/html/v4.12/admin-guide/pm/intel_p... E.g., if you're using the Intel pstate driver in active mode, you can disable turbo via: echo "1" | sudo tee /sys/devices/system/cpu/intel_pstate/no_turbo (and re-enable by writing "0".)
- pizza234 5y agoThanks! I'll check the reference document to search for something that works. In the past, I had tried (to no avail) with the `userspace` governor, `cpufreq-set`, `cpupower` and also `/sys/*/cpufreq/scaling_max_freq`; hopefully, I'll find other info in the document.
- Svetlitski 5y agoI spun up another instance, and now know that the filesystem used on the Linux benchmarks is xfs.
- Svetlitski 5y agoInspired by many of the amazing new-wave Unix tools that have come about over the past several years (fzf, ripgrep, fd, lsd, etc.), I set out to create a more modern and performant version of the classic Unix cp command. Written in Rust due to the excellent ergonomics and performance characteristics of the language, fcp copies large files and directories in a fraction of the time that cp does. Feel free to ask me any questions about the project!
- jepler 5y ago_Why_ is it faster? (particularly on Linux, I see you're using syscalls on mac that mac's cp doesn't, since apple doesn't really maintain their unix userspace)
- Svetlitski 5y agoThe primary reason is that it uses all of your computer's available cores to walk through directories and copy files, effectively speeding things up by a factor directly proportional to how many cores you have. On macOS, this is augmented by regular files being copied using the OS's fclonefileat and fcopyfile mechanisms under the hood, which allows even very large files to be copied near-instantly (when compared to copying them block-by-block as classic cp does).
- CalChris 5y agoIs it available on homebrew?
- Svetlitski 5y agoNo, it is not. I would've submitted a Homebrew formula for it, but that would go directly against Homebrew's guidelines, which frown upon authors submitting formulas for their own tools [1]. [1] https://docs.brew.sh/Acceptable-Formulae#niche-or-self-submitted-stuff https://docs.brew.sh/Acceptable-Formulae#niche-or-self-submi...
- fulafel 5y ago
- qolop 5y agoAn even faster O(1) alternative ln -s /path/to/source /path/to/dest
- ViViDboarder 5y agoOr, even closer, drop the -s. That way deleting the source doesn’t delete the destination. That said, neither of these are truly the same as an edit to one will impact the other.
- 411111111111111 5y agoHe might've gotten confused by newer filesystems like btrfs which let you create links to files instantly which effectively work exactly like a copy. The second file however isn't actually created until a change has occurred, so the linking is only an entry in the catalog to the inodes i believe. i.e. btrfs snapshots use this feature.
- Klasiaster 5y agoThis can be enabled as available with the --reflink=auto flag for cp. When copying between different filesystems it still makes sense to add the --sparse=auto flag to keep a copy of a sparse file sparse, too. My alias for cp is: cp -i --reflink=auto --sparse=auto
- nextaccountic 5y agoDoes this support reflinks on filesystems that support deduplication? (like cp --reflink)
- CUViper 5y agoThe implementation of `fs::copy` uses `copy_file_range` on Linux, which does allow reflink optimizations.
- gregoriol 5y agoThe only thing I'd like "new" in cp is an optional progress indicator, for large files Btw, copying files is mostly bound by disk speed (or network if share), not so much by CPU, so it doesn't seem to be a valid reason to rewrite it in Rust for cpu performance...
- Y_Y 5y agoYou can get progress for cp by piping through pv from moreutils. What I do though is use rsync -P, which accepts most of cp syntax.
- _joel 5y agoI also use pv when dd'ing stuff. Super useful, no need to send USR1 signals
- selfhoster11 5y agoThat's for one file. It won't help for directories, unless you tar on one side and then untar on the other.
- the8472 5y agoIt looks like this is using multiple threads to copy files. That's fine for NVMe which tends to achieve more throughput with higher queue depths. But it will degrade performance when copying large files on spinning rust.
- idoubtit 5y agoMy experiments confirms your analyze. On a SATA hard drive using ext4, I copied a directory with a large hierarchy of file and 216MB overall. After 2 warm-ups, I launch this simple benchmark. time cp -r moodle X 13.76s s (0.06s u + 2.44s k) 18% CPU time fcp-0.1.0-x86_64-unknown-linux-gnu moodle X 46.12s s (0.15s u + 2.47s k) 3% CPU I emptied linux cache before each run with `echo 1 > /proc/sys/vm/drop_caches`, but this does not remove the HDD buffering. I suspect that fcp would be worse in real life conditions (no warm-up). This does not mean the fcp is bad or not performant. But it some cases, it can be 3x slower.
- kwdc 5y agoSince you've already started, can you do a benchmark of a pair of copies started at same time? One copy operation is the little files and deep hierarchy. The other copy operation is just a big file. Both operations start at same time and use the same tool. I'm curious what cp and fcp do to the cache as they operate.
- hpb42 5y agoWhat do you mean by "After 2 warm-ups"? I've seen this expression in the context of benchmarks before and it puzzles me. Does it mean you are doing this operation enough times to trick the kernel to cache everything in RAM? Or you run it 3 times and chose the fastest timings?
- snet0 5y agoI'm not sure about the specific context here, but benchmark warm-ups are often used to make sure that the program's run time is nothing but that, and doesn't include things like setting up an environment or compiling to a cache. I think there are sometimes arguments to be made that, for the user, that is run time. When using benchmarks to compare programs like this, though, I think isolating the "actual" (i.e. useful) program execution is productive.
- bb88 5y agoUnfortunately I find it's not enough to copy large files, you need to verify them as well. I don't trust the hard drive controller or the nvme controller to have not f'ed something up.
- tialaramex 5y agoYou're in the wrong abstraction layer. There are a whole bunch of intervening levels between the layer where you've imagined that you "need to verify them as well" and the layer where the "hard drive controller" that you don't trust lives. Those layers may conceal a real problem (if there is one) so that you can't see it even though it exists, or they can create problems, which you will then mistakenly blame on the "hard drive controller" even if it was not at fault. So you've created a situation where you can at best confuse yourself into not understanding what's actually going on, and at worst satisfy yourself nothing is wrong when actually there's a catastrophe awaiting you. The worst of all worlds.
- formerly_proven 5y agoFor SSDs this is probably the right thing (maybe use uring tho?) For hard drives this is the opposite of what you want. For hard drives, especially when copying large trees, you only want one thread to access the disk, and preferably sort its stat()s and open/read/close() by inode. Also this tool seems to lack most options that cp(1) has.
- vthriller 5y ago> preferably sort its stat()s and open/read/close() by inode A better approach would be to use information about location of data extents. Luckily there's already a crate that can help with that: https://github.com/the8472/platter-walk https://github.com/the8472/platter-walk
- formerly_proven 5y agoThat's better for reading the data, but doesn't work for ordering stat()s because looking up extent information requires reading the inode, but that's what we're trying to make linear-ish at that point.
- the8472 5y agoyou can optimize for both. obtain directory entries, sort by inode, do a batch stat (ideally via io_uring) and fiemap and then sort again by extent order. And if you only need the `d_type` instead of the full stat struct, then you get that for free from getdents(2) on some filesystems, this is what platter_walk does.
- CWuestefeld 5y agooptions that cp(1) has OK, this is something that's bugged me for a long time. I've been in the field forever, but working on Windows. So I've seen this kind of thing many times before, particularly when referencing man, and it seems completely cryptic. The way linux people occasionally put a parenthetic number after a generic command seems to me one of the ways that normal civilians, and even pros like me, are intimidated into staying away from serious linux usage. So what gives? What the heck is the parenthetic number for?
- hprotagonist 5y ago“rewrite (the easy parts of) $battle-tested-ancient-coreutil in rust” seems interesting as an exercise or as a way to learn a new language, but i’m not convinced of it’s broader utility.
- hexo 5y agohere, the utility could be proof of concept that can be implemented in original tool
- carlmr 5y agoBut then again there's ripgrep, which did just that and is much faster than grep and works perfectly.
- hprotagonist 5y agoi use ripgrep all the time, it's great. i don't think it was ever meant to be "exactly GNU/coreutils grep", and I don't think it ever even tried. The API is totally different, i don't think it's ever been the same, it scratches a similar-but-not-identical itch. https://github.com/BurntSushi/ripgrep/blob/master/FAQ.md#posix4ever https://github.com/BurntSushi/ripgrep/blob/master/FAQ.md#pos...
- Svetlitski 5y agoI would argue that "the easy parts" of cp account for a very significant fraction of its usage (i.e. copy some files/directories without any other options). The idea with fcp is to make that common use case very fast, not to replace cp and all the options it handles. I definitely can understand the fatigue with "rewrite it in Rust" though, as there is a lot of that. Something you might be interested in is that people have actually gone through the effort of rewriting the actual coreutils to spec in Rust (https://github.com/uutils/coreutils https://github.com/uutils/coreutils).
- tyingq 5y agoThere was a tool for Solaris called "mtwrite" that would use LD_PRELOAD to intercept writes and farm them out to threads. This way, it would work with not just cp, but also tar extracts to files, etc. http://www.maier-komor.de/mtwrite.html http://www.maier-komor.de/mtwrite.html
- michael-ax 5y agohttps://www.unix.com/man-page/mojave/2/fclonefileat/ https://www.unix.com/man-page/mojave/2/fclonefileat/ summary: this is a nice cow/cloning wrapper for use on a single ssd partition! -- but cloning not copying between volumes or partitions is .. and when you actually want to copy with fcp, it can be very slow as detailed in other comments.
- dekhn 5y agoI worked with a large cluster filesystem and it had some interesting properties. In particular, files were effectively append-only and immutable when closed, but the FS supported "snapshot for append" which allowed to to make a snapshot of the file, and append to it. Under the hood it managed all the pointers but when the FS designers found out we used the feature heavily they got worried.
- kccqzy 5y agoWhy do people think it is acceptable for utility programs like file copies to launch many threads and saturate my CPU? It's usually the case that there are many things being done on the system, and I don't want one task take up all the resources I have. Most classic Unix command line tools only use a single thread, so I know they won't bog down the entire system without any effort on my part (adjusting niceness, cgroups, ulimit, etc).
- slava_kiose 5y agoWhat file systems are supported here?
- kazinator 5y ago> The massive difference in performance in this case is due to fcp using fclonefileat and fcopyfile under the hood GNU Coreutils' cp, the thing you use on most GNU/Linux systems, is not the "Classic Unix cp". It has options for copy-on-write cloning via kernel-specific methods. COW copying is opt-in probably because it's not a true copy. If either file sustains damage, both are toast.
- qwerty456127 5y agoI'm afraid the only way there can be a faster file copying tool is by sacrificing contiguousness and increasing fragmentation. Isn't this the case? I want fragmentation to be as little as possible so it wouldn't be too hard to manually recover a file if I delete something accidentally or damage the file system.
- virtuallynathan 5y agoI just tried this going between my 2 servers with NFS, as compared to cp, rsync, and rclone. This tool wins by >2x. Great work! ~1.5Gbps vs ~3-4Gbps.