3 ms·
It looks like this is using multiple threads to copy files. That's fine for NVMe which tends to achieve more throughput with higher queue depths. But it will de
by the8472 5y ago
It looks like this is using multiple threads to copy files. That's fine for NVMe which tends to achieve more throughput with higher queue depths. But it will degrade performance when copying large files on spinning rust.
- idoubtit 5y agoMy experiments confirms your analyze. On a SATA hard drive using ext4, I copied a directory with a large hierarchy of file and 216MB overall. After 2 warm-ups, I launch this simple benchmark. time cp -r moodle X 13.76s s (0.06s u + 2.44s k) 18% CPU time fcp-0.1.0-x86_64-unknown-linux-gnu moodle X 46.12s s (0.15s u + 2.47s k) 3% CPU I emptied linux cache before each run with `echo 1 > /proc/sys/vm/drop_caches`, but this does not remove the HDD buffering. I suspect that fcp would be worse in real life conditions (no warm-up). This does not mean the fcp is bad or not performant. But it some cases, it can be 3x slower.
- kwdc 5y agoSince you've already started, can you do a benchmark of a pair of copies started at same time? One copy operation is the little files and deep hierarchy. The other copy operation is just a big file. Both operations start at same time and use the same tool. I'm curious what cp and fcp do to the cache as they operate.
- hpb42 5y agoWhat do you mean by "After 2 warm-ups"? I've seen this expression in the context of benchmarks before and it puzzles me. Does it mean you are doing this operation enough times to trick the kernel to cache everything in RAM? Or you run it 3 times and chose the fastest timings?
- snet0 5y agoI'm not sure about the specific context here, but benchmark warm-ups are often used to make sure that the program's run time is nothing but that, and doesn't include things like setting up an environment or compiling to a cache. I think there are sometimes arguments to be made that, for the user, that is run time. When using benchmarks to compare programs like this, though, I think isolating the "actual" (i.e. useful) program execution is productive.
- juancampa 5y agoFirst run: fcp binary has to be read from disk, so it's slow(ish) to start. Second run: fcp binary is cached by the OS in RAM so it starts up faster. Third run: just to make sure :) This is just one of the things of a type of cache that can be warmed up.
- RicoElectrico 5y agoThe irony is that spinning rust is everything but ;) Most platters are made of glass, or maybe aluminum. I learned this the hard way when dismantling and abusing a dead HDD. Had to vacuum all the tiny shards from the floor.
- AndrewStephens 5y agoThe rust in this case is the iron oxide coating on the inert platters.
- unknown_error 5y agoSoooo all my data is stored on magnetized rust?
- AndrewStephens 5y agoYep. See [0] for a demonstration of do-it-yourself audio tape but the same applies for hard drives. https://youtu.be/g1JlUcFKm5o?t=537 https://youtu.be/g1JlUcFKm5o?t=537
- unknown_error 5y agoHah, this is amazing! I wish they taught that in schools. I don't know who they are but they're like a mix between Bill Nye, Carl Sagan, and the Mythbusters. Cool duo. Great old video. Thanks for sharing. Terrifying goose though.
- Svetlitski 5y agoThis was definitely an oversight on my part; the benchmarks I ran were all conducted on systems with SSDs (as noted in the README). Especially considering the testing done in some of the comments below this one, I'll update the README to mention that fcp is geared towards SSDs. As a developer tool though, I still think this is the right choice, as at least anecdotally the vast majority of developers I know have an SSD in their primary machine.