5 ms·

Inspired by many of the amazing new-wave Unix tools that have come about over the past several years (fzf, ripgrep, fd, lsd, etc.), I set out to create a more m

Inspired by many of the amazing new-wave Unix tools that have come about over the past several years (fzf, ripgrep, fd, lsd, etc.), I set out to create a more modern and performant version of the classic Unix cp command. Written in Rust due to the excellent ergonomics and performance characteristics of the language, fcp copies large files and directories in a fraction of the time that cp does. Feel free to ask me any questions about the project!

jepler 5y ago
_Why_ is it faster? (particularly on Linux, I see you're using syscalls on mac that mac's cp doesn't, since apple doesn't really maintain their unix userspace)
- Svetlitski 5y ago
  The primary reason is that it uses all of your computer's available cores to walk through directories and copy files, effectively speeding things up by a factor directly proportional to how many cores you have. On macOS, this is augmented by regular files being copied using the OS's fclonefileat and fcopyfile mechanisms under the hood, which allows even very large files to be copied near-instantly (when compared to copying them block-by-block as classic cp does).
  - CalChris 5y ago
    Is it available on homebrew?
    - Svetlitski 5y ago
      No, it is not. I would've submitted a Homebrew formula for it, but that would go directly against Homebrew's guidelines, which frown upon authors submitting formulas for their own tools [1]. [1] https://docs.brew.sh/Acceptable-Formulae#niche-or-self-submitted-stuff https://docs.brew.sh/Acceptable-Formulae#niche-or-self-submi...
  - fulafel 5y ago
    This works out assuming that file copying is CPU bound. Traditionally this hasn't been the case but maybe things have changed with widespread fast SSDs and usage of disk crypto. Traditionally the reason to use threads in small-requests i/o is to get multiple outstanding requests in flight in the hardware io stack which some hardware can use to get you more IOPS throughput. (They might end up hitting different disks in RAID, or taking advantage SSD internal parallelism, or whatever tricks spinning disks use to get more throughput with TCQ) (Re your MacOS remark, Linux has support for copy-on-write file copies as well, see reflink and FICLONE / copy_file_range)
    - yokaze 5y ago
      > Traditionally the reason to use threads in small-requests i/o is to get multiple outstanding requests in flight in the hardware io stack which some hardware can use to get you more IOPS throughput I would assume that is the case here too. The threads are likely hiding the latency of the IOPS. I can't see any CPU load statistics, so it is hard to say, if it is indeed CPU bound or not. A quick run with time shows user+sys being only slightly higher than real, so with 6 threads for my 6 cpus, it uses a single cpu. (and practically no time in user-space)
      - fulafel 5y ago
        Ok, so the core count is not the right level of parallelism (except by accident).
  - mnw21cam 5y ago
    Does this software gain any speed by sacrificing any safety? As in, does it switch off fsync in any way? Are the files guaranteed durable by the time the software terminates?
  - jjav 5y ago
    > The primary reason is that it uses all of your computer's available cores to walk through directories and copy files, effectively speeding things up by a factor directly proportional to how many cores you have. CPU cores can't read/write files, those are IO operations by the storage device. The CPU could have 64 cores but on a system with a single hard disk that can do just one concurrent operation.
- Siira 5y ago
  Does GNU cp (installed from brew) use those syscalls?
nextaccountic 5y ago
My only gripe with those new unix tools is that many of them don't come with a man page, even though they have essentially the same information elsewhere So, perhaps, take a look at https://github.com/rust-cli/man https://github.com/rust-cli/man or something
- drran 5y ago
  IMHO, just convert README.md into the man page using a md2man converter, and be done with it.
  - attractivechaos 5y ago
    This assumes README.md lists all the command-line options like in a manpage, which is not often the case. Also, manpages follow a conventional structure. You can expect the content in a manpage. Most READMEs are free-style. You don't know what is there. md2man doesn't solve the problem.
    - drran 5y ago
      Ok. Then just fire browser, then google project page, and then read free-style README.md at it original location. It's just a dozen clicks away.
      - selfhoster11 5y ago
        You assume that I have a browser that starts up quickly, or that it's not already loaded with a hundred other tabs on a good day.
- Y_Y 5y ago
  Well tldr (or tealdeer) or whatever is a nice example of another modern (semi-)alternative to man. tldr usually just gives a handful of explained example commands rather than an actual manual, but it's often enough information to solve the problem. I bet these some of these tools have pages on that at least. (I agree that a real manual is necessary too. Maybe you could convert one from the online docs?)
nicoburns 5y ago
What's the performance like with large numbers of small files? I find that's typically much slower than copying a few big files.
robthebrew 5y ago
Thanks for this. I'll be testing it on Apple Silicon. Here's the build: https://workupload.com/file/kvwMb2DGSrS https://workupload.com/file/kvwMb2DGSrS
wheybags 5y ago
Not a question, but I made a similar tool in c++[1], for Linux only using io_uring, and a blog post explaining its internals [2]. I'll definitely have a look some time soon, I'd be interested to see how performance compares (I gathered from some other comments here that you're using blocking io in threads?) 1: https://github.com/wheybags/wcp https://github.com/wheybags/wcp 2: https://wheybags.com/blog/wcp.html https://wheybags.com/blog/wcp.html
- Svetlitski 5y ago
  Sounds super interesting, I'll definitely check out the blog post. Yes, fcp is indeed using blocking io in threads. In an earlier prototype I was using async (i.e. M on N threading), but was running into issues with exhausting file descriptors. Because of this (and because writing async Rust is quite difficult) I settled on using blocking io in threads. Perhaps in a future version I'll try to implement the io_uring approach for Linux.
  - wheybags 5y ago
    Ah, I also tried unlocking the file descriptor 1024 limit, which worked fine most of the time, but didn't actually give me a perf boost in any of my tests. I got rid of it in the end because it caused problems when copying from an smb mount. iirc I exhausted the file descriptor limit on the samba server.

jepler 5y ago

_Why_ is it faster? (particularly on Linux, I see you're using syscalls on mac that mac's cp doesn't, since apple doesn't really maintain their unix userspace)

Svetlitski 5y ago
The primary reason is that it uses all of your computer's available cores to walk through directories and copy files, effectively speeding things up by a factor directly proportional to how many cores you have. On macOS, this is augmented by regular files being copied using the OS's fclonefileat and fcopyfile mechanisms under the hood, which allows even very large files to be copied near-instantly (when compared to copying them block-by-block as classic cp does).
- CalChris 5y ago
  Is it available on homebrew?
  - Svetlitski 5y ago
    No, it is not. I would've submitted a Homebrew formula for it, but that would go directly against Homebrew's guidelines, which frown upon authors submitting formulas for their own tools [1]. [1] https://docs.brew.sh/Acceptable-Formulae#niche-or-self-submitted-stuff https://docs.brew.sh/Acceptable-Formulae#niche-or-self-submi...
- fulafel 5y ago
  This works out assuming that file copying is CPU bound. Traditionally this hasn't been the case but maybe things have changed with widespread fast SSDs and usage of disk crypto. Traditionally the reason to use threads in small-requests i/o is to get multiple outstanding requests in flight in the hardware io stack which some hardware can use to get you more IOPS throughput. (They might end up hitting different disks in RAID, or taking advantage SSD internal parallelism, or whatever tricks spinning disks use to get more throughput with TCQ) (Re your MacOS remark, Linux has support for copy-on-write file copies as well, see reflink and FICLONE / copy_file_range)
  - yokaze 5y ago
    > Traditionally the reason to use threads in small-requests i/o is to get multiple outstanding requests in flight in the hardware io stack which some hardware can use to get you more IOPS throughput I would assume that is the case here too. The threads are likely hiding the latency of the IOPS. I can't see any CPU load statistics, so it is hard to say, if it is indeed CPU bound or not. A quick run with time shows user+sys being only slightly higher than real, so with 6 threads for my 6 cpus, it uses a single cpu. (and practically no time in user-space)
    - fulafel 5y ago
      Ok, so the core count is not the right level of parallelism (except by accident).
- mnw21cam 5y ago
  Does this software gain any speed by sacrificing any safety? As in, does it switch off fsync in any way? Are the files guaranteed durable by the time the software terminates?
- jjav 5y ago
  > The primary reason is that it uses all of your computer's available cores to walk through directories and copy files, effectively speeding things up by a factor directly proportional to how many cores you have. CPU cores can't read/write files, those are IO operations by the storage device. The CPU could have 64 cores but on a system with a single hard disk that can do just one concurrent operation.
Siira 5y ago
Does GNU cp (installed from brew) use those syscalls?

nextaccountic 5y ago

My only gripe with those new unix tools is that many of them don't come with a man page, even though they have essentially the same information elsewhere So, perhaps, take a look at https://github.com/rust-cli/man https://github.com/rust-cli/man or something

drran 5y ago
IMHO, just convert README.md into the man page using a md2man converter, and be done with it.
- attractivechaos 5y ago
  This assumes README.md lists all the command-line options like in a manpage, which is not often the case. Also, manpages follow a conventional structure. You can expect the content in a manpage. Most READMEs are free-style. You don't know what is there. md2man doesn't solve the problem.
  - drran 5y ago
    Ok. Then just fire browser, then google project page, and then read free-style README.md at it original location. It's just a dozen clicks away.
    - selfhoster11 5y ago
      You assume that I have a browser that starts up quickly, or that it's not already loaded with a hundred other tabs on a good day.
Y_Y 5y ago
Well tldr (or tealdeer) or whatever is a nice example of another modern (semi-)alternative to man. tldr usually just gives a handful of explained example commands rather than an actual manual, but it's often enough information to solve the problem. I bet these some of these tools have pages on that at least. (I agree that a real manual is necessary too. Maybe you could convert one from the online docs?)

nicoburns 5y ago

What's the performance like with large numbers of small files? I find that's typically much slower than copying a few big files.

robthebrew 5y ago

Thanks for this. I'll be testing it on Apple Silicon. Here's the build: https://workupload.com/file/kvwMb2DGSrS https://workupload.com/file/kvwMb2DGSrS

wheybags 5y ago

Not a question, but I made a similar tool in c++[1], for Linux only using io_uring, and a blog post explaining its internals [2]. I'll definitely have a look some time soon, I'd be interested to see how performance compares (I gathered from some other comments here that you're using blocking io in threads?) 1: https://github.com/wheybags/wcp https://github.com/wheybags/wcp 2: https://wheybags.com/blog/wcp.html https://wheybags.com/blog/wcp.html

Svetlitski 5y ago
Sounds super interesting, I'll definitely check out the blog post. Yes, fcp is indeed using blocking io in threads. In an earlier prototype I was using async (i.e. M on N threading), but was running into issues with exhausting file descriptors. Because of this (and because writing async Rust is quite difficult) I settled on using blocking io in threads. Perhaps in a future version I'll try to implement the io_uring approach for Linux.
- wheybags 5y ago
  Ah, I also tried unlocking the file descriptor 1024 limit, which worked fine most of the time, but didn't actually give me a perf boost in any of my tests. I got rid of it in the end because it caused problems when copying from an smb mount. iirc I exhausted the file descriptor limit on the samba server.