9 ms·
4x faster network file sync with rclone (vs rsync) (2025)
- baal80spam 9mo agoI'll keep saying that rclone is a fantastic and underrated piece of software.
- digiown 9mo agorclone is super cool, but unfortunately many of the providers it supports has such low ratelimits, that it's fairly difficult to use it to transfer much data at all.
- plasticsoprano 9mo agoThis has been my problem. Not necessarily that the rate limits are low, many can be gotten around by using multiple users to do the work since the limits are per user, but how rclone handles those rate limits when they hit them. The exponential back off will end up making hours and days long delays that will screw a migration.
- PunchyHamster 9mo agoI had to tweak some options because it was hitting ratelimit on B2 and the reaction of rclone to that was just... disabling deletes
- indigodaddy 9mo agoOne thing that sets rsync apart perhaps is the handling of hard links when you don't want to send both/duplicated files to the destination? Not sure if rclone can do that.
- deleted 9mo ago[deleted]
- cachius 9mo agorclone --multi-thread-streams allows transfers in parallel, like robocopy /MT You can also run multiple instances of rsync, the problem seems how to efficiently divide the set of files.
- pama 9mo agoSometimes find (with desired maxdepth) piped to gnu-parallel rsync is fine.
- SoftTalker 9mo agorobocopy! Wow, blast from the past. Used to use it all the time when I worked in a Windows shop.
- bob1029 9mo agoI am using robocopy right now on a project. The /MIR option is extremely useful for incrementally maintaining copies of large local directories.
- cachius 9mo ago> efficiently divide the set of files. It turns out, fpart does just that! Fpart is a Filesystem partitioner. It helps you sort file trees and pack them into bags (called "partitions"). It is developed in C and available under the BSD license. It comes with an rsync wrapper, fpsync. Now I'd like to see a benchmark of that vs rclone! via https://unix.stackexchange.com/q/189878/#688469 https://unix.stackexchange.com/q/189878/#688469 via https://stackoverflow.com/q/24058544/#comment93435424_25532027 https://stackoverflow.com/q/24058544/#comment93435424_255320... https://www.fpart.org/ https://www.fpart.org/
- adolph 9mo agoMy go-to for fast and easy parallelization is xargs -P. find a-bunch-of-files | xargs -P 10 do-something-with-a-file -P max-procs --max-procs=max-procs Run up to max-procs processes at a time; the default is 1. If max-procs is 0, xargs will run as many processes as possible at a time.
- ericpauley 9mo agoRclone is a fantastic tool, but my favorite part of it is actually the underyling FS library. I've started baking Rclone FS into internal Go tooling and now everything transparently supports reading/writing to either local or remote storage. Really great for being able to test data analysis code locally and then running as batch jobs elsewhere.
- absoflutely 9mo agoWhat kind of data analysis do you run with Go and do you use an open source library for it? My experience with stats libraries in Go has been lukewarm so far.
- rsync 9mo ago"Rclone is a fantastic tool, but my favorite part of it is actually the underyling FS library." Related to this is the very useful: rclone serve restic ... .. workflow that allows you to create append-only (immutable) backups. This howto is not rsync.net-specific - you can follow this recipe at any standard SSH endpoint: https://www.rsync.net/resources/notes/2025-q4-rsync.net_technotes.html https://www.rsync.net/resources/notes/2025-q4-rsync.net_tech...
- digiown 9mo agoNote there is no intrinsic reason running multiple streams should be faster than one [EDIT: "at this scale"]. It almost always indicates some bottleneck in the application or TCP tuning. (Though, very fast links can overwhelm slow hardware, and ISPs might do some traffic shaping too, but this doesn't apply to local links). SSH was never really meant to be a high performance data transfer tool, and it shows. For example, it has a hardcoded maximum receive buffer of 2MiB (separate from the TCP one), which drastically limits transfer speed over high BDP links (even a fast local link, like the 10gbps one the author has). The encryption can also be a bottleneck. hpn-ssh [1] aims to solve this issue but I'm not so sure about running an ssh fork on important systems. 1. https://github.com/rapier1/hpn-ssh https://github.com/rapier1/hpn-ssh
- mprovost 9mo agoIn general TCP just isn't great for high performance. In the film industry we used to use a commercial product Aspera (now owned by IBM) which emulated ftp or scp but used UDP with forward error correction (instead of TCP retransmission). You could configure it to use a specific amount of bandwidth and it would just push everything else off the network to achieve it.
- digiown 9mo agoThere's an open source implementation that does something similar but for a more specific use case: https://github.com/apernet/tcp-brutal https://github.com/apernet/tcp-brutal There's gotta be a less antisocial way though. I'd say using BBR and increasing the buffer sizes to 64 MiB does the trick in most cases.
- coreylane 9mo agoRClone has been so useful over the years I built a fully managed service on top of it specifically for moving data between cloud storage providers: https://dataraven.io/ https://dataraven.io/ My goal is to smooth out some of the operational rough edges I've seen companies deal with when using the tool: - Team workspaces with role-based access control - Event notifications & webhooks – Alerts on transfer failure or resource changes via Slack, Teams, Discord, etc. - Centralized log storage - Vault integrations – Connect 1Password, Doppler, or Infisical for zero-knowledge credential handling (no more plain text files with credentials) - 10 Gbps connected infrastructure (Pro tier) – High-throughput Linux systems for large transfers
- noname120 9mo agoI hope that you sponsor the rclone project given that it’s the core of your business! I couldn’t find any indication online that you do give back to the project. I hope I’m wrong.
- jfbaro 9mo agoMe too!
- coreylane 9mo agoI'm certainly planning on sponsoring the project as soon as possible, but so far I have zero paying customers, hopefully that will change soon
- znnajdla 9mo agofirst thing that popped into my mind is that your free plan is crazy generous. cut it out.
- PunchyHamster 9mo agofirst thing that popped into mine is $30/mo for running a vm with a command is something people will now just tell LLM to do
- packetlost 9mo agoI use tab-complete to navigate remote folder structures with rsync all the time, does rclone have that?
- nh2 9mo agoThis is not a feature of rsync, but of your shell. So the question "does rclone have that" doesn't make much sense, because it usually wouldn't be rclone implementing it. For example, zsh does it here for rsync, which actually invokes `ssh` itself: https://github.com/zsh-users/zsh/blob/3e72a52e27d8ce8d8be0ee65ae5f77f756c06fda/Completion/Unix/Command/_rsync https://github.com/zsh-users/zsh/blob/3e72a52e27d8ce8d8be0ee... https://github.com/zsh-users/zsh/blob/3e72a52e27d8ce8d8be0ee65ae5f77f756c06fda/Completion/Unix/Type/_remote_files#L73C15-L73C23 https://github.com/zsh-users/zsh/blob/3e72a52e27d8ce8d8be0ee... That said, some CLI tools come with tools for shells to help them implement such things. E.g. `mytool completion-helper ...` But I don't get rclone SSH completions in zsh, as it doesn't call `_remote_files` for rclone: https://github.com/zsh-users/zsh/blob/3e72a52e27d8ce8d8be0ee65ae5f77f756c06fda/Completion/Unix/Command/_rclone https://github.com/zsh-users/zsh/blob/3e72a52e27d8ce8d8be0ee...
- deleted 9mo ago[deleted]
- KolmogorovComp 9mo agoWhy are rclone/rsync never used by default for app updates? Especially games with large assets.
- rjmunro 9mo agozsync is better for that. zsync precalculates all the hashes and puts them in a file alongside the main one. The client downloads the hashes, compares them to what it has then downloads the parts it is missing. With rsync, you upload hashes of what you have, then the source has to do all the hashing work to figure out what to send you. It's slightly more efficient, but If you are supporting even 10s of downloads it's a lot of work for the source. The other option is to send just a diff, which I believe e.g. Google Chrome does. Google invented Courgette and Zucchini which partially decompile binaries then recompile them on the other end to reduce the size of diffs. These only work for exact known previous versions, though. I wonder if the ideas of Courgette and Zucchini can be incorporated into zsync's hashes so that you get the minimal diff, but the flexibility of not having a perfect previous version to work from.
- plagiarist 9mo agoDo a CRDT but for binary executables
- newsoftheday 9mo agoI prefer rsync because of its delta transfer which doesn't resend files already on the destination, saving bandwidth. This combined with rsync's ability to work over ssh lets me sync anywhere rsync runs, including the cloud. It may not be faster than rclone but it is more conserving on bandwidth.
- HPsquared 9mo agoRclone can "sync" with a range of different ways to check if the existing files are the same. If no hashes are available (e.g. WebDAV) I think you can set it to check by timestamp (with a tolerance) and size. Edit: oh I see, delta transfer only sends the changed parts of files?
- newsoftheday 9mo agoIt only sends the changed parts of files (the diffs) is my understanding which saves bandwidth.
- mnw21cam 9mo agoHowever this is only really good for slow connections. If your connection is faster than about 50MB/s then the delta calculation mechanism becomes the bottleneck. On fast connections you should use the -W option for rsync which switches the delta algorithm off.
- newsoftheday 9mo agoSaving on transmitted bandwidth costs is more important to me than speed of transmission for my use cases which I feel are likely to be common use cases.
- plagiarist 9mo agoDoes rclone not do that? I thought they were specifically naming themselves similarly because they also did that.
- xoa 9mo agoThanks for sharing, hadn't seen it but at almost the same time he made that post I too was struggling to get decent NAS<>NAS transfer speeds with rsync. I should have thought to play more with rclone! I ended up using iSCSI but that is a lot more trouble. >In fact, some compression modes would actually slow things down as my energy-efficient NAS is running on some slower Arm cores Depending on the number/type of devices in the setup and usage patterns, it can be effective sometimes to have a single more powerful router and then use it directly as a hop for security or compression (or both) to a set of lower power devices. Like, I know it's not E2EE the same way to send unencrypted data to one OPNsense router, Wireguard (or Nebula or whatever tunnel you prefer) to another over the internet, and then from there to a NAS. But if the NAS is in the same physically secure rack directly attached by hardline to the router (or via isolated switch), I don't think in practice it's significantly enough less secure at the private service level to matter. If the router is a pretty important lynchpin anyone, it can be favorable to lean more heavily on that so one can go cheaper and lower power elsewhere. Not that more efficiency, hardware acceleration etc are at all bad, and conversely sometimes might make sense to have a powerful NAS/other servers and a low power router, but there are good degrees of freedom there. Handier then ever in the current crazy times where sometimes hardware that was formerly easily and cheaply available is now a king's ransom or gone and one has to improvise.
- aidenn0 9mo agorclone is not as good as rsync for doing ad-hoc transfers; for anything not using the filesystem, you need to set up a configuration, which adds friction. It realy is purpose built for recurring transfers rather than "I need to move X to Y just once"
- ruuda 9mo agoWe wrote https://github.com/chorusone/fastsync https://github.com/chorusone/fastsync for fast ad-hoc transfers over multiple TCP streams.
- Dunedan 9mo agoI wonder if the at least partially the reason for the speed up isn't the multi-threading, but instead that rclone maybe doesn't compress transferred data by default. That's what rsync does when using SSH, so for already compressed data (like videos for example) disabling SSH compression when invoking rsync speeds it up significantly: rsync -e "ssh -o Compression=no" ...
- nh2 9mo agoCompression is off by default in OpenSSH, at least `man 5 ssh_config` says: > Specifies whether to use compression. The argument must be yes or no (the default). So I'm surprised you see speedups with your invocation.
- Dunedan 9mo agoGood point. Seems like I enabled it in ~/.ssh/config ages ago and did forget about it. Nonetheless, it's good to check whether it's enabled when using rsync to transfer large, already well compressed files.
- dspillett 9mo agoIIRC rsync uses your default SSH options, so turning off compression is only needed if your default config explicitly turns it on (generally or just for that host). If sending compressible content using rsync's compression instead of SSH's is more effective when updating files because even if not sending everything it can use it to form the compression dictionary window for what does get sent (though for sending whoe files, SSH's compression may be preferable as rsync is single threaded and using SSH's compression moves that chunk of work to the SSH process).
- sneak 9mo agoWhat’s sad to me is that rsync hasn’t been touched to fix these issues in what feels like decades.
- restalis 9mo agorsync does what was designed to do and the lack of scope creep is not a bad thing. There is "fpsync" - another tool on top of rsync (which was mentioned in one of the comments at article's page) that covers the parallel processing use-case: https://manpages.debian.org/bullseye/fpart/fpsync.1.en.html https://manpages.debian.org/bullseye/fpart/fpsync.1.en.html
- sneak 9mo agorsync was designed to synchronize directories of files from one directory to another, optionally on another host, and to do so efficiently. It doesn’t do that. Even before this conversation, a few weeks ago I started working on a modern replacement.
- gjvc 9mo agoMay 6, 2025 May 6, 2025 May 6, 2025 May 6, 2025 May 6, 2025 May 6, 2025 May 6, 2025
- rurban 9mo agoThanks for the lms tips in the comments. Amazing!
- kwanbix 9mo agoIt is crazy to see how difficult google makes it for anyone to download their own pictures from google photos. Rclone used to allow you to download them, but not anymore. Only the ones uploaded by Rclone are available to download. I wish someone forced all cloud providers to allow you to download your own data. And no, google takout doesn't count. It is horrible to use.
- buu709 9mo agoNot just bad to use, but doesn't fully work. I've been trying to get my photos off Google Photos to backup elsewhere, but takeout misses something like 20%-30% of them.
- kwanbix 9mo agoyes, that is what I meant. Once I tried to download 600GB of photos, and it crashed.
- MisterTea 9mo agoHow did you verify that takeout shorted you on 20-30% which is a huge number? This worries me as I've done some takeouts but never fully poked through them.
- buu709 9mo agoHonestly? It's an estimate based on spot checking a half dozen smaller albums where I could tell what was missing.
- ftchd 9mo agoRclone is such an elegant piece of software, reminds me of the time where most software worked well most of the time. There's few people that wouldn't benefit from it, either as a developer or end-user. I'm currently working on the GUI if you're interested: https://github.com/rclone-ui/rclone-ui https://github.com/rclone-ui/rclone-ui
- edvardsire 9mo agoInteresting that nobody has mentioned: Warp speed Data Transfer (WDT)[1]. From the readme: - Warp speed Data Transfer (WDT) is an embeddedable library (and command line tool) aiming to transfer data between 2 systems as fast as possible over multiple TCP paths. - Goal: Lowest possible total transfer time - to be only hardware limited (disc or network bandwidth not latency) and as efficient as possible (low CPU/memory/resources utilization) 1. https://github.com/facebook/wdt https://github.com/facebook/wdt
- tonymet 9mo agogolang concurrent IO is so accessible that even trivial IO transform scripts (e.g. compression, base64, md5sum/cksum) are very easy to multicore. You'd be astonished at how much faster even seemingly fast local IO can go when you unblock the IO
- SloopJon 9mo agoThe article links to a YouTube mini-review of USB enclosures from UGreen and Acasis, neither of which he loves.[1] I've been happy with the OWC 1M2 as a boot drive on a Mac Studio with Thunderbolt 5 ports.[2] I just noticed that there is an OWC 1M2 80G, based on USB4 v2.[3] I didn't know that was a thing, but I guess it's the USB cousin to Thunderbolt 5. [1] https://www.youtube.com/watch?v=gaV-O6NPWrI https://www.youtube.com/watch?v=gaV-O6NPWrI [2] https://eshop.macsales.com/shop/owc-express-1m2 https://eshop.macsales.com/shop/owc-express-1m2 [3] https://eshop.macsales.com/item/OWC/US4V2EXP1M2/ https://eshop.macsales.com/item/OWC/US4V2EXP1M2/
- weddingbell 9mo agoYesterday I set up Rclone, and the download speed from Google Drive was so slow that caching took a long time. After applying the method described in this article, the speed improved. This is my mount configuration. What do you think? Is there anything that might be causing issues?? rclone mount google_drive: X: ^ --vfs-cache-mode full ^ --vfs-cache-max-age 24h ^ --vfs-cache-max-size 50G ^ --vfs-read-ahead 1G ^ --cache-dir "./rclone_cache" ^ --vfs-read-chunk-size 128M ^ --vfs-read-chunk-size-limit off ^ --buffer-size 128M ^ --dir-cache-time 1000h ^ --drive-chunk-size 64M ^ --poll-interval 15s ^ --vfs-cache-poll-interval 1m ^ --multi-thread-streams 32 ^ --drive-skip-shortcuts ^ --drive-acknowledge-abuse ^ --network-mode
- xen2xen1 9mo agoDo you have your own api key? Been a while, but that's the most common problem.
- weddingbell 9mo agoI set up your API key and secret. However, I couldn't verify if it was working with the API.
- hsbauauvhabzb 9mo agoI love rclone. I use it in place of scp/sftp frequently. My biggest complaint is that it seems to require a config file, having the ability to `rclone copy root@192.168.1.1:/tmp/foo ./` would be a game changer.
- cranberryturkey 9mo agoThe parallelism advantage of rclone is real but undersold here. rsync's single-stream design made sense when networks were the bottleneck. Now with high-bandwidth links (especially to cloud storage), the bottleneck is often the round-trip latency of per-file metadata operations. rclone's multi-threaded transfers effectively pipeline those operations. It's the same principle as why HTTP/2 multiplexing was such a win — you stop paying the latency tax sequentially. One thing I'd add: for local-to-local or LAN sync, rsync still often wins because the overhead of rclone's abstraction layer isn't worth it when latency is already sub-millisecond. The 4x speedup is really a story about high-latency, high-bandwidth paths where parallelism dominates.
- sigmonsays 9mo agofor fun try this 1. tar + nc 2. rsync 3. scp #1 is the fastest way to send because it keeps the buffers full in a consistent stream which allows tcp windows to grow as large as possible. 2 and 3 ( scp and rsync ) do round trip acks to the remote side which drastically slows things down, even if done in parallel.
- ndsipa_pomu 9mo agoNote that tar and rsync can handle sparse files, but scp doesn't. Also, rsync will likely be faster if some files are already in the destination as it handles that intelligently as well as only transferring changed blocks from files.
- apt-apt-apt-apt 9mo agoI have a few qualms about rclone: You can already perform parallel transfers quite trivially by typing "find .. | xargs -P$(nproc) -n 1 rsync ..", carefully managing sync folder sets and just being a Unix god in general like me. It doesn't seem reasonable to expect people to use and install it, when we can just use rsync.
- creatonez 9mo ago> You can already perform parallel transfers quite trivially by typing "find .. | xargs -P$(nproc) -n 1 rsync ..", carefully managing sync folder sets and just being a Unix god in general. This solution sucks in a number of different ways, and is downright incorrect. I'm baffled by the frequency at which these hacky parallel commands are recommended when rclone exists, works perfectly fine, and has literally no runtime dependencies.
- Maakuth 9mo agoIt's a reference to a classic comment: https://news.ycombinator.com/item?id=9224 https://news.ycombinator.com/item?id=9224
- 31337Logic 9mo agoHe was being sarcastic.
- throwawaymaroon 9mo ago[dead]