11 ms·

Rclone: Rsync for cloud storage – CLI to sync between cloud storage providers

246 points by gilad 5y ago

petre 5y ago
I wonder if I could use it with a rsync endpoint? It's not listed in the backends.
zuj 5y ago
I have used Rclone for both personal file backup and production data. It is dead simple and amazingly robust.
aborsy 5y ago
Rclone is popular for good reasons. I have even replaced Dropbox with rclone nowadays. Why replacing Dropbox sync, one may ask? Because: * Rclone offers client side encryption (with a good algorithm and from a good library). Cloud providers usually offer server side encryption, but don’t want to offer client side encryption. * You can mount the remote. Files don’t have to exist on client’s side to occupy space. This tool mounts anything! * You learn one tool and use it everywhere * It’s is under user’s control. You run it with a schedule that you want. I don’t have to run a closed source app continuously scanning my computer * Robust, feature full, open source (especially for verification of encryption), large community for getting help, and under heavy development Rclone combined with Restic is how I manage my data in the cloud.
- CodeGlitch 5y ago
  I use it for OneDrive on Linux. It's not as fast as Dropbox (which I used previously to OneDrive) but works great for backups.
- hank_z 5y ago
  > Rclone offers client side encryption Can you view files in Dropbox website if they are encrypted locally by Rclone? > You can mount the remote. Do you have to mount the remote every time after rebooting your machine? If I use Rclone to sync all dropbox files to my local machine, are they still accessible if my machine is offline?
- mcny 5y ago
  » * It’s is under user’s control. You run it with a schedule that you want. I don’t have to run a closed source app continuously scanning my computer A few years back, around 2016 I wanted to see how One drive syncing worked. So I set my IRC client (hexchat) to save logs on my OneDrive folder. If I remember correctly, I had hundreds of gigabytes uploaded to onedrive under a month. I think I had the default 15 GB or 25 GB storage iirc. My guess is every time someone said something in a chat, the file changed and one drive thought it was prudent to sync it with its servers immediately? When a file changes, should we upload the entire file? Is it possible to somehow only transmit the diff and only send what has changed? Would it even make sense as most "normal" users are probably on mobile and diff has a cost in CPU and therefore battery, right? Or is CPU cheaper than WiFi for battery? One of the features I like about Amazon.com Photos on Android is I can say only upload photos when charging.
  - tinus_hn 5y ago
    > Is it possible to somehow only transmit the diff and only send what has changed? This is what rsync does. edit this refers to rsync the software, not rsync the company that provides storage.
    - karatinversion 5y ago
      Really? How does it know what the diff is when it only has the new version of the file?
      - ZeroGravitas 5y ago
        https://en.m.wikipedia.org/wiki/Rsync#Algorithm https://en.m.wikipedia.org/wiki/Rsync#Algorithm
- rishabhd 5y ago
  Also popular for some unfortunate reasons as well, it is becoming the exfil and sync tool of choice for ransomware operators. context : first hand experience in IR
  - InGoodFaith 5y ago
    Slight tangent: I always found it to be an added plus/bonus to a tool if it can survive being scrutinized by well-formed and long-lasting groups. What would be my preferred VPN/Mail/Payment/Storage/Encryption tools? The one that has been scrutinized by TLA to catch these groups. On a long enough time scale, it can expose if they truly are "no logs", "fully encrypted", "privacy-conscious" or if it's just marketing buzzwords when they actually face a warrant.
tttsxhub 5y ago
Does it support S3 Glacier?
- wscott 5y ago
  Yes, you would have to been pretty obscure to not be supported. https://rclone.org/s3/ https://rclone.org/s3/
stavros 5y ago
Rclone is excellent, I use it to deploy my static site to Neocities and it's worked excellently for ages.
ehPReth 5y ago
Love this software, use it for everything from mounting cloud storage to syncing backups
glitchcrab 5y ago
Nick Craig-Wood (ncw) was the CEO of the last company I worked for, and I consider it a privilege to have worked for him. He's one of the smartest people I've ever met, and an incredibly nice person too. I have many fond memories of the annual staff barbecue in his garden.
- ivvv 5y ago
  I attended his talk on deadlocks in Go at Gophercon UK this year. It was great, he showed some real life examples from rclone. Here is a link if anyone is interested - https://www.youtube.com/watch?v=9j0oQkqzhAE https://www.youtube.com/watch?v=9j0oQkqzhAE
- meirelles 5y ago
  He reviewed my first Go PR a few years back when I posted something new for rclone. He's super nice, indeed. Kudos Nick!
  - rajishx 5y ago
    I worked with nick on cloud storage systems, he is one of the nicest guy ever
barelyusable 5y ago
Love this tool. Along with Golang's Excelize library I use it to build a bunch of tools
bckygldstn 5y ago
Rclone is fantastic. It's much easier to use that most of the cloud storage APIs themselves. Plus rclone makes it easy to migrate between providers. I just moved a backup process from Gcloud to B2 now my side business is growing up: you can start with whatever storage is free/convenient then switch to something more mature later with minimal effort.
- flatiron 5y ago
  I run a ~75TB plex server with 99% on Google drive with 0 issues using rclone mount on a haswell i3. Can get close to 20 folks on gigabit before it blinks.
  - ValentineC 5y ago
    Are you using the VFS cache liberally? I'm quite stingy with my settings, but get downloadQuotaExceeded errors from time to time.
    - flatiron 5y ago
      No rclone cache. 2 week local storage cache with unionfs
  - deeblering4 5y ago
    Thats an impressive setup. Is that across multiple g drives?
    - fatboy93 5y ago
      Probably a single user business that's has unlimited storage. I think the plan is for $12?
      - jffry 5y ago
        The only Google Workspace plan that offers unlimited is the contact-sales-for-pricing "Enterprise" tier. The $12/user/month plan only includes 2TB/user: https://workspace.google.com/pricing.html https://workspace.google.com/pricing.html
guilhas 5y ago
Is this a good multi threaded local/ssh alternative to rsync?
- throwaway999901 5y ago
  It is. Another good alternative is lftp.
techisnotmagic 5y ago
I'm not sure why this bothers me so much but it does not "sync between cloud storage providers". I don't really know how this misunderstanding is so pervasive, even among the HN crowd, but no, no amount of magical code can make it possible to directly migrate between S3 and Google Drive. It's going to download, and upload to sync. This is even reflected in the project's description line, which is more accurate than the current HN headline: Rclone ("rsync for cloud storage") is a command-line program to sync files and directories to and from different cloud storage providers. That having been said, it's of course invaluable software. Mounting 10TB of Google Drive into a Raspberry Pi running Plex is still something that feels like it should be harder to accomplish than it is.
- trog 5y ago
  This was exactly the comment I was looking for. I've had to move a lot of data to GDrive from Dropbox before and have been keeping an eye out for tools to help with this. Using Google Colab is the best way I've found so far - mount your GDrive and you can wget to it directly from Dropbox (or anything else).
- BHSPitMonkey 5y ago
  Does something the word "sync" require that no intermediary hardware must exist between the endpoints? And does this definition disqualify the use of devices like, say, routers and switches?
- trevyn 5y ago
  >no amount of magical code can make it possible to directly migrate between S3 and Google Drive writes magical code that fires up an unmetered-bandwidth billed-hourly VPS for a few hours (Scaleway is nice)
divbzero 5y ago
I like rclone for the wide array of storage options [1] plus a special crypt remote [2] that provides client-side encryption for any of the other storage options. There is also a browser GUI you can spin up locally if that floats your boat [3]. [1]: https://rclone.org/overview/ https://rclone.org/overview/ [2]: https://rclone.org/crypt/ https://rclone.org/crypt/ [3]: https://rclone.org/gui/ https://rclone.org/gui/
folli 5y ago
Since we're on the topic: Which cloud provider do you recommend for backup of personal files?
- lelandfe 5y ago
  I like Backblaze because it works well on my Mac and I don't have to think about it. That's about all I want!
- tinus_hn 5y ago
  Whatever provider you choose, make sure you encrypt your files locally to avoid the scenario where the provider blocks you because the AI decided one of your files is a virus, a copyright violation or matches some image hash. There is no need for your backup provider to be able to look into your files so there’s only downsides to allowing them to.
- prirun 5y ago
  I've been using Backblaze B2 since it launched, and HashBackup (I'm the author) was one of the first B2 integrations. What I like most about B2 is the transparent, no gimmick pricing. No monthly minimum, no minimum file size, no paying for storage for 90 days after you delete a file. And they give generous free allowances: 10GB/mo (forever, with no credit card required), 1GB of free egress per day, and up to 5000 free transactions per day. I backup the HashBackup dev server to B2 every night, around 80GB with mostly VM images, and the total backup size while keeping the last 30 days + 12 monthly backups is around 50 GB and costs 25 cents/month. They bill me every few months, I guess when enough charges accumulate. Here are the backup stats for the HashBackup dev server: [jim@bs ~]$ hb stats -c /hbbackup HashBackup #2527 Copyright 2009-2021 HashBackup, LLC Backup directory: /hbbackup 3,617 backups 334 TB file bytes checked since initial backup 80 TB file bytes saved since initial backup 910 total backup hours 79 GB average file bytes checked per backup in last 5 backups 23 GB average file bytes saved per backup in last 5 backups 29.78% average changed data percentage per backup in last 5 backups 13m 43s average backup time for last 5 backups 637 archives 55 GB archive space 84.11% archive space utilization 46 GB 810:1 industry standard dedup ratio 9.5 MB average archive space per backup for last 5 backups 2465:1 reduction ratio of backed up files for last 5 backups 234 MB dedup table current size 9,442,920 dedup table entries 48% dedup table utilization at current size 724,059 files 671,701 paths 10,550,610 blocks 127,965,672 block references 117,415,062 deduped blocks 12.1:1 block dedup ratio 5.2 KB average stored block size 616 GB backup space saved by dedup 49,648 average variable-block length
rsync 5y ago
Here is a keystroke-by-keystroke HOWTO that we wrote last year: https://rsync.net/resources/howto/rclone.html https://rsync.net/resources/howto/rclone.html and although the example is S3 <-> rsync.net, it is not necessarily rsync.net-specific. You could use this HOWTO/workflow for any two targets.
- jffry 5y ago
  Thanks for this write-up, it's what finally got me over the activation energy barrier to setting up backups of a small home-automation setup I run. The rclone docs are very extensive but having a short, simple example was the kick I needed to get past the myriad options.
- earthscienceman 5y ago
  Do you contribute code to rclone?
- TAForObvReasons 5y ago
  FYI You can choose the type of storage by using the short name instead of the index 4 / Amazon S3 Compliant Storage Provider (AWS, Alibaba, Ceph, Digital Ocean, Dreamhost, IBM COS, Minio, etc) \ "s3" "4" is fragile because future changes could change that offset (it happened at least once with Google Drive), but "s3" is fixed.
  - rsync 5y ago
    Thanks! I will pass that along and see that it is "fixed" ...
gary_0 5y ago
Past discussions: https://news.ycombinator.com/item?id=12398303 https://news.ycombinator.com/item?id=12398303 (5 years ago) https://news.ycombinator.com/item?id=22791036 https://news.ycombinator.com/item?id=22791036 (2 years ago)
xanaxagoras 5y ago
Rclone is great! All of my duplicati backups go to S3, then in the run-script-after script rclone copies them from S3 to B2.
dang 5y ago
Past related threads: Rclone syncs your files to cloud storage - https://news.ycombinator.com/item?id=29213024 https://news.ycombinator.com/item?id=29213024 - Nov 2021 (8 comments) Rclone – Sync files and directories to many cloud storage providers - https://news.ycombinator.com/item?id=22791036 https://news.ycombinator.com/item?id=22791036 - April 2020 (100 comments) Rclone – Rsync for Cloud Storage - https://news.ycombinator.com/item?id=21906779 https://news.ycombinator.com/item?id=21906779 - Dec 2019 (3 comments) Rclone: rsync for cloud storage - https://news.ycombinator.com/item?id=12398303 https://news.ycombinator.com/item?id=12398303 - Aug 2016 (129 comments) Rclone: rsync for cloud storage - https://news.ycombinator.com/item?id=12049507 https://news.ycombinator.com/item?id=12049507 - July 2016 (1 comment) Rclone – rsync for cloud storage - https://news.ycombinator.com/item?id=12045729 https://news.ycombinator.com/item?id=12045729 - July 2016 (3 comments) Rclone – rsync for cloud storage - https://news.ycombinator.com/item?id=11404178 https://news.ycombinator.com/item?id=11404178 - April 2016 (1 comment)
modeless 5y ago
I once made a software distribution system for software with thousands of files in a huge directory tree using rclone. I'd upload it all to S3 and then clients could run an update script that used rclone to sync just the changed files. No server needed. A better solution probably exists but I couldn't find any other system at the time that could work directly from S3 or any other dumb file host and still selectively sync only changed files.
- lazypenguin 5y ago
  I recently noticed in the rclone source tree[0] that they are working on making “librclone”. I would love it if they exposed the rclone internals as a library as the only other solutions (e.g. bita, zsync, librsync) still require a fair bit of tedium to use as a generic application updater. [0] https://github.com/rclone/rclone/tree/master/librclone https://github.com/rclone/rclone/tree/master/librclone
joshenders 5y ago
“Rclone” and not “Csync”? Missed opportunity! ;)
sbt567 5y ago
I'm using rclone beta right now for its bisync feature. This was the last missing piece that I badly needed to synchronize my notes. And I'm very happy that it's finally implemented. Kudos to all rclone contributors!
- quchen 5y ago
  That sounds cool! Is there some preview documentation available somewhere?

246 points by gilad 5y ago

petre 5y ago

I wonder if I could use it with a rsync endpoint? It's not listed in the backends.

zuj 5y ago

I have used Rclone for both personal file backup and production data. It is dead simple and amazingly robust.

aborsy 5y ago

Rclone is popular for good reasons. I have even replaced Dropbox with rclone nowadays. Why replacing Dropbox sync, one may ask? Because: * Rclone offers client side encryption (with a good algorithm and from a good library). Cloud providers usually offer server side encryption, but don’t want to offer client side encryption. * You can mount the remote. Files don’t have to exist on client’s side to occupy space. This tool mounts anything! * You learn one tool and use it everywhere * It’s is under user’s control. You run it with a schedule that you want. I don’t have to run a closed source app continuously scanning my computer * Robust, feature full, open source (especially for verification of encryption), large community for getting help, and under heavy development Rclone combined with Restic is how I manage my data in the cloud.

CodeGlitch 5y ago
I use it for OneDrive on Linux. It's not as fast as Dropbox (which I used previously to OneDrive) but works great for backups.
hank_z 5y ago
> Rclone offers client side encryption Can you view files in Dropbox website if they are encrypted locally by Rclone? > You can mount the remote. Do you have to mount the remote every time after rebooting your machine? If I use Rclone to sync all dropbox files to my local machine, are they still accessible if my machine is offline?
mcny 5y ago
» * It’s is under user’s control. You run it with a schedule that you want. I don’t have to run a closed source app continuously scanning my computer A few years back, around 2016 I wanted to see how One drive syncing worked. So I set my IRC client (hexchat) to save logs on my OneDrive folder. If I remember correctly, I had hundreds of gigabytes uploaded to onedrive under a month. I think I had the default 15 GB or 25 GB storage iirc. My guess is every time someone said something in a chat, the file changed and one drive thought it was prudent to sync it with its servers immediately? When a file changes, should we upload the entire file? Is it possible to somehow only transmit the diff and only send what has changed? Would it even make sense as most "normal" users are probably on mobile and diff has a cost in CPU and therefore battery, right? Or is CPU cheaper than WiFi for battery? One of the features I like about Amazon.com Photos on Android is I can say only upload photos when charging.
- tinus_hn 5y ago
  > Is it possible to somehow only transmit the diff and only send what has changed? This is what rsync does. edit this refers to rsync the software, not rsync the company that provides storage.
  - karatinversion 5y ago
    Really? How does it know what the diff is when it only has the new version of the file?
    - ZeroGravitas 5y ago
      https://en.m.wikipedia.org/wiki/Rsync#Algorithm https://en.m.wikipedia.org/wiki/Rsync#Algorithm
rishabhd 5y ago
Also popular for some unfortunate reasons as well, it is becoming the exfil and sync tool of choice for ransomware operators. context : first hand experience in IR
- InGoodFaith 5y ago
  Slight tangent: I always found it to be an added plus/bonus to a tool if it can survive being scrutinized by well-formed and long-lasting groups. What would be my preferred VPN/Mail/Payment/Storage/Encryption tools? The one that has been scrutinized by TLA to catch these groups. On a long enough time scale, it can expose if they truly are "no logs", "fully encrypted", "privacy-conscious" or if it's just marketing buzzwords when they actually face a warrant.

tttsxhub 5y ago

Does it support S3 Glacier?

wscott 5y ago
Yes, you would have to been pretty obscure to not be supported. https://rclone.org/s3/ https://rclone.org/s3/

stavros 5y ago

Rclone is excellent, I use it to deploy my static site to Neocities and it's worked excellently for ages.

ehPReth 5y ago

Love this software, use it for everything from mounting cloud storage to syncing backups

glitchcrab 5y ago

Nick Craig-Wood (ncw) was the CEO of the last company I worked for, and I consider it a privilege to have worked for him. He's one of the smartest people I've ever met, and an incredibly nice person too. I have many fond memories of the annual staff barbecue in his garden.

ivvv 5y ago
I attended his talk on deadlocks in Go at Gophercon UK this year. It was great, he showed some real life examples from rclone. Here is a link if anyone is interested - https://www.youtube.com/watch?v=9j0oQkqzhAE https://www.youtube.com/watch?v=9j0oQkqzhAE
meirelles 5y ago
He reviewed my first Go PR a few years back when I posted something new for rclone. He's super nice, indeed. Kudos Nick!
- rajishx 5y ago
  I worked with nick on cloud storage systems, he is one of the nicest guy ever

barelyusable 5y ago

Love this tool. Along with Golang's Excelize library I use it to build a bunch of tools

bckygldstn 5y ago

Rclone is fantastic. It's much easier to use that most of the cloud storage APIs themselves. Plus rclone makes it easy to migrate between providers. I just moved a backup process from Gcloud to B2 now my side business is growing up: you can start with whatever storage is free/convenient then switch to something more mature later with minimal effort.

flatiron 5y ago
I run a ~75TB plex server with 99% on Google drive with 0 issues using rclone mount on a haswell i3. Can get close to 20 folks on gigabit before it blinks.
- ValentineC 5y ago
  Are you using the VFS cache liberally? I'm quite stingy with my settings, but get downloadQuotaExceeded errors from time to time.
  - flatiron 5y ago
    No rclone cache. 2 week local storage cache with unionfs
- deeblering4 5y ago
  Thats an impressive setup. Is that across multiple g drives?
  - fatboy93 5y ago
    Probably a single user business that's has unlimited storage. I think the plan is for $12?
    - jffry 5y ago
      The only Google Workspace plan that offers unlimited is the contact-sales-for-pricing "Enterprise" tier. The $12/user/month plan only includes 2TB/user: https://workspace.google.com/pricing.html https://workspace.google.com/pricing.html

guilhas 5y ago

Is this a good multi threaded local/ssh alternative to rsync?

throwaway999901 5y ago
It is. Another good alternative is lftp.

techisnotmagic 5y ago

I'm not sure why this bothers me so much but it does not "sync between cloud storage providers". I don't really know how this misunderstanding is so pervasive, even among the HN crowd, but no, no amount of magical code can make it possible to directly migrate between S3 and Google Drive. It's going to download, and upload to sync. This is even reflected in the project's description line, which is more accurate than the current HN headline: Rclone ("rsync for cloud storage") is a command-line program to sync files and directories to and from different cloud storage providers. That having been said, it's of course invaluable software. Mounting 10TB of Google Drive into a Raspberry Pi running Plex is still something that feels like it should be harder to accomplish than it is.

trog 5y ago
This was exactly the comment I was looking for. I've had to move a lot of data to GDrive from Dropbox before and have been keeping an eye out for tools to help with this. Using Google Colab is the best way I've found so far - mount your GDrive and you can wget to it directly from Dropbox (or anything else).
BHSPitMonkey 5y ago
Does something the word "sync" require that no intermediary hardware must exist between the endpoints? And does this definition disqualify the use of devices like, say, routers and switches?
trevyn 5y ago
>no amount of magical code can make it possible to directly migrate between S3 and Google Drive writes magical code that fires up an unmetered-bandwidth billed-hourly VPS for a few hours (Scaleway is nice)

divbzero 5y ago

I like rclone for the wide array of storage options [1] plus a special crypt remote [2] that provides client-side encryption for any of the other storage options. There is also a browser GUI you can spin up locally if that floats your boat [3]. [1]: https://rclone.org/overview/ https://rclone.org/overview/ [2]: https://rclone.org/crypt/ https://rclone.org/crypt/ [3]: https://rclone.org/gui/ https://rclone.org/gui/

folli 5y ago

Since we're on the topic: Which cloud provider do you recommend for backup of personal files?

lelandfe 5y ago
I like Backblaze because it works well on my Mac and I don't have to think about it. That's about all I want!
tinus_hn 5y ago
Whatever provider you choose, make sure you encrypt your files locally to avoid the scenario where the provider blocks you because the AI decided one of your files is a virus, a copyright violation or matches some image hash. There is no need for your backup provider to be able to look into your files so there’s only downsides to allowing them to.
prirun 5y ago
I've been using Backblaze B2 since it launched, and HashBackup (I'm the author) was one of the first B2 integrations. What I like most about B2 is the transparent, no gimmick pricing. No monthly minimum, no minimum file size, no paying for storage for 90 days after you delete a file. And they give generous free allowances: 10GB/mo (forever, with no credit card required), 1GB of free egress per day, and up to 5000 free transactions per day. I backup the HashBackup dev server to B2 every night, around 80GB with mostly VM images, and the total backup size while keeping the last 30 days + 12 monthly backups is around 50 GB and costs 25 cents/month. They bill me every few months, I guess when enough charges accumulate. Here are the backup stats for the HashBackup dev server: [jim@bs ~]$ hb stats -c /hbbackup HashBackup #2527 Copyright 2009-2021 HashBackup, LLC Backup directory: /hbbackup 3,617 backups 334 TB file bytes checked since initial backup 80 TB file bytes saved since initial backup 910 total backup hours 79 GB average file bytes checked per backup in last 5 backups 23 GB average file bytes saved per backup in last 5 backups 29.78% average changed data percentage per backup in last 5 backups 13m 43s average backup time for last 5 backups 637 archives 55 GB archive space 84.11% archive space utilization 46 GB 810:1 industry standard dedup ratio 9.5 MB average archive space per backup for last 5 backups 2465:1 reduction ratio of backed up files for last 5 backups 234 MB dedup table current size 9,442,920 dedup table entries 48% dedup table utilization at current size 724,059 files 671,701 paths 10,550,610 blocks 127,965,672 block references 117,415,062 deduped blocks 12.1:1 block dedup ratio 5.2 KB average stored block size 616 GB backup space saved by dedup 49,648 average variable-block length

rsync 5y ago

Here is a keystroke-by-keystroke HOWTO that we wrote last year: https://rsync.net/resources/howto/rclone.html https://rsync.net/resources/howto/rclone.html and although the example is S3 <-> rsync.net, it is not necessarily rsync.net-specific. You could use this HOWTO/workflow for any two targets.

jffry 5y ago
Thanks for this write-up, it's what finally got me over the activation energy barrier to setting up backups of a small home-automation setup I run. The rclone docs are very extensive but having a short, simple example was the kick I needed to get past the myriad options.
earthscienceman 5y ago
Do you contribute code to rclone?
TAForObvReasons 5y ago
FYI You can choose the type of storage by using the short name instead of the index 4 / Amazon S3 Compliant Storage Provider (AWS, Alibaba, Ceph, Digital Ocean, Dreamhost, IBM COS, Minio, etc) \ "s3" "4" is fragile because future changes could change that offset (it happened at least once with Google Drive), but "s3" is fixed.
- rsync 5y ago
  Thanks! I will pass that along and see that it is "fixed" ...

gary_0 5y ago

Past discussions: https://news.ycombinator.com/item?id=12398303 https://news.ycombinator.com/item?id=12398303 (5 years ago) https://news.ycombinator.com/item?id=22791036 https://news.ycombinator.com/item?id=22791036 (2 years ago)

xanaxagoras 5y ago

Rclone is great! All of my duplicati backups go to S3, then in the run-script-after script rclone copies them from S3 to B2.

dang 5y ago

Past related threads: Rclone syncs your files to cloud storage - https://news.ycombinator.com/item?id=29213024 https://news.ycombinator.com/item?id=29213024 - Nov 2021 (8 comments) Rclone – Sync files and directories to many cloud storage providers - https://news.ycombinator.com/item?id=22791036 https://news.ycombinator.com/item?id=22791036 - April 2020 (100 comments) Rclone – Rsync for Cloud Storage - https://news.ycombinator.com/item?id=21906779 https://news.ycombinator.com/item?id=21906779 - Dec 2019 (3 comments) Rclone: rsync for cloud storage - https://news.ycombinator.com/item?id=12398303 https://news.ycombinator.com/item?id=12398303 - Aug 2016 (129 comments) Rclone: rsync for cloud storage - https://news.ycombinator.com/item?id=12049507 https://news.ycombinator.com/item?id=12049507 - July 2016 (1 comment) Rclone – rsync for cloud storage - https://news.ycombinator.com/item?id=12045729 https://news.ycombinator.com/item?id=12045729 - July 2016 (3 comments) Rclone – rsync for cloud storage - https://news.ycombinator.com/item?id=11404178 https://news.ycombinator.com/item?id=11404178 - April 2016 (1 comment)

modeless 5y ago

I once made a software distribution system for software with thousands of files in a huge directory tree using rclone. I'd upload it all to S3 and then clients could run an update script that used rclone to sync just the changed files. No server needed. A better solution probably exists but I couldn't find any other system at the time that could work directly from S3 or any other dumb file host and still selectively sync only changed files.

lazypenguin 5y ago
I recently noticed in the rclone source tree[0] that they are working on making “librclone”. I would love it if they exposed the rclone internals as a library as the only other solutions (e.g. bita, zsync, librsync) still require a fair bit of tedium to use as a generic application updater. [0] https://github.com/rclone/rclone/tree/master/librclone https://github.com/rclone/rclone/tree/master/librclone

joshenders 5y ago

“Rclone” and not “Csync”? Missed opportunity! ;)

sbt567 5y ago

I'm using rclone beta right now for its bisync feature. This was the last missing piece that I badly needed to synchronize my notes. And I'm very happy that it's finally implemented. Kudos to all rclone contributors!

quchen 5y ago
That sounds cool! Is there some preview documentation available somewhere?