6 ms·

RClone. Probably the best tool I have come across for interacting with cloud storage solutions. Move Docs: https://rclone.org/commands/rclone_move/ https://rcl

RClone. Probably the best tool I have come across for interacting with cloud storage solutions. Move Docs: https://rclone.org/commands/rclone_move/ https://rclone.org/commands/rclone_move/ S3 Docs: https://rclone.org/s3/#amazon-s3 https://rclone.org/s3/#amazon-s3 Supports parallel server-side copies and deletes (no server-side moves, unfortunately) so this would have been much faster.

pdimitar 6y ago
Agreed, I regularly sync with servers that have >1M files and rclone makes the whole thing a joy. Used it once with S3 and it performed admirably there as well.
vitaliyf 6y ago
+1 on rclone - one of my favorite tools in the toolbox. With sufficient --transfers and --checkers values you can easily saturate 10-20 gigabit/second links, so with a handful of machines you can transfer 25TB in about an hour, no engineers required. Worth noting that rclone also supports many other storage providers or even plain SSH, so it pretty much obsoletes rsync. It also has many other nifty features useful outside of the "sync files" use case.
thejosh 6y ago
Yep, and it "just works" with Google Drive, which is amazing.
- reincarnate0x14 6y ago
  I have noticed rclone deletes sometimes don't ever get cleared out of Google Drive's trash cache, but that's something you can work around with some periodic check scripts. That trash behavior isn't specific to rclone and probably something on Google's end, but rclone is what I notice it with most.
  - darthShadow 6y ago
    I did get an update from Google recently that they are implementing automatic clearing of trash after 30 days which may help you. The exact wording of the relevant section of the mail: We are writing to let you know that starting October 13, 2020, Google Drive is making a change so that its trash behaves more consistently with the rest of our G Suite services with regards to automatic deletion. This means that any file that is put into Google Drive’s ‘My Drive’ trash will be automatically deleted after 30 days. Items in trash will still continue to consume quota. Please note that starting October 13, 2020 any files already in a user’s trash will remain there for 30 days. After the 30-day-period files that have been in the trash for longer than 30 days will begin to be automatically deleted. What does this mean for my organization? Any file that has been in the trash for longer than 30 days after October 13, 2020 will be automatically deleted. We will be showing in-app messaging in Drive starting September 15, 2020 and in our Editors products (such as Google Docs and Google Forms) starting September 29, 2020. A few things to note: * Files in shared drives trash are already automatically deleted after 30 days. * These changes affect items that are trashed from any device and any platform. * Files deleted from Drive File Stream will be purged from the system trash after 30 days. There is no impact to Backup and Sync behavior. * G Suite administrators can still restore items from any emptied trash on behalf of their users for up to 25 days. * Retention policies set by G Suite administrators in Google Vault are not affected by this change. * These changes apply to all G Suite editions and end-users.
- aloknnikhil 6y ago
  I can vouch for this too. I have a nightly cron job that uses rclone to sync my Apple Time Machine disk to Google Drive and I have yet to see it fail. Particularly, the Time Machine disk is a bunch of tiny (8 MB) files.
fctorial 6y ago
The source bucket had a ton of small files and most of the time was spent making network requests (O(<number of files>)). The suggested solution was to do a batch operation (open a ticket for a batch operation). Would rclone have helped in this scenario?
- oarsinsync 6y ago
  rclone lets you specify how many files to handle in parallel. It defaults to 4, and auto retries in the event of failures. Ramp up the parallelism, and don’t make 7 people monitor it round the clock.
- darthShadow 6y ago
  Yep, as mentioned in the other comment, with configurable `--transfers` & `--checkers`, the only bottleneck is the amount of CPU you have available and can allocate.
  - fctorial 6y ago
    > the only bottleneck is the amount of CPU you have available and can allocate. In this case S3 will be the bottleneck. 25 TB will have around 700M files (assuming it's as dense a linux installation). S3 can do 5000 operations per second. It'll take at least 40 hrs (assuming everything works at peak speed during the whole process).
rsync 6y ago
Are there any cloud storage providers that have 'rclone' built into their environment ? So I could, for instance: ssh user@rsync.net rclone s3:/some/bucket gdrive:/blah or: ssh user@rsync.net rclone s3:/source/files /my/rsync.net/account Well I'll be damned! It appears that there is such a provider : https://www.rsync.net/products/rclone.html https://www.rsync.net/products/rclone.html https://www.rsync.net/products/universal.html https://www.rsync.net/products/universal.html
- brylie 6y ago
  > NO Charges for ingress/egress Is this too good to be true?! Thank you for sharing! Yet another gem found in HN comments.
  - rsync 6y ago
    Don't thank us, thank Hurricane Electric (he.net). 20 years ago, I was on irc.lightning.net and thought that was the best run efnet server. The MOTD advertised their IP transit services. They became he.net and that is why rsync.net does (most) of their IP transit with them.
  - chromedev 6y ago
    Google Cloud doesn't charge for egress to Google Drive as well. If you use GCP to transfer files to Google Drive via rclone, then you can download them elsewhere without any charges.
  - throwaway9d0291 6y ago
    No, the tradeoff is that even with volume, rsync.net storage costs are $15/TBmonth [0], which is ~3x managed alternatives [1,2] and ~10x DIY alternatives [3]. To be fair, ZFS snapshots are "free" but whether that's worth 3x cost will depend on your data. The lack of bandwidth charges sounds really nice but [1,3] have that too and unlike [1,3], Rsync.net doesn't support protocols that are suited for bandwidth-intensive applications (like serving large files to end-users). It only supports SSH-based transfers [4]. [0]: https://www.rsync.net/pricing.html https://www.rsync.net/pricing.html [1]: https://wasabi.com/cloud-storage-pricing/ https://wasabi.com/cloud-storage-pricing/ [2]: https://www.backblaze.com/b2/cloud-storage-pricing.html https://www.backblaze.com/b2/cloud-storage-pricing.html [3]: https://www.hetzner.de/dedicated-rootserver/matrix-sx?country=us https://www.hetzner.de/dedicated-rootserver/matrix-sx?countr... [4]: https://www.rsync.net/products/platform.html https://www.rsync.net/products/platform.html
HenryBemis 6y ago
Silly question.. can't one just cut off/limit access to the "previous"? And then let it the time it needs to Delete? I saw in the Reddit discussion that one can set up role/privileges (as expected), use the (new) user/role/group X for the move, and cut all others. I understand that the original planning went bust 2h --> 48h, but that it buffled me was the "It sounds more like you had 2 hours for the delete part of the operation."
kureikain 6y ago
Doesn't RClone still need to download files to the server from the source bucket, before syncing to dest bucket. That means it still has some upper limit(CPU, network of where we run that). S3 replication and email to AWS supports to apply replication for existing object(per reddit discussion) seems a better method.

pdimitar 6y ago

Agreed, I regularly sync with servers that have >1M files and rclone makes the whole thing a joy. Used it once with S3 and it performed admirably there as well.

vitaliyf 6y ago

+1 on rclone - one of my favorite tools in the toolbox. With sufficient --transfers and --checkers values you can easily saturate 10-20 gigabit/second links, so with a handful of machines you can transfer 25TB in about an hour, no engineers required. Worth noting that rclone also supports many other storage providers or even plain SSH, so it pretty much obsoletes rsync. It also has many other nifty features useful outside of the "sync files" use case.

thejosh 6y ago

Yep, and it "just works" with Google Drive, which is amazing.

reincarnate0x14 6y ago
I have noticed rclone deletes sometimes don't ever get cleared out of Google Drive's trash cache, but that's something you can work around with some periodic check scripts. That trash behavior isn't specific to rclone and probably something on Google's end, but rclone is what I notice it with most.
- darthShadow 6y ago
  I did get an update from Google recently that they are implementing automatic clearing of trash after 30 days which may help you. The exact wording of the relevant section of the mail: We are writing to let you know that starting October 13, 2020, Google Drive is making a change so that its trash behaves more consistently with the rest of our G Suite services with regards to automatic deletion. This means that any file that is put into Google Drive’s ‘My Drive’ trash will be automatically deleted after 30 days. Items in trash will still continue to consume quota. Please note that starting October 13, 2020 any files already in a user’s trash will remain there for 30 days. After the 30-day-period files that have been in the trash for longer than 30 days will begin to be automatically deleted. What does this mean for my organization? Any file that has been in the trash for longer than 30 days after October 13, 2020 will be automatically deleted. We will be showing in-app messaging in Drive starting September 15, 2020 and in our Editors products (such as Google Docs and Google Forms) starting September 29, 2020. A few things to note: * Files in shared drives trash are already automatically deleted after 30 days. * These changes affect items that are trashed from any device and any platform. * Files deleted from Drive File Stream will be purged from the system trash after 30 days. There is no impact to Backup and Sync behavior. * G Suite administrators can still restore items from any emptied trash on behalf of their users for up to 25 days. * Retention policies set by G Suite administrators in Google Vault are not affected by this change. * These changes apply to all G Suite editions and end-users.
aloknnikhil 6y ago
I can vouch for this too. I have a nightly cron job that uses rclone to sync my Apple Time Machine disk to Google Drive and I have yet to see it fail. Particularly, the Time Machine disk is a bunch of tiny (8 MB) files.

fctorial 6y ago

The source bucket had a ton of small files and most of the time was spent making network requests (O(<number of files>)). The suggested solution was to do a batch operation (open a ticket for a batch operation). Would rclone have helped in this scenario?

oarsinsync 6y ago
rclone lets you specify how many files to handle in parallel. It defaults to 4, and auto retries in the event of failures. Ramp up the parallelism, and don’t make 7 people monitor it round the clock.
darthShadow 6y ago
Yep, as mentioned in the other comment, with configurable `--transfers` & `--checkers`, the only bottleneck is the amount of CPU you have available and can allocate.
- fctorial 6y ago
  > the only bottleneck is the amount of CPU you have available and can allocate. In this case S3 will be the bottleneck. 25 TB will have around 700M files (assuming it's as dense a linux installation). S3 can do 5000 operations per second. It'll take at least 40 hrs (assuming everything works at peak speed during the whole process).

rsync 6y ago

Are there any cloud storage providers that have 'rclone' built into their environment ? So I could, for instance: ssh user@rsync.net rclone s3:/some/bucket gdrive:/blah or: ssh user@rsync.net rclone s3:/source/files /my/rsync.net/account Well I'll be damned! It appears that there is such a provider : https://www.rsync.net/products/rclone.html https://www.rsync.net/products/rclone.html https://www.rsync.net/products/universal.html https://www.rsync.net/products/universal.html

brylie 6y ago
> NO Charges for ingress/egress Is this too good to be true?! Thank you for sharing! Yet another gem found in HN comments.
- rsync 6y ago
  Don't thank us, thank Hurricane Electric (he.net). 20 years ago, I was on irc.lightning.net and thought that was the best run efnet server. The MOTD advertised their IP transit services. They became he.net and that is why rsync.net does (most) of their IP transit with them.
- chromedev 6y ago
  Google Cloud doesn't charge for egress to Google Drive as well. If you use GCP to transfer files to Google Drive via rclone, then you can download them elsewhere without any charges.
- throwaway9d0291 6y ago
  No, the tradeoff is that even with volume, rsync.net storage costs are $15/TBmonth [0], which is ~3x managed alternatives [1,2] and ~10x DIY alternatives [3]. To be fair, ZFS snapshots are "free" but whether that's worth 3x cost will depend on your data. The lack of bandwidth charges sounds really nice but [1,3] have that too and unlike [1,3], Rsync.net doesn't support protocols that are suited for bandwidth-intensive applications (like serving large files to end-users). It only supports SSH-based transfers [4]. [0]: https://www.rsync.net/pricing.html https://www.rsync.net/pricing.html [1]: https://wasabi.com/cloud-storage-pricing/ https://wasabi.com/cloud-storage-pricing/ [2]: https://www.backblaze.com/b2/cloud-storage-pricing.html https://www.backblaze.com/b2/cloud-storage-pricing.html [3]: https://www.hetzner.de/dedicated-rootserver/matrix-sx?country=us https://www.hetzner.de/dedicated-rootserver/matrix-sx?countr... [4]: https://www.rsync.net/products/platform.html https://www.rsync.net/products/platform.html

HenryBemis 6y ago

Silly question.. can't one just cut off/limit access to the "previous"? And then let it the time it needs to Delete? I saw in the Reddit discussion that one can set up role/privileges (as expected), use the (new) user/role/group X for the move, and cut all others. I understand that the original planning went bust 2h --> 48h, but that it buffled me was the "It sounds more like you had 2 hours for the delete part of the operation."

kureikain 6y ago

Doesn't RClone still need to download files to the server from the source bucket, before syncing to dest bucket. That means it still has some upper limit(CPU, network of where we run that). S3 replication and email to AWS supports to apply replication for existing object(per reddit discussion) seems a better method.