4 ms·

"Correct. 7 days is not enough time to stream >200TB of data, let alone find a place to stream it to. This puts a human being in a crisis position." Agreed tha

by rsync 3y ago

"Correct. 7 days is not enough time to stream >200TB of data, let alone find a place to stream it to. This puts a human being in a crisis position." Agreed that this is a crisis but it is not impossible. You can use 'rclone' - running on your own system or hosted elsewhere - to move data between cloud providers. You don't rely on your own bandwidth - they talk directly to one another: rclone gdrive:blah s3:some/bucket/whatever ... and if you don't want to bother even installing rclone, you can run it somewhere it is already installed: ssh user@rsync.net rclone gdrive:blah s3:some/bucket/whatever Which means ... for either the price of (zero) or (minimum rsync.net) you can move that data at (whatever speeds google <--> aws are capable of). I am, admittedly, hand-waving away the actual configuration of those "remotes" as they are called, which involve your login and/or API keys, etc. - here is an example of what that config might look like: https://www.rsync.net/resources/howto/rclone.html https://www.rsync.net/resources/howto/rclone.html

mderazon 3y ago
I've been looking for a solution to backup Google Takeout periodically to Back laze. Can rclone be used with Google Takeout ?
zamadatix 3y ago
This still requires Google Drive be able to serve your 237 TB at over 3 gbps for the full 7 days. In my experience this is not the case with cloud storage offerings. This guy realistically had plenty of time to figure it out without a fancy transfer method though. The real problem is they don't want to pay $$$ for somewhere else to host the files and have now reached the point it has become a problem.
alwa 3y ago
(Minimum rsync.net) is still (237.22TB x 1024 x $0.006/GB =) $1457/month. Most “independent journalists” I know aren’t operating on the kind of household budget where they can handwave that kind of bill away.
- sangnoir 3y ago
  Perhaps they ought to invest in tape drives and (encrypted) offsite backups at friends' homes. If they want convenience of always-available data, they have to pay for that
- rsync 3y ago
  You misunderstand. You're not storing any data on rsync.net - you're just using our 'rclone' that is built into the platform. The minimum is $9/mo. Or less if you ask about the HN discount.
  - alwa 3y ago
    Oh! That’s very cool, I definitely did misunderstand. That seems like incredible value for that kind of connectivity. Thank you for broadening my toolset. I can immediately think of a few little projects where such a technique will be useful.
sllabres 3y ago
I wasn't aware that rclone could create a direct connections and had a quick skim through the documentation where I found this in the FAQ (https://rclone.org/faq/ https://rclone.org/faq/) ---- Can rclone sync directly from drive to s3 Rclone can sync between two remote cloud storage systems just fine. Note that it effectively downloads the file and uploads it again, so the node running rclone would need to have lots of bandwidth. ----

by rsync 3y ago

mderazon 3y ago

I've been looking for a solution to backup Google Takeout periodically to Back laze. Can rclone be used with Google Takeout ?

zamadatix 3y ago

This still requires Google Drive be able to serve your 237 TB at over 3 gbps for the full 7 days. In my experience this is not the case with cloud storage offerings. This guy realistically had plenty of time to figure it out without a fancy transfer method though. The real problem is they don't want to pay $$$ for somewhere else to host the files and have now reached the point it has become a problem.

alwa 3y ago

(Minimum rsync.net) is still (237.22TB x 1024 x $0.006/GB =) $1457/month. Most “independent journalists” I know aren’t operating on the kind of household budget where they can handwave that kind of bill away.

sangnoir 3y ago
Perhaps they ought to invest in tape drives and (encrypted) offsite backups at friends' homes. If they want convenience of always-available data, they have to pay for that
rsync 3y ago
You misunderstand. You're not storing any data on rsync.net - you're just using our 'rclone' that is built into the platform. The minimum is $9/mo. Or less if you ask about the HN discount.
- alwa 3y ago
  Oh! That’s very cool, I definitely did misunderstand. That seems like incredible value for that kind of connectivity. Thank you for broadening my toolset. I can immediately think of a few little projects where such a technique will be useful.

sllabres 3y ago

I wasn't aware that rclone could create a direct connections and had a quick skim through the documentation where I found this in the FAQ (https://rclone.org/faq/ https://rclone.org/faq/) ---- Can rclone sync directly from drive to s3 Rclone can sync between two remote cloud storage systems just fine. Note that it effectively downloads the file and uploads it again, so the node running rclone would need to have lots of bandwidth. ----