6 ms·

For context: Amazon Cloud Drive is a Google Drive/Dropbox analog, except that it provides "unlimited" storage space and bandwidth for $5/month. The built-in use

For context: Amazon Cloud Drive is a Google Drive/Dropbox analog, except that it provides "unlimited" storage space and bandwidth for $5/month. The built-in user interface requires you to manually upload/download files through a web browser, but the service also supports an API for programmatic access. So tools like rclone and acd_cli were developed to let you do bulk transfers and/or mount your storage as a network filesystem, optionally with encryption (which defeats any attempts Amazon might try to use for deduplication). Now Amazon is suffering the obvious consequence of offering unlimited storage: people are using it to store tens of terabytes of media and/or backups at very low cost. In an attempt to kill off heavy users, they shut down registration of new API keys several months ago, and now they're systematically revoking the API keys used by popular open-source tools.

QUFB 9y ago
$5/month for unlimited access via API was really a steal. This would buy you only 217 GB via standard S3 storage, and then you'd have to pay an exorbitant $0.090 per GB for egress data transfer; you'd burn through your $5 after downloading 55 GB.
brandur 9y ago
Yeah, this is a hard one. There are users on /r/DataHoarder/ that claim to have uploaded literally 100s of (encrypted) TBs to Cloud Drive, which is plainly unsustainable and abusive. On the other hand, they've also killed the product for a lot of more legitimate users as well. The Amazon web interface and apps for Cloud Drive are obnoxiously terrible, and Rclone really is just a better way to use it. I've been using it to sync 10s of GBs of photos between all my different computers, but with Rclone unavailable, I'll have to fall back to Google Drive, S3, or some other option (the unlimitedness of Cloud Drive was good peace of mind). I'll be keeping an eye on it over the next few weeks to see whether shipping a binary with OAuth secrets was actually the reason for the ban, or just a pretext for getting the Rclone users off the service (personally, I suspect the latter).
- saurik 9y ago
  So why not just take apart the official apps for Google Cloud Drive and use whatever API and authority into has to build your alternative client?
  - brandur 9y ago
    The nice thing about Rclone is that it already supports Google Cloud Drive (and a host of other providers) so the technology isn't the problem. I'm lamenting Amazon Cloud Drive in particular because it was the best deal. $60 a year for unlimited storage and with no caveats (or so it seemed before last week).
    - saurik 9y ago
      I am sorry, "Google" was clearly a typo: I meant "Amazon". You do not need Amazon's permission to write a client that targets their API.
- Oletros 9y ago
  If a service claims that they provide UNLIMITED storage why using it is abusing?
  - burkaman 9y ago
    abuse: use (something) to bad effect or for a bad purpose; misuse. Abuse doesn't mean breaking the rules. It's like going to an all-you-can-eat buffet and staying for a week. Or taking a job with unlimited vacation time and coming to work once a month.
    - Oletros 9y ago
      Then we agree to disagree.
      - burkaman 9y ago
        What do we disagree about?
joering2 9y ago
I think DropBox will shut down their API in July of this year. I got email warning me about six months ago - since then I moved our internal system to FTP...
- hemancuso 9y ago
  They just updated it to a new/improved version of the API
msravi 9y ago
I'm a bit skeptical about how much deduplication saves them. I have a couple of 100 gigs on ACD, unencrypted, most being family photos/videos - I don't think they're going to find any portion of those that's remotely common to other users - at least, I hope not!
- FroshKiller 9y ago
  If it's file-level deduplication, sure. But if it's chunk-level, I think you may find that you actually have many chunks identical to many other users' chunks.
  - nimrody 9y ago
    Unlikely given any non-negligible chunk size and accounting for the randomization effect of compression (image and video). I think deduplication buys them most when people upload common assets - videos downloaded from the net, ebooks, ISO images, etc.
    - FroshKiller 9y ago
      Doesn't compression generally reduce entropy?
clumsysmurf 9y ago
> now they're systematically revoking the API keys used by popular open-source tools. Would this also impact Arq users? I was hoping Arq would support Backblaze...
Veratyr 9y ago
> In an attempt to kill off heavy users, they shut down registration of new API keys several months ago, and now they're systematically revoking the API keys used by popular open-source tools. Actually this seems to have been more triggered by a serious issue with acd_cli's authentication server that resulted in its users seeing other people's files: - https://web.archive.org/web/20170514020241/https://github.com/yadayada/acd_cli/issues/549 https://web.archive.org/web/20170514020241/https://github.co... - https://www.reddit.com/r/DataHoarder/comments/6bi5p5/amazons_reply_about_the_acd_cli_issue/dhmrdl1/ https://www.reddit.com/r/DataHoarder/comments/6bi5p5/amazons... Amazon has started paying a lot more attention to open source tools as a result.

QUFB 9y ago

$5/month for unlimited access via API was really a steal. This would buy you only 217 GB via standard S3 storage, and then you'd have to pay an exorbitant $0.090 per GB for egress data transfer; you'd burn through your $5 after downloading 55 GB.

brandur 9y ago

Yeah, this is a hard one. There are users on /r/DataHoarder/ that claim to have uploaded literally 100s of (encrypted) TBs to Cloud Drive, which is plainly unsustainable and abusive. On the other hand, they've also killed the product for a lot of more legitimate users as well. The Amazon web interface and apps for Cloud Drive are obnoxiously terrible, and Rclone really is just a better way to use it. I've been using it to sync 10s of GBs of photos between all my different computers, but with Rclone unavailable, I'll have to fall back to Google Drive, S3, or some other option (the unlimitedness of Cloud Drive was good peace of mind). I'll be keeping an eye on it over the next few weeks to see whether shipping a binary with OAuth secrets was actually the reason for the ban, or just a pretext for getting the Rclone users off the service (personally, I suspect the latter).

saurik 9y ago
So why not just take apart the official apps for Google Cloud Drive and use whatever API and authority into has to build your alternative client?
- brandur 9y ago
  The nice thing about Rclone is that it already supports Google Cloud Drive (and a host of other providers) so the technology isn't the problem. I'm lamenting Amazon Cloud Drive in particular because it was the best deal. $60 a year for unlimited storage and with no caveats (or so it seemed before last week).
  - saurik 9y ago
    I am sorry, "Google" was clearly a typo: I meant "Amazon". You do not need Amazon's permission to write a client that targets their API.
Oletros 9y ago
If a service claims that they provide UNLIMITED storage why using it is abusing?
- burkaman 9y ago
  abuse: use (something) to bad effect or for a bad purpose; misuse. Abuse doesn't mean breaking the rules. It's like going to an all-you-can-eat buffet and staying for a week. Or taking a job with unlimited vacation time and coming to work once a month.
  - Oletros 9y ago
    Then we agree to disagree.
    - burkaman 9y ago
      What do we disagree about?

joering2 9y ago

I think DropBox will shut down their API in July of this year. I got email warning me about six months ago - since then I moved our internal system to FTP...

hemancuso 9y ago
They just updated it to a new/improved version of the API

msravi 9y ago

I'm a bit skeptical about how much deduplication saves them. I have a couple of 100 gigs on ACD, unencrypted, most being family photos/videos - I don't think they're going to find any portion of those that's remotely common to other users - at least, I hope not!

FroshKiller 9y ago
If it's file-level deduplication, sure. But if it's chunk-level, I think you may find that you actually have many chunks identical to many other users' chunks.
- nimrody 9y ago
  Unlikely given any non-negligible chunk size and accounting for the randomization effect of compression (image and video). I think deduplication buys them most when people upload common assets - videos downloaded from the net, ebooks, ISO images, etc.
  - FroshKiller 9y ago
    Doesn't compression generally reduce entropy?

clumsysmurf 9y ago

> now they're systematically revoking the API keys used by popular open-source tools. Would this also impact Arq users? I was hoping Arq would support Backblaze...

Veratyr 9y ago

> In an attempt to kill off heavy users, they shut down registration of new API keys several months ago, and now they're systematically revoking the API keys used by popular open-source tools. Actually this seems to have been more triggered by a serious issue with acd_cli's authentication server that resulted in its users seeing other people's files: - https://web.archive.org/web/20170514020241/https://github.com/yadayada/acd_cli/issues/549 https://web.archive.org/web/20170514020241/https://github.co... - https://www.reddit.com/r/DataHoarder/comments/6bi5p5/amazons_reply_about_the_acd_cli_issue/dhmrdl1/ https://www.reddit.com/r/DataHoarder/comments/6bi5p5/amazons... Amazon has started paying a lot more attention to open source tools as a result.