7 ms·

Rclone banned from Amazon Drive for having the encrypted secrets in code

89 points by msravi 9y ago

firloop 9y ago
Link to the infringing source, for the curious: https://github.com/ncw/rclone/blob/a9d29c22645c9b5d2ab938ebbbed0809a2044087/amazonclouddrive/amazonclouddrive.go#L37-L38 https://github.com/ncw/rclone/blob/a9d29c22645c9b5d2ab938ebb...
- msravi 9y ago
  > My guess is that the software will probably just require users to get their own secrets as part of the initial configuration. Amazon has stopped giving out any more API keys (https://developer.amazon.com/amazon-drive https://developer.amazon.com/amazon-drive). So I guess that isn't an option unless Amazon opens it up again...
- bm1362 9y ago
  So is this basically sharing the same API key? Wouldn't that make using this insecure, as you'd be seeing the same data as other rclone users?
  - koolba 9y ago
    Not if the source data is encrypted using a local key. What may be possible is deleting other user's data as I believe it'd all be authorized with the same API key.
    - tgsovlerkhgsel 9y ago
      In OAuth, you have basically two tokens: One for the "software"/"third party service provider", one for the account itself. So for example, if you want to use SomeMagicTweetTool to send tweets, they use their token to prove it's them, and you grant them a special token for your account. Someone who has their token can pretend to be them, use their quota if there is per-software/per-provider quota, but cannot access your account unless you gave them a token for your account.
- niftich 9y ago
  See also, the de-obfuscation wrapper [1], decryption implementation [2], and the key [3]. [1] https://github.com/ncw/rclone/blob/66e8c1600ec8c18cbcdce56647b63488599d1b95/fs/config.go#L161-L185 https://github.com/ncw/rclone/blob/66e8c1600ec8c18cbcdce5664... [2] https://github.com/ncw/rclone/blob/66e8c1600ec8c18cbcdce56647b63488599d1b95/fs/config.go#L118-L134 https://github.com/ncw/rclone/blob/66e8c1600ec8c18cbcdce5664... [3] https://github.com/ncw/rclone/blob/66e8c1600ec8c18cbcdce56647b63488599d1b95/fs/config.go#L106-L116 https://github.com/ncw/rclone/blob/66e8c1600ec8c18cbcdce5664...
teraflop 9y ago
For context: Amazon Cloud Drive is a Google Drive/Dropbox analog, except that it provides "unlimited" storage space and bandwidth for $5/month. The built-in user interface requires you to manually upload/download files through a web browser, but the service also supports an API for programmatic access. So tools like rclone and acd_cli were developed to let you do bulk transfers and/or mount your storage as a network filesystem, optionally with encryption (which defeats any attempts Amazon might try to use for deduplication). Now Amazon is suffering the obvious consequence of offering unlimited storage: people are using it to store tens of terabytes of media and/or backups at very low cost. In an attempt to kill off heavy users, they shut down registration of new API keys several months ago, and now they're systematically revoking the API keys used by popular open-source tools.
- QUFB 9y ago
  $5/month for unlimited access via API was really a steal. This would buy you only 217 GB via standard S3 storage, and then you'd have to pay an exorbitant $0.090 per GB for egress data transfer; you'd burn through your $5 after downloading 55 GB.
- brandur 9y ago
  Yeah, this is a hard one. There are users on /r/DataHoarder/ that claim to have uploaded literally 100s of (encrypted) TBs to Cloud Drive, which is plainly unsustainable and abusive. On the other hand, they've also killed the product for a lot of more legitimate users as well. The Amazon web interface and apps for Cloud Drive are obnoxiously terrible, and Rclone really is just a better way to use it. I've been using it to sync 10s of GBs of photos between all my different computers, but with Rclone unavailable, I'll have to fall back to Google Drive, S3, or some other option (the unlimitedness of Cloud Drive was good peace of mind). I'll be keeping an eye on it over the next few weeks to see whether shipping a binary with OAuth secrets was actually the reason for the ban, or just a pretext for getting the Rclone users off the service (personally, I suspect the latter).
  - saurik 9y ago
    So why not just take apart the official apps for Google Cloud Drive and use whatever API and authority into has to build your alternative client?
geofft 9y ago
From the discussion at https://www.reddit.com/r/crypto/comments/61zupq/amazon_drive_developer_guide_dont_build_apps_that/ https://www.reddit.com/r/crypto/comments/61zupq/amazon_drive... I learned that Amazon Drive is a service that provides unlimited storage for a fixed cost, and that various applications, including rclone, use it for an application that I would expect (as a developer) to use S3 for, and therefore to charge my users per amount of data stored. (Amazon Drive is not part of AWS, I believe.) I am not extremely surprised that the terms of service of Amazon Drive make it hard to build a compliant application like what rclone is trying to build. I'd expect they can do this with S3 straightforwardly -- at the expense of users needing to spend O(n) instead of O(1) on their data.
- bad_user 9y ago
  I paid for Amazon Drive in order to do backups, some of them with rclone. I am not using Amazon Drive for commercial purposes. They said Unlimited storage, they should provide Unlimited storage, otherwise it's false advertising. In other words, I hate apologists.
  - chaosfox 9y ago
    they are providing unlimited storage sure, but they also say they can terminate your account at any moment for any reason. If you stop to think about this part for a second you will realize this service is completely inadequate for backups.
    - bm1362 9y ago
      I agree, if you want to hold them to their word then you should include all of them. They agreed to unlimited storage at their discretion, and so did you.
    - scott00 9y ago
      The difference here is they splash "Unlimited" on their website in 32 point font in the 1 sentence summary, and put "at our discretion" buried in their 2500 word terms of service. If they want to operate the service under those terms, fine, but they should have to advertise "as little as 0 bytes of storage", not "unlimited".
niftich 9y ago
In a comment of mine earlier this week [1], I (along with others) speculated whether the shipping of OAuth2 secrets was the reason for the ban, and whether they're other contributing, strategic factors. I wrote: Amazon has revoked rclone's OAuth2 API key. However, consider that rclone's default OAuth2 client id and secret are compiled into the rclone executable, and thus effectively public; aka. anyone can extract them and pretend to be rclone, and fool users into obtaining access and abuse them for unrelated purposes. A far better option is for the cloud provider to let users generate their own OAuth2 clients, such as Google does (and supposedly Microsoft, although for me it's always errored out). Unfortunately, Amazon has a "call us" style of Developer access, which effectively translates to no new API access being granted to these types of users. The speculation around the web is that Amazon also wanted to shut this down because they offered "unlimited" storage, and people were using it to store very large amounts of hard-to-compress, hard-to-dedup data. Breaking a popular tool used to accomplish this (e.g. it supported on-the-fly encryption, producing the exact style of difficult data) will cause some portion of less profitable users to migrate elsewhere. This may or may not be true, but it's certainly an intriguing point. Some companies like Google allow easy registration of OAuth2 clients, so any rclone user can make their own API keys at console.developers.google.com and feed those to rclone, instead of having to use the built-in credential. Rclone's documentation refers to this ability, but positions it as a performance improvement for advanced users to sidestep a shared rclone-wide quota, rather than for any other purpose. However, it, unlike many, many other applications, at least lets you plug in your own credential -- this shouldn't be an exotic flow, but rather the normal way to grant access to a third-party application, instead of distributing a hardcoded secret. Sadly, the app developer community has been slow to realize and demand this, and providers have been slow to implement it. Google lets you register as many applications as you want; GitHub even lets you make your own OAuth2 scopes. But there's few others who come close. [1] https://news.ycombinator.com/item?id=14380106 https://news.ycombinator.com/item?id=14380106
- eridius 9y ago
  > Some companies like Google allow easy registration of OAuth2 clients, so any rclone user can make their own API keys at console.developers.google.com and feed those to rclone, instead of having to use the built-in credential How does making new Google API keys help with an Amazon service?
jedberg 9y ago
This is a perfect example of the tragedy of the commons. No one person feels the impact of abuse, but ends up ruining it for everyone. Clearly they didn't intend for this service to be a general backup system.
- SeanDav 9y ago
  This is a perfect example of why companies should never offer unlimited services, if they don't expect users to actually take them up on the offer... While I can understand and agree that fair-play should be a part of the service, if the offer on hand is Unlimited, then it is not abuse to upload tera-bytes, peta-bytes, exa-bytes or even zetta-bytes. Amazon is hardly a neophyte as far as understanding big data, they are very well placed to know that offering unlimited storage could actually attract really big chunks of data.
  - detaro 9y ago
    They also aren't a new to reserve to right to cut users off and knowing that the wider world doesn't care much if they do that to "extreme cases". And on the other side, many of the more extreme users of drive probably aren't new to storing a lot of data and knew that Amazons offer had to be unsustainable when used to much and thus would disappear in some way. I'm not particular sympathetic to either sides messaging here. It's sad that rclone got caught in the middle of it.
    - harshreality 9y ago
      100% Amazon's fault. They could have simply said something like, "We reserve the right to cap amazon cloud drive usage at <x> TB (currently, but increased occasionally)." But they don't want to do that, apparently. Why? I don't know, but it looks like they're worried some of their customers (the same ones they don't like because of excessive usage) will go over to B2 or whatever other cheap services are available, for the peace of mind of not running into any caps even if the service costs more at 2TB or 5TB worth of storage. "Unlimited" storage companies, that nevertheless discriminate against very high usage, want to have their cake and eat it too. Either state a limit and accept the risk that some of your customers decide to go somewhere else, or don't limit and don't discriminate against them. Those are the reasonable choices. Other choices are not reasonable.
markwillis82 9y ago
Notice our backups failing over the weekend. Set up a new account with backblaze and re-ran the backups. It sounded to good to be true, fortunately we were still on the trial.
- rsync 9y ago
  Backblaze is also unlimited, flat rate storage and is therefore also too good to be true. "paying a flat rate for unlimited storage, or transfer, pits you against your provider in an antagonistic relationship. This is not the kind of relationship you want to have with someone providing critical functions."[1] [1] http://blog.kozubik.com/john_kozubik/2009/11/flat-rate-storage-services-vs-rsyncnet.html http://blog.kozubik.com/john_kozubik/2009/11/flat-rate-stora...

89 points by msravi 9y ago

Link to the infringing source, for the curious: https://github.com/ncw/rclone/blob/a9d29c22645c9b5d2ab938ebbbed0809a2044087/amazonclouddrive/amazonclouddrive.go#L37-L38 https://github.com/ncw/rclone/blob/a9d29c22645c9b5d2ab938ebb...

msravi 9y ago
> My guess is that the software will probably just require users to get their own secrets as part of the initial configuration. Amazon has stopped giving out any more API keys (https://developer.amazon.com/amazon-drive https://developer.amazon.com/amazon-drive). So I guess that isn't an option unless Amazon opens it up again...
bm1362 9y ago
So is this basically sharing the same API key? Wouldn't that make using this insecure, as you'd be seeing the same data as other rclone users?
- koolba 9y ago
  Not if the source data is encrypted using a local key. What may be possible is deleting other user's data as I believe it'd all be authorized with the same API key.
  - tgsovlerkhgsel 9y ago
    In OAuth, you have basically two tokens: One for the "software"/"third party service provider", one for the account itself. So for example, if you want to use SomeMagicTweetTool to send tweets, they use their token to prove it's them, and you grant them a special token for your account. Someone who has their token can pretend to be them, use their quota if there is per-software/per-provider quota, but cannot access your account unless you gave them a token for your account.
niftich 9y ago
See also, the de-obfuscation wrapper [1], decryption implementation [2], and the key [3]. [1] https://github.com/ncw/rclone/blob/66e8c1600ec8c18cbcdce56647b63488599d1b95/fs/config.go#L161-L185 https://github.com/ncw/rclone/blob/66e8c1600ec8c18cbcdce5664... [2] https://github.com/ncw/rclone/blob/66e8c1600ec8c18cbcdce56647b63488599d1b95/fs/config.go#L118-L134 https://github.com/ncw/rclone/blob/66e8c1600ec8c18cbcdce5664... [3] https://github.com/ncw/rclone/blob/66e8c1600ec8c18cbcdce56647b63488599d1b95/fs/config.go#L106-L116 https://github.com/ncw/rclone/blob/66e8c1600ec8c18cbcdce5664...

teraflop 9y ago

For context: Amazon Cloud Drive is a Google Drive/Dropbox analog, except that it provides "unlimited" storage space and bandwidth for $5/month. The built-in user interface requires you to manually upload/download files through a web browser, but the service also supports an API for programmatic access. So tools like rclone and acd_cli were developed to let you do bulk transfers and/or mount your storage as a network filesystem, optionally with encryption (which defeats any attempts Amazon might try to use for deduplication). Now Amazon is suffering the obvious consequence of offering unlimited storage: people are using it to store tens of terabytes of media and/or backups at very low cost. In an attempt to kill off heavy users, they shut down registration of new API keys several months ago, and now they're systematically revoking the API keys used by popular open-source tools.

QUFB 9y ago
$5/month for unlimited access via API was really a steal. This would buy you only 217 GB via standard S3 storage, and then you'd have to pay an exorbitant $0.090 per GB for egress data transfer; you'd burn through your $5 after downloading 55 GB.
brandur 9y ago
Yeah, this is a hard one. There are users on /r/DataHoarder/ that claim to have uploaded literally 100s of (encrypted) TBs to Cloud Drive, which is plainly unsustainable and abusive. On the other hand, they've also killed the product for a lot of more legitimate users as well. The Amazon web interface and apps for Cloud Drive are obnoxiously terrible, and Rclone really is just a better way to use it. I've been using it to sync 10s of GBs of photos between all my different computers, but with Rclone unavailable, I'll have to fall back to Google Drive, S3, or some other option (the unlimitedness of Cloud Drive was good peace of mind). I'll be keeping an eye on it over the next few weeks to see whether shipping a binary with OAuth secrets was actually the reason for the ban, or just a pretext for getting the Rclone users off the service (personally, I suspect the latter).
- saurik 9y ago
  So why not just take apart the official apps for Google Cloud Drive and use whatever API and authority into has to build your alternative client?

geofft 9y ago

From the discussion at https://www.reddit.com/r/crypto/comments/61zupq/amazon_drive_developer_guide_dont_build_apps_that/ https://www.reddit.com/r/crypto/comments/61zupq/amazon_drive... I learned that Amazon Drive is a service that provides unlimited storage for a fixed cost, and that various applications, including rclone, use it for an application that I would expect (as a developer) to use S3 for, and therefore to charge my users per amount of data stored. (Amazon Drive is not part of AWS, I believe.) I am not extremely surprised that the terms of service of Amazon Drive make it hard to build a compliant application like what rclone is trying to build. I'd expect they can do this with S3 straightforwardly -- at the expense of users needing to spend O(n) instead of O(1) on their data.

bad_user 9y ago
I paid for Amazon Drive in order to do backups, some of them with rclone. I am not using Amazon Drive for commercial purposes. They said Unlimited storage, they should provide Unlimited storage, otherwise it's false advertising. In other words, I hate apologists.
- chaosfox 9y ago
  they are providing unlimited storage sure, but they also say they can terminate your account at any moment for any reason. If you stop to think about this part for a second you will realize this service is completely inadequate for backups.
  - bm1362 9y ago
    I agree, if you want to hold them to their word then you should include all of them. They agreed to unlimited storage at their discretion, and so did you.
  - scott00 9y ago
    The difference here is they splash "Unlimited" on their website in 32 point font in the 1 sentence summary, and put "at our discretion" buried in their 2500 word terms of service. If they want to operate the service under those terms, fine, but they should have to advertise "as little as 0 bytes of storage", not "unlimited".

niftich 9y ago

In a comment of mine earlier this week [1], I (along with others) speculated whether the shipping of OAuth2 secrets was the reason for the ban, and whether they're other contributing, strategic factors. I wrote: Amazon has revoked rclone's OAuth2 API key. However, consider that rclone's default OAuth2 client id and secret are compiled into the rclone executable, and thus effectively public; aka. anyone can extract them and pretend to be rclone, and fool users into obtaining access and abuse them for unrelated purposes. A far better option is for the cloud provider to let users generate their own OAuth2 clients, such as Google does (and supposedly Microsoft, although for me it's always errored out). Unfortunately, Amazon has a "call us" style of Developer access, which effectively translates to no new API access being granted to these types of users. The speculation around the web is that Amazon also wanted to shut this down because they offered "unlimited" storage, and people were using it to store very large amounts of hard-to-compress, hard-to-dedup data. Breaking a popular tool used to accomplish this (e.g. it supported on-the-fly encryption, producing the exact style of difficult data) will cause some portion of less profitable users to migrate elsewhere. This may or may not be true, but it's certainly an intriguing point. Some companies like Google allow easy registration of OAuth2 clients, so any rclone user can make their own API keys at console.developers.google.com and feed those to rclone, instead of having to use the built-in credential. Rclone's documentation refers to this ability, but positions it as a performance improvement for advanced users to sidestep a shared rclone-wide quota, rather than for any other purpose. However, it, unlike many, many other applications, at least lets you plug in your own credential -- this shouldn't be an exotic flow, but rather the normal way to grant access to a third-party application, instead of distributing a hardcoded secret. Sadly, the app developer community has been slow to realize and demand this, and providers have been slow to implement it. Google lets you register as many applications as you want; GitHub even lets you make your own OAuth2 scopes. But there's few others who come close. [1] https://news.ycombinator.com/item?id=14380106 https://news.ycombinator.com/item?id=14380106

eridius 9y ago
> Some companies like Google allow easy registration of OAuth2 clients, so any rclone user can make their own API keys at console.developers.google.com and feed those to rclone, instead of having to use the built-in credential How does making new Google API keys help with an Amazon service?

jedberg 9y ago

This is a perfect example of the tragedy of the commons. No one person feels the impact of abuse, but ends up ruining it for everyone. Clearly they didn't intend for this service to be a general backup system.

SeanDav 9y ago
This is a perfect example of why companies should never offer unlimited services, if they don't expect users to actually take them up on the offer... While I can understand and agree that fair-play should be a part of the service, if the offer on hand is Unlimited, then it is not abuse to upload tera-bytes, peta-bytes, exa-bytes or even zetta-bytes. Amazon is hardly a neophyte as far as understanding big data, they are very well placed to know that offering unlimited storage could actually attract really big chunks of data.
- detaro 9y ago
  They also aren't a new to reserve to right to cut users off and knowing that the wider world doesn't care much if they do that to "extreme cases". And on the other side, many of the more extreme users of drive probably aren't new to storing a lot of data and knew that Amazons offer had to be unsustainable when used to much and thus would disappear in some way. I'm not particular sympathetic to either sides messaging here. It's sad that rclone got caught in the middle of it.
  - harshreality 9y ago
    100% Amazon's fault. They could have simply said something like, "We reserve the right to cap amazon cloud drive usage at <x> TB (currently, but increased occasionally)." But they don't want to do that, apparently. Why? I don't know, but it looks like they're worried some of their customers (the same ones they don't like because of excessive usage) will go over to B2 or whatever other cheap services are available, for the peace of mind of not running into any caps even if the service costs more at 2TB or 5TB worth of storage. "Unlimited" storage companies, that nevertheless discriminate against very high usage, want to have their cake and eat it too. Either state a limit and accept the risk that some of your customers decide to go somewhere else, or don't limit and don't discriminate against them. Those are the reasonable choices. Other choices are not reasonable.

markwillis82 9y ago

Notice our backups failing over the weekend. Set up a new account with backblaze and re-ran the backups. It sounded to good to be true, fortunately we were still on the trial.

rsync 9y ago
Backblaze is also unlimited, flat rate storage and is therefore also too good to be true. "paying a flat rate for unlimited storage, or transfer, pits you against your provider in an antagonistic relationship. This is not the kind of relationship you want to have with someone providing critical functions."[1] [1] http://blog.kozubik.com/john_kozubik/2009/11/flat-rate-storage-services-vs-rsyncnet.html http://blog.kozubik.com/john_kozubik/2009/11/flat-rate-stora...