5 ms·

I don't use S3, but it's kind of insane that unauthorized requests contribute to your bill... couldn't a bored/malicious actor with a botnet just spam common bu

by tazu 2y ago

I don't use S3, but it's kind of insane that unauthorized requests contribute to your bill... couldn't a bored/malicious actor with a botnet just spam common bucket names?

silisili 2y ago
This is exactly where my mind went. I assume most people, myself included, didn't realize this. This is absolutely going to turn into the economic equivalent of a ddos.
- utensil4778 2y ago
  I'm far too lazy to do the math, but I do wonder how many typical residential connections it would take to make a noticeable dent on a big company's bottom line with this method? I'd wager it's a surprisingly low number.
jsheard 2y ago
A bored or malicious actor can also just DDoS your CloudFront endpoint and run up an effectively unlimited bill on your end. That exposure is par for the course with AWS, they don't have a spending hard-stop and at this point they probably never will, you just have to hope that they have mercy and waive the bill if you get a surprise extra few zeroes on the end.
- immibis 2y ago
  and their bills cost up to 100x what they should (and then other services that run on top of AWS may inflate it 10x again)
  - deleted 2y ago
    [deleted]
- belter 2y ago
  Shield Advanced claims cost protection, although as always, the devil is in the details... - https://aws.amazon.com/shield/features/ https://aws.amazon.com/shield/features/ "AWS Shield Advanced comes with DDoS cost protection to safeguard against scaling charges resulting from DDoS-related usage spikes on protected EC2, ELB, CloudFront, Global Accelerator, and Route 53 resources. If any of these protected resources scale up in response to a DDoS attack, you can request Shield Advanced service credits through your regular AWS Support channel."
  - jsheard 2y ago
    Shield Advanced is $3000 per month with a 1 year commitment, plus extra bandwidth fees on top of your existing bandwidth fees, I somehow doubt that many of the users who would be impacted the most by an unexpectedly large bill (individuals and smaller businesses) are proactively paying that much for DDoS insurance just in case.
    - belter 2y ago
      Agree on that. Was not claiming that Shield Advanced is the solution for this nutty request Rodeo...or maybe we should call it 404 festival...
    - tedmiston 2y ago
      I think CloudWatch billing alarms [1] are the only viable "solution" for an average joe here. Of course, they do not actually stop the usage, but at least they bring automated awareness. Even with that, there's no easy way to rename an S3 bucket [2], plus doing so would commonly require updating config + re-deploying any apps using the bucket too... All of this to say, the customer level really feels like the wrong layer of abstraction to solve this problem. [1]: https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/monitor_estimated_charges_with_cloudwatch.html https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitori... [2]: https://stackoverflow.com/questions/41511034/how-to-rename-aws-s3-bucket https://stackoverflow.com/questions/41511034/how-to-rename-a...
      - deleted 2y ago
pid-1 2y ago
Press F12 to screw competition
andersa 2y ago
It's not just "kind of insane", it's completely bananas. Like what the actual fuck? We really need some kind of regulation to ban predatory cloud pricing.
mdasen 2y ago
This kinda happened with Netlify recently (https://www.reddit.com/r/webdev/comments/1b14bty/netlify_just_sent_me_a_104k_bill_for_a_simple/ https://www.reddit.com/r/webdev/comments/1b14bty/netlify_jus..., https://news.ycombinator.com/item?id=39520776 https://news.ycombinator.com/item?id=39520776). They got a bill over $100,000 via spam requests. Unauthorized requests are a weird thing. They still do cost money for a web service to handle and if you (the customer) have screwed things up that's kinda on you. I've run SaaS where a customer forgot to send their key and their software would just keep retrying requests to the point that they were sending thousands of requests per second with no key. However, it's kinda crappy when it's happening through no fault of your own. One doesn't even have to make unauthorized requests to run up a bill. One can simply make lots of spam requests to a real file in a bucket - run up the request total and bandwidth at the same time.
- margalabargala 2y ago
  > Unauthorized requests are a weird thing. They still do cost money for a web service to handle and if you (the customer) have screwed things up that's kinda on you. Who pays when the unauthorized requests contains truly random data, i.e. they point at an org and bucket that do not exist? Amazon does. Part of their cost of doing business is building the infrastructure that routes authorized requests correctly, and if a request is for invalid data, it's dropped. That should be par the course. Large numbers of unauthorized requests can be effectively considered a DDOS on the service itself. That the attacker can add a note onto their attack that says "btw bill X customer for all this" and have that honored is ridiculous.
- yencabulator 2y ago
  > They still do cost money for a web service to handle and if you (the customer) have screwed things up that's kinda on you. They cost a lot less than a multiple disk write (PUT), a disk read (GET). They cost the same to AWS whether the target bucket belongs to me, Amazon themselves, or does not exist. And in this case, the customer definitely didn't "screw things up". Charging for unauthorized requests is just weird and customer-hostile. Like, how am I supposed to prevent that? I get that a service provider might want to "punish" a customer who misconfigures their systems and DoSes themselves with e.g. no backoff on 403 error, but I find it hard to believe that that's the common case -- and how is typo in authentication supposed to be so different from typo in bucket name?
dvfjsdhgfv 2y ago
That's why, even though during the day I work on AWS for people who pay me to do so, in the evening I work on my startups hosted on Hetzner where I know in advance what my bill will be not only this month but also this year.
- xandrius 2y ago
  Same here, it might be slightly less convenient but boy how well I sleep at night.
belter 2y ago
First of all, I agree this is all bananas:-) but it seems is the same with GCP and Azure. For Backblaze B2 it seems its unclear in the documentation. Willing to be corrected on this....
- tstrimple 2y ago
  I'm less familiar with AWS. But on Azure, you can have private storage accounts which aren't internet routable so they wouldn't even be able to hit the endpoints to rack up charges. It's a bit ridiculous that you'd have to go that far to avoid being charged for unauthorized access, but I'd be surprised if AWS and GCP didn't have similar capabilities.
- nicolas_17 2y ago
  I don't know if Backblaze charges for failed requests, but it lets you set spending limits, so the worst case is that the attacker drains the daily quota that you set and later requests fail (DoS), not that you get a surprise $1000 bill.
- gsf 2y ago
  Not in GCP if https://cloud.google.com/storage/pricing https://cloud.google.com/storage/pricing is to be believed: > Generally, you are not charged for operations that return 307, 4xx, or 5xx responses. The exception is 404 responses returned by buckets with Website Configuration enabled and the NotFoundPage property set to a public object in that bucket.
  - yencabulator 2y ago
    Heh, charging for 5xx would be even more insane than charging for 403. "Our system is not working correctly at this moment. That'll be $0.25."
Havoc 2y ago
>kind of insane that unauthorized requests contribute to your bill Abandoned GCP API gateway because of this...best as I can tell even with a rate limit the calls for the gateway get charged. I guess if it helps if the thing behind it is computationally expensive but still feels a bit silly
INTPenis 2y ago
That's exactly it. I just spent the morning cleaning up my buckets. I had two old buckets left over from when I was using Elasticbeanstalk 5 years ago, and they had predictable names with my account ID in them. And as we heard in February, account IDs are easily leaked and not considered private. Besides that AWS themselves recommend you to use your domain name as bucket name when using S3 static site hosting. Thankfully I've been using Cloudfront infront of randomized bucket names for a few years already. So now I'm forced to increase the random string I use in all my bucket names from 8 to 16 characters. Every bucket is now a password for a malicious actor who wants to attack your finances. Obviously all accounts should have budget alerts, but there is no budget ceiling feature. Technically AWS does offer you all the tools to create a budget ceiling feature using budget alerts and lambdas that will shutdown the affected service. I just wish someone would do it because I don't have the time.
- busymom0 2y ago
  Reminder that if you are using pre-signed urls, then that url also exposes the bucket name. So, using hard to guess bucket names isn't a solution for many people.
- aveao 2y ago
  You cannot shut down an S3 bucket with a lambda tho, the mentioned issue above is only solvable by deleting the bucket.

by tazu 2y ago

I don't use S3, but it's kind of insane that unauthorized requests contribute to your bill... couldn't a bored/malicious actor with a botnet just spam common bucket names?

silisili 2y ago

This is exactly where my mind went. I assume most people, myself included, didn't realize this. This is absolutely going to turn into the economic equivalent of a ddos.

utensil4778 2y ago
I'm far too lazy to do the math, but I do wonder how many typical residential connections it would take to make a noticeable dent on a big company's bottom line with this method? I'd wager it's a surprisingly low number.

jsheard 2y ago

A bored or malicious actor can also just DDoS your CloudFront endpoint and run up an effectively unlimited bill on your end. That exposure is par for the course with AWS, they don't have a spending hard-stop and at this point they probably never will, you just have to hope that they have mercy and waive the bill if you get a surprise extra few zeroes on the end.

immibis 2y ago
and their bills cost up to 100x what they should (and then other services that run on top of AWS may inflate it 10x again)
- deleted 2y ago
  [deleted]
belter 2y ago
Shield Advanced claims cost protection, although as always, the devil is in the details... - https://aws.amazon.com/shield/features/ https://aws.amazon.com/shield/features/ "AWS Shield Advanced comes with DDoS cost protection to safeguard against scaling charges resulting from DDoS-related usage spikes on protected EC2, ELB, CloudFront, Global Accelerator, and Route 53 resources. If any of these protected resources scale up in response to a DDoS attack, you can request Shield Advanced service credits through your regular AWS Support channel."
- jsheard 2y ago
  Shield Advanced is $3000 per month with a 1 year commitment, plus extra bandwidth fees on top of your existing bandwidth fees, I somehow doubt that many of the users who would be impacted the most by an unexpectedly large bill (individuals and smaller businesses) are proactively paying that much for DDoS insurance just in case.
  - belter 2y ago
    Agree on that. Was not claiming that Shield Advanced is the solution for this nutty request Rodeo...or maybe we should call it 404 festival...
  - tedmiston 2y ago
    I think CloudWatch billing alarms [1] are the only viable "solution" for an average joe here. Of course, they do not actually stop the usage, but at least they bring automated awareness. Even with that, there's no easy way to rename an S3 bucket [2], plus doing so would commonly require updating config + re-deploying any apps using the bucket too... All of this to say, the customer level really feels like the wrong layer of abstraction to solve this problem. [1]: https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/monitor_estimated_charges_with_cloudwatch.html https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitori... [2]: https://stackoverflow.com/questions/41511034/how-to-rename-aws-s3-bucket https://stackoverflow.com/questions/41511034/how-to-rename-a...
    - deleted 2y ago

pid-1 2y ago

Press F12 to screw competition

andersa 2y ago

It's not just "kind of insane", it's completely bananas. Like what the actual fuck? We really need some kind of regulation to ban predatory cloud pricing.

mdasen 2y ago

This kinda happened with Netlify recently (https://www.reddit.com/r/webdev/comments/1b14bty/netlify_just_sent_me_a_104k_bill_for_a_simple/ https://www.reddit.com/r/webdev/comments/1b14bty/netlify_jus..., https://news.ycombinator.com/item?id=39520776 https://news.ycombinator.com/item?id=39520776). They got a bill over $100,000 via spam requests. Unauthorized requests are a weird thing. They still do cost money for a web service to handle and if you (the customer) have screwed things up that's kinda on you. I've run SaaS where a customer forgot to send their key and their software would just keep retrying requests to the point that they were sending thousands of requests per second with no key. However, it's kinda crappy when it's happening through no fault of your own. One doesn't even have to make unauthorized requests to run up a bill. One can simply make lots of spam requests to a real file in a bucket - run up the request total and bandwidth at the same time.

margalabargala 2y ago
> Unauthorized requests are a weird thing. They still do cost money for a web service to handle and if you (the customer) have screwed things up that's kinda on you. Who pays when the unauthorized requests contains truly random data, i.e. they point at an org and bucket that do not exist? Amazon does. Part of their cost of doing business is building the infrastructure that routes authorized requests correctly, and if a request is for invalid data, it's dropped. That should be par the course. Large numbers of unauthorized requests can be effectively considered a DDOS on the service itself. That the attacker can add a note onto their attack that says "btw bill X customer for all this" and have that honored is ridiculous.
yencabulator 2y ago
> They still do cost money for a web service to handle and if you (the customer) have screwed things up that's kinda on you. They cost a lot less than a multiple disk write (PUT), a disk read (GET). They cost the same to AWS whether the target bucket belongs to me, Amazon themselves, or does not exist. And in this case, the customer definitely didn't "screw things up". Charging for unauthorized requests is just weird and customer-hostile. Like, how am I supposed to prevent that? I get that a service provider might want to "punish" a customer who misconfigures their systems and DoSes themselves with e.g. no backoff on 403 error, but I find it hard to believe that that's the common case -- and how is typo in authentication supposed to be so different from typo in bucket name?

dvfjsdhgfv 2y ago

That's why, even though during the day I work on AWS for people who pay me to do so, in the evening I work on my startups hosted on Hetzner where I know in advance what my bill will be not only this month but also this year.

xandrius 2y ago
Same here, it might be slightly less convenient but boy how well I sleep at night.

belter 2y ago

First of all, I agree this is all bananas:-) but it seems is the same with GCP and Azure. For Backblaze B2 it seems its unclear in the documentation. Willing to be corrected on this....

tstrimple 2y ago
I'm less familiar with AWS. But on Azure, you can have private storage accounts which aren't internet routable so they wouldn't even be able to hit the endpoints to rack up charges. It's a bit ridiculous that you'd have to go that far to avoid being charged for unauthorized access, but I'd be surprised if AWS and GCP didn't have similar capabilities.
nicolas_17 2y ago
I don't know if Backblaze charges for failed requests, but it lets you set spending limits, so the worst case is that the attacker drains the daily quota that you set and later requests fail (DoS), not that you get a surprise $1000 bill.
gsf 2y ago
Not in GCP if https://cloud.google.com/storage/pricing https://cloud.google.com/storage/pricing is to be believed: > Generally, you are not charged for operations that return 307, 4xx, or 5xx responses. The exception is 404 responses returned by buckets with Website Configuration enabled and the NotFoundPage property set to a public object in that bucket.
- yencabulator 2y ago
  Heh, charging for 5xx would be even more insane than charging for 403. "Our system is not working correctly at this moment. That'll be $0.25."

Havoc 2y ago

>kind of insane that unauthorized requests contribute to your bill Abandoned GCP API gateway because of this...best as I can tell even with a rate limit the calls for the gateway get charged. I guess if it helps if the thing behind it is computationally expensive but still feels a bit silly

INTPenis 2y ago

That's exactly it. I just spent the morning cleaning up my buckets. I had two old buckets left over from when I was using Elasticbeanstalk 5 years ago, and they had predictable names with my account ID in them. And as we heard in February, account IDs are easily leaked and not considered private. Besides that AWS themselves recommend you to use your domain name as bucket name when using S3 static site hosting. Thankfully I've been using Cloudfront infront of randomized bucket names for a few years already. So now I'm forced to increase the random string I use in all my bucket names from 8 to 16 characters. Every bucket is now a password for a malicious actor who wants to attack your finances. Obviously all accounts should have budget alerts, but there is no budget ceiling feature. Technically AWS does offer you all the tools to create a budget ceiling feature using budget alerts and lambdas that will shutdown the affected service. I just wish someone would do it because I don't have the time.

busymom0 2y ago
Reminder that if you are using pre-signed urls, then that url also exposes the bucket name. So, using hard to guess bucket names isn't a solution for many people.
aveao 2y ago
You cannot shut down an S3 bucket with a lambda tho, the mentioned issue above is only solvable by deleting the bucket.