6 ms·

The fact that AWS has no way to limit billing seems insane to me. Your only recourse for an accidental (or malicious) overcharge is beg customer support. It's a

The fact that AWS has no way to limit billing seems insane to me. Your only recourse for an accidental (or malicious) overcharge is beg customer support. It's an incredible liability.

thejosh 5y ago
If you were "hacked" (ie password reuse), the attacker can just relimit this.
- notreallyserio 5y ago
  AWS could simply require the user verify they have access to the card or other payment method on file.
  - oneplane 5y ago
    That is super impractical when you run 200 AWS accounts in production.
    - notreallyserio 5y ago
      Does AWS require distinct billing accounts for each project? I use GCP, I can attach the same account to multiple projects. (It also lacks this basic billing limit feature, unfortunately.)
      - oneplane 5y ago
        No, that's why it would be silly to have to change it for every account. (and also problematic for a team with a bunch of child accounts to then have to gain access to the org's card or something like that to change settings for their childs)
        notreallyserio 5y ago
        I don't follow. But they could make the quota feature a toggle (that has the same mechanic to unlock). Easy, easy enough for a junior dev. And I'm sure folks spending more than a few minutes chatting about it could come up with an even better solution.
    - Retric 5y ago
      The option to set a limit doesn’t require you to set one. Assure lets users have a limit without any significant issues.
- jetrink 5y ago
  There's still a lot they could do around that though, e.g. requiring that billing details be re-entered upon change or sending an email alerting the user with instructions to secure the account if it was not authorized.
- stephbu 5y ago
  Just as likely as basic credential compromise is lateral attacks on compute resources from vulnerabilities such as Log4J. Enabling MFA, restricting intra/inter-VPC access, removing hard-coding credentials from configuration files/source etc., switching to SSO/removing user accounts with passwords, creating and applying restricted IAM roles, and applying those reduced privileges to EC2/ECS/EKS instances are all things that and should be done as soon as possible. (Non-exhaustive, but illustrative list)
poxrud 5y ago
That is not true, you can setup a budget, and billing alerts.
- specialp 5y ago
  AFAIK setting a budget or alerts does not say "Not authorized to spend more than X amount, deny any spending over this limit". It will alert you, yes. (Also, not on by default)
- simcop2387 5y ago
  This does not stop the high bills from your account being hacked. Just reduces the amount before you notice. spinngin up gpu and ec2 resources until limits are hit in how many are created is an operation that takes seconds.
  - poxrud 5y ago
    Can be stopped with Budget Actions.
- MuffinFlavored 5y ago
  Can you set a "hard" budget? Like, never let me get more than $20/mo in services from AWS and after that just disable my account?
  - notreallyserio 5y ago
    No, that's not possible.
- poxrud 5y ago
  You can execute Budget Actions to stop services and revoke permissions. It’s not simple but it’s possible.
- DanHulton 5y ago
  And these can then be turned off by the hacker.
YPPH 5y ago
They should either add technical measures to prevent overcharging on these types of accounts, or amend the terms of service to explicitly say that the account owner is not liable for charges incurred by an unauthorised user who obtains user credentials. I find it hard to fathom that a company which sells machine learning as a service cannot detect that an account sitting dormant for months, which suddenly spins up several large VMs, has been compromised. It's unsatisfactory that the only recourse is hoping for a favourable exercise of discretion on Amazon's part to waive the charges.
- toomanydoubts 5y ago
  Jeff Bezos did not become a billionaire by being user-friendly. Guess anyone trying to cancel their Amazon Prime subscription would agree.
  - gruez 5y ago
    >Guess anyone trying to cancel their Amazon Prime subscription would agree. The unsubscribe button was easy enough to find, but had to maybe go through 2 nag screens before letting me cancel. I guess it's not pro-user, but it's also not exactly a roadblock to me canceling.
    - toomanydoubts 5y ago
      Well, it's not a roadblock but it's a dark-pattern and user hostile. Just like the lack of spending limits on AWS. Compare that to the experience of unsubscribing from netflix, for instance.
      - gruez 5y ago
        > Well, it's not a roadblock but it's a dark-pattern and user hostile. Just like the lack of spending limits on AWS. Yeah I explicitly say it wasn't pro user. I was more disputing the "Jeff Bezos did not become a billionaire" (presumably by making it hard to cancel prime) part.
karmasimida 5y ago
Limiting billing is ambiguous. The computation spent is spent, unless AWS can predict how much you are going to cost by even initiating certain operation, which is not going to be straightforward for all the services they provided.
- halpert 5y ago
  They bill by the request or minute for most of the services. Billing for storage is a little more complicated, but many services could easily be limited.
- gruez 5y ago
  azure seems to manage that just fine with their free azure credit that they give to MSDN subscribers ($150/month spending limit, refilled every month, no chance of going over).
- derefr 5y ago
  In theory, all IaaS services could be built like an Ethereum node: 1. fill an account with abstract "multi-resource usage credits" (or gas); 2. have each API call specify how many credits the caller is willing to spend from their account on the operation (a gas limit); 3. operations in progress are tracked using real-time resource accounting (= distributed tracing, but cheap-as-possible at the expense of granularity); 4. any operation that goes over-budget is cancelled (even if it completed successfully, the user doesn't get the benefit of that completion), with the user being charged exactly the limit they specified. (Yes, that means that the IaaS eats the overhead costs of any additional processing they did before noticing and stopping the workload going beyond its limit. This incentivizes the IaaS to optimize their scheduling, at all levels of the stack, to minimize the amount of "overhang" work it allows.) And yes, to be clear, this would "infect" every layer of every service. If the IaaS provided a DBaaS, the DB's query planning and execution layers would need to know about these limits and accounting mechanisms. Etc. I'm not saying it's practical. But it's definitely possible, with no foreknowledge of cost.
- heavyset_go 5y ago
  Somehow Oracle of all companies is able to get this right with their free tiers for their cloud services.
heavyset_go 5y ago
This is the reason I refuse to use AWS for anything that I work on in my own time. I don't want to wake up to a $20k bill because someone thought it would be funny to DDoS my weekend project.

The fact that AWS has no way to limit billing seems insane to me. Your only recourse for an accidental (or malicious) overcharge is beg customer support. It's an incredible liability.

thejosh 5y ago

If you were "hacked" (ie password reuse), the attacker can just relimit this.

notreallyserio 5y ago
AWS could simply require the user verify they have access to the card or other payment method on file.
- oneplane 5y ago
  That is super impractical when you run 200 AWS accounts in production.
  - notreallyserio 5y ago
    Does AWS require distinct billing accounts for each project? I use GCP, I can attach the same account to multiple projects. (It also lacks this basic billing limit feature, unfortunately.)
    - oneplane 5y ago
      No, that's why it would be silly to have to change it for every account. (and also problematic for a team with a bunch of child accounts to then have to gain access to the org's card or something like that to change settings for their childs)
      - notreallyserio 5y ago
        I don't follow. But they could make the quota feature a toggle (that has the same mechanic to unlock). Easy, easy enough for a junior dev. And I'm sure folks spending more than a few minutes chatting about it could come up with an even better solution.
  - Retric 5y ago
    The option to set a limit doesn’t require you to set one. Assure lets users have a limit without any significant issues.
jetrink 5y ago
There's still a lot they could do around that though, e.g. requiring that billing details be re-entered upon change or sending an email alerting the user with instructions to secure the account if it was not authorized.
stephbu 5y ago
Just as likely as basic credential compromise is lateral attacks on compute resources from vulnerabilities such as Log4J. Enabling MFA, restricting intra/inter-VPC access, removing hard-coding credentials from configuration files/source etc., switching to SSO/removing user accounts with passwords, creating and applying restricted IAM roles, and applying those reduced privileges to EC2/ECS/EKS instances are all things that and should be done as soon as possible. (Non-exhaustive, but illustrative list)

poxrud 5y ago

That is not true, you can setup a budget, and billing alerts.

specialp 5y ago
AFAIK setting a budget or alerts does not say "Not authorized to spend more than X amount, deny any spending over this limit". It will alert you, yes. (Also, not on by default)
simcop2387 5y ago
This does not stop the high bills from your account being hacked. Just reduces the amount before you notice. spinngin up gpu and ec2 resources until limits are hit in how many are created is an operation that takes seconds.
- poxrud 5y ago
  Can be stopped with Budget Actions.
MuffinFlavored 5y ago
Can you set a "hard" budget? Like, never let me get more than $20/mo in services from AWS and after that just disable my account?
- notreallyserio 5y ago
  No, that's not possible.
poxrud 5y ago
You can execute Budget Actions to stop services and revoke permissions. It’s not simple but it’s possible.
DanHulton 5y ago
And these can then be turned off by the hacker.

YPPH 5y ago

They should either add technical measures to prevent overcharging on these types of accounts, or amend the terms of service to explicitly say that the account owner is not liable for charges incurred by an unauthorised user who obtains user credentials. I find it hard to fathom that a company which sells machine learning as a service cannot detect that an account sitting dormant for months, which suddenly spins up several large VMs, has been compromised. It's unsatisfactory that the only recourse is hoping for a favourable exercise of discretion on Amazon's part to waive the charges.

toomanydoubts 5y ago
Jeff Bezos did not become a billionaire by being user-friendly. Guess anyone trying to cancel their Amazon Prime subscription would agree.
- gruez 5y ago
  >Guess anyone trying to cancel their Amazon Prime subscription would agree. The unsubscribe button was easy enough to find, but had to maybe go through 2 nag screens before letting me cancel. I guess it's not pro-user, but it's also not exactly a roadblock to me canceling.
  - toomanydoubts 5y ago
    Well, it's not a roadblock but it's a dark-pattern and user hostile. Just like the lack of spending limits on AWS. Compare that to the experience of unsubscribing from netflix, for instance.
    - gruez 5y ago
      > Well, it's not a roadblock but it's a dark-pattern and user hostile. Just like the lack of spending limits on AWS. Yeah I explicitly say it wasn't pro user. I was more disputing the "Jeff Bezos did not become a billionaire" (presumably by making it hard to cancel prime) part.

karmasimida 5y ago

Limiting billing is ambiguous. The computation spent is spent, unless AWS can predict how much you are going to cost by even initiating certain operation, which is not going to be straightforward for all the services they provided.

halpert 5y ago
They bill by the request or minute for most of the services. Billing for storage is a little more complicated, but many services could easily be limited.
gruez 5y ago
azure seems to manage that just fine with their free azure credit that they give to MSDN subscribers ($150/month spending limit, refilled every month, no chance of going over).
derefr 5y ago
In theory, all IaaS services could be built like an Ethereum node: 1. fill an account with abstract "multi-resource usage credits" (or gas); 2. have each API call specify how many credits the caller is willing to spend from their account on the operation (a gas limit); 3. operations in progress are tracked using real-time resource accounting (= distributed tracing, but cheap-as-possible at the expense of granularity); 4. any operation that goes over-budget is cancelled (even if it completed successfully, the user doesn't get the benefit of that completion), with the user being charged exactly the limit they specified. (Yes, that means that the IaaS eats the overhead costs of any additional processing they did before noticing and stopping the workload going beyond its limit. This incentivizes the IaaS to optimize their scheduling, at all levels of the stack, to minimize the amount of "overhang" work it allows.) And yes, to be clear, this would "infect" every layer of every service. If the IaaS provided a DBaaS, the DB's query planning and execution layers would need to know about these limits and accounting mechanisms. Etc. I'm not saying it's practical. But it's definitely possible, with no foreknowledge of cost.
heavyset_go 5y ago
Somehow Oracle of all companies is able to get this right with their free tiers for their cloud services.

heavyset_go 5y ago

This is the reason I refuse to use AWS for anything that I work on in my own time. I don't want to wake up to a $20k bill because someone thought it would be funny to DDoS my weekend project.