2 ms·

That's the typical story. Something goes wrong and it costs you (typically a small company) a lot of money. At that time just nobody is looking at metrics. Even

That's the typical story. Something goes wrong and it costs you (typically a small company) a lot of money. At that time just nobody is looking at metrics. Even alarms don't help absolutely because they can also be missed. The only thing that would really help were a hard spending limit that stops all services except storage. If your site is important there will be such an amount of user feedback that it is impossible to miss it for a long time.

dspillett 5y ago
Alerts can also fail to be timely due to mail/SMS/other delivery issues, or the right people being in the middle of something else. This delay means it is still possible to rack up and unexpected cost. Or they can fail completely. And the alerts themselves cost if you want something reliable so you have to weight that against the danger. Pay as you go cloud can be a maze of costing concerns.. > The only thing that would really help were a hard spending limit that stops all services except storage. Yep. Though that is small comfort if you need to guarantee more than a couple if 9s of uptime, hopefully those with that requirement can soak up the unexpected billing blips.
alfiedotwtf 5y ago
> The only thing that would really help were a hard spending limit that stops all services except storage. Sadly, I haven't found a way to do that with AWS
- dx034 5y ago
  It's funny that even Hetzner can do that and AWS can't. Shows that there's no interest from AWS to prevent these things from happening.
  - bencollier49 5y ago
    IMO this is something which ought to be written into law. It'd be easy to implement a kill switch, and would actually encourage innovation, as people would feel more empowered to experiment with the technology.
    - wccrawford 5y ago
      Absolutely, and I would make it a bit broader: Anything that automatically charges a client a variable amount should have a maximum-spend limit that the client can set, and it should default to a reasonable number based on the client's expected usage. In fact, you could even just change that to any auto-billing service or product and the default for constant-charge services would simply be the amount of the constant charge.
      - cure 5y ago
        Yes. And the law would simply state that the customer may not be billed more than that maximum for the service period. Up to the business to restrict their use of the service, if they want to. If it's really so 'complicated' that that part can't be done, well, I'm sure it would suddenly become a business priority to cut off use when the limit is hit, if the business can't charge for it.
  - danparsonson 5y ago
    *won't
  - Nextgrid 5y ago
    To be fair, Hetzner doesn't nickel & dime you on everything. Some things are legitimately very difficult to meter in a real-time way with adequate performance. Imagine if the S3 server had to do a DB query to lookup your account balance & limit on every HTTP request. That'll completely kill performance and availability (what if the billing DB is down? Should that DB now be replicated across all regions? etc). The reason a lot of cloud alerts and metrics lag is because billing is done asynchronously by parsing logs way after the actual usage occurred. Of course, the real solution here is to just not bill for things you can't easily measure & limit especially when those things are super expensive, but cloud providers' C-suites have to eat & pay their bills somehow.
- UnFleshedOne 5y ago
  I just looked at my AWS account and there seems to be a way to set budget, attach alerts to it and attach actions to alerts. For example there is an action to stop EC2 instances. Not sure if other AWS services have something similar, but at least you can kill your instances if something weird happens. Actions weren't there last time I checked (few years ago).
  - alfiedotwtf 5y ago
    Thank you, I'll check it out
- Monotoko 5y ago
  Kill switches in lambda I believe is possible, running when the alert is triggered
  - alfiedotwtf 5y ago
    Nice, I'll have a look. Thanks!

dspillett 5y ago

Alerts can also fail to be timely due to mail/SMS/other delivery issues, or the right people being in the middle of something else. This delay means it is still possible to rack up and unexpected cost. Or they can fail completely. And the alerts themselves cost if you want something reliable so you have to weight that against the danger. Pay as you go cloud can be a maze of costing concerns.. > The only thing that would really help were a hard spending limit that stops all services except storage. Yep. Though that is small comfort if you need to guarantee more than a couple if 9s of uptime, hopefully those with that requirement can soak up the unexpected billing blips.

alfiedotwtf 5y ago

> The only thing that would really help were a hard spending limit that stops all services except storage. Sadly, I haven't found a way to do that with AWS

dx034 5y ago
It's funny that even Hetzner can do that and AWS can't. Shows that there's no interest from AWS to prevent these things from happening.
- bencollier49 5y ago
  IMO this is something which ought to be written into law. It'd be easy to implement a kill switch, and would actually encourage innovation, as people would feel more empowered to experiment with the technology.
  - wccrawford 5y ago
    Absolutely, and I would make it a bit broader: Anything that automatically charges a client a variable amount should have a maximum-spend limit that the client can set, and it should default to a reasonable number based on the client's expected usage. In fact, you could even just change that to any auto-billing service or product and the default for constant-charge services would simply be the amount of the constant charge.
    - cure 5y ago
      Yes. And the law would simply state that the customer may not be billed more than that maximum for the service period. Up to the business to restrict their use of the service, if they want to. If it's really so 'complicated' that that part can't be done, well, I'm sure it would suddenly become a business priority to cut off use when the limit is hit, if the business can't charge for it.
- danparsonson 5y ago
  *won't
- Nextgrid 5y ago
  To be fair, Hetzner doesn't nickel & dime you on everything. Some things are legitimately very difficult to meter in a real-time way with adequate performance. Imagine if the S3 server had to do a DB query to lookup your account balance & limit on every HTTP request. That'll completely kill performance and availability (what if the billing DB is down? Should that DB now be replicated across all regions? etc). The reason a lot of cloud alerts and metrics lag is because billing is done asynchronously by parsing logs way after the actual usage occurred. Of course, the real solution here is to just not bill for things you can't easily measure & limit especially when those things are super expensive, but cloud providers' C-suites have to eat & pay their bills somehow.
UnFleshedOne 5y ago
I just looked at my AWS account and there seems to be a way to set budget, attach alerts to it and attach actions to alerts. For example there is an action to stop EC2 instances. Not sure if other AWS services have something similar, but at least you can kill your instances if something weird happens. Actions weren't there last time I checked (few years ago).
- alfiedotwtf 5y ago
  Thank you, I'll check it out
Monotoko 5y ago
Kill switches in lambda I believe is possible, running when the alert is triggered
- alfiedotwtf 5y ago
  Nice, I'll have a look. Thanks!