8 ms·

As an ex-Googler working in a customer facing role in Cloud you did very well to get a $72k bill written off! It's definitely possible but requires a lot of app

As an ex-Googler working in a customer facing role in Cloud you did very well to get a $72k bill written off! It's definitely possible but requires a lot of approvals and pulling in a few favours. I went through the process to write off a ~$50k bill for one of my customers and it required action every day for 3 months of my life. Whoever helped you inside Google will have gone to a LOT of trouble, opened a bunch of tickets and attended many, many meetings to make this happen.

cogman10 6y ago
I know there's no reason for Google or AWS to do this, but man do I wish there was a way to put down a spending limit and simply disable anything that goes over that limit. It's a little bit nuts that there are no guardrails to prevent you from incurring such huge bills (especially as a solo developer that might just be trying out their services).
- fweespeech 6y ago
  Tbh, its lack is why I don't use Google or AWS for projects.
- brianwawok 6y ago
  There are guard rails in quotas. Like you can only spin up X servers without opening a ticket to ask for more. Now, think some of these quotas can still lead to some pretty crazy bills.. but that is the point of at least some of them....
  - dathinab 6y ago
    They are broken, unreliable, hard to correctly setup guard rails. I mean like the article mentioned they could have set the instances and concurrency settings to lower values. Which in this case would have worked. But finding the right settings to balance intentional auto-scaling and limiting auto-scaling to limit of how fast unexpected cost might rise is hard and prone to get wrong. Let's be honest it's in the end a very flawed workaround which maybe might help (if you know about it, and did it right).
- donavanm 6y ago
  The downside of disabling active resources is huge. It would mean a catastrophic interruption to the customers application exactly when its the most popular/active. And theres no practical way to determine whether the customer is “trying it out” or running a key part of their business on any particular resource. On the other hand retroactively forgiving the cost of unexpected/unintentional usage doesnt impact the customers users. And with billing alerts the customer is able to make the choice of whether the cost is worth it as it happens. Note: Principal at AWS. Have worked to issue many credits/bill amendments, but dont work in that area nor do I speak for AWS.
  - tpmx 6y ago
    The upside for the noob trying out/learning is huge. I'm no longer that person, but I think GCP/AWS are just being lazy about this - perhaps because they earn a lot of money from engineer mistakes. Of course it's possible to create an actual limit. There'll be some engineering cost, like 0.5%-1% extra per service? Edit: Being European I think legislation might be the fix, since both Amazon and Google have demonstrated an unwillingness to fix this issue, for a very long time.
  - Alupis 6y ago
    Why not alert thresholds, configurable by the user? Email me when we cross $X amount in one day, Text when we cross $Y, and Call when we cross $Z. Additionally, allow the user to configure a hard cut-off limit if they desire. Just provide the mechanisms and allow users to make the call. Google et al would have a much stronger leg to stand on when enforcing delinquent account collections if they provided these mechanisms and the user chose to ignore them. Additionally, Google et al should protect _themselves_ by tracking usage patterns, and reaching out to customers that grossly surpass the average billable amount - just like OP with their near $100k bill in 1 day. Zero vetting to even have a reasonable guarantee the individual or company is even capable of paying such a large bill. And then what? Sue a company that doesn't have $100k for $100k? This makes zero sense.
    - patch_cable 6y ago
      Something like this for alerts? https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/monitor_estimated_charges_with_cloudwatch.html https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitori...
- dijit 6y ago
  > but man do I wish there was a way to put down a spending limit and simply disable anything that goes over that limit. Literally did this my first week when trying out GCP for my company. It is entirely possible and documented (with code): https://cloud.google.com/billing/docs/how-to/notify#cap_disable_billing_to_stop_usage https://cloud.google.com/billing/docs/how-to/notify#cap_disa...
  - alasdair_ 6y ago
    The article says that they had a limit in place but that in practice the billing limit lags up to 24 hours behind the "real" number.
  - dathinab 6y ago
    > Note: There is a delay of up to a few days between incurring costs and receiving budget notifications. Due to usage latency from the time that a resource is used to the time that the activity is billed, you might incur additional costs for usage that hasn't arrived at the time that all services are stopped. Following the steps in this capping example is not a guarantee that you will not spend more than your budget. Recommendation: If you have a hard funds limit, set your maximum budget below your available funds to account for billing delays. (Source link in parent post, emphasis mine). In this case they had a additional cost due to delay of $72k. Which, lets be honest means this feature kinda useless for anything but the somewhat harmless case. Only by combining this with resource limits in load balancers, instance and concurrency limits and similar can the maximal worst cost be limited. But tbh. this partially cripples auto-scaling functionality and it's really hard to find a good setting which doesn't allow to much "over" cost and at the same time doesn't hinder the intended auto-scaling use-case.
    - throwclowd 6y ago
      There's something uncanny about understanding the situation enough to turn on the budget alerts, while at the same time not realizing it's not going to help in time if your system runs amok.
    - donmcronald 6y ago
- paxys 6y ago
  This is already present and very easy to set up.
- Retailer 6y ago
  If you're on App Engine, we did an article about that https://retailingplatform.com/blog/how-to-control-costs-on-a-google-app-engine-project/ https://retailingplatform.com/blog/how-to-control-costs-on-a...
- betwixthewires 6y ago
  In my opinion, and maybe I'm an absolutist about this, the fact that there aren't these guardrails is opportunistic and predatory. Agile, iterative design and testing will inevitably lead to failures, that's the whole point. Marketing a cloud service to developers who need scalable and changing access to computing during that process should take that into consideration.
  - KorematsuFred 6y ago
    I do not think the intentions here are to be opportunistic and predatory but inability to empathize with small developers. A large customer will very likely just pay off few hundred thousand dollars extra expenses. It is only individual developers who are at risk here and cloud operators do not have much interest in them.
    - betwixthewires 6y ago
      I don't know about that. Large customers can almost always find the capital to run their own infrastructure and save on cost. That's not to say that there aren't big customers of these types of services or that certain business models make using them more attractive than maintaining infrastructure yourself, but I would guess that revenue from these sorts of services are largely built on appealing to smaller customers, so their needs would be taken into consideration. Not taking potential cost overruns into consideration to me seems a bit deliberate. To me it looks very similar to the personal checking overdraft schemes banks were using up until a few years ago.
      - rightbyte 6y ago
        When I was a young and naive student I thought I could not charge my debit card under 0$. Got down to -3$ and had to pay a 40$ something fee when I already was out of money.
        m1gu3l 6y ago
        Definitely drank a few 45 dollar lattes in my day. Sucks.
- KorematsuFred 6y ago
  There are already such features but a lot of indie developers are lazy to configure their infra properly. Now, default low limit does not make sense as it will piss off large customers. I run so many websites on Google Cloud Run that sometimes I feel I might be abusing them, but I have ensured each of my site has max limit of 2 hosts.
- Aeolun 6y ago
  Yeah, had that when I just started using it and it happily kept scaling like crazy. $200 bill in one day. I never used google again.
sudcha 6y ago
OP here. Thanks for sharing! I have no idea what they did internally, but something like this was my guess. I only communicated through customer support channel and replied to emails, and shared my doc (which cited all the loopholes) with them. It took them 10-15 days to get back and make a one-time good will contribution. The contribution didn't cover logging cost, so we did pay few hundred dollars.
- salmonlogs 6y ago
  Sounds like you found an amazing support rep and made a great case for it - good job!
vincentmarle 6y ago
I went through this very scary experience recently as well (although in our case it was $17K, not $72K). One of our devs accidentally caused an infinite loop with one of the Google Maps paid APIs on our dev account and within hours both our prod and dev accounts were suspended (pro tip: don't link your prod account to the billing account of your dev account). The worst part was that after removing the suspension, our cloud functions were broken and had to be investigated and fixed by Google engineers themselves resulting in our prod app being down for 24 hours... be very careful. Luckily we were able to get $11K refunded on our card and received $6K credits after spending all night with Google support.
lathiat 6y ago
By contrast I hear stories of AWS doing this quite often for one-off mistakes (crediting thousands of dollars). It doesn't make much sense to me not to consider well-intentioned requests for this sort of thing. Especially if you consider the dollar value of all those approvals and the business you might lose to some other platform and/or hesitance people will have to use those platforms for such things in the future.
xwdv 6y ago
If I were in this situation I would probably offer 10% of the bill to the employee as a reward for their help.
- rightbyte 6y ago
  Sounds like a bribe.
  - xwdv 6y ago
    Right, better hope they help me out of the goodness of their heart instead.
- sfkdjf9j3j 6y ago
  That's too low. I usually tip my customer service reps 25% of whatever they save me.
  - xwdv 6y ago
    So zero.

cogman10 6y ago

I know there's no reason for Google or AWS to do this, but man do I wish there was a way to put down a spending limit and simply disable anything that goes over that limit. It's a little bit nuts that there are no guardrails to prevent you from incurring such huge bills (especially as a solo developer that might just be trying out their services).

fweespeech 6y ago
Tbh, its lack is why I don't use Google or AWS for projects.
brianwawok 6y ago
There are guard rails in quotas. Like you can only spin up X servers without opening a ticket to ask for more. Now, think some of these quotas can still lead to some pretty crazy bills.. but that is the point of at least some of them....
- dathinab 6y ago
  They are broken, unreliable, hard to correctly setup guard rails. I mean like the article mentioned they could have set the instances and concurrency settings to lower values. Which in this case would have worked. But finding the right settings to balance intentional auto-scaling and limiting auto-scaling to limit of how fast unexpected cost might rise is hard and prone to get wrong. Let's be honest it's in the end a very flawed workaround which maybe might help (if you know about it, and did it right).
donavanm 6y ago
The downside of disabling active resources is huge. It would mean a catastrophic interruption to the customers application exactly when its the most popular/active. And theres no practical way to determine whether the customer is “trying it out” or running a key part of their business on any particular resource. On the other hand retroactively forgiving the cost of unexpected/unintentional usage doesnt impact the customers users. And with billing alerts the customer is able to make the choice of whether the cost is worth it as it happens. Note: Principal at AWS. Have worked to issue many credits/bill amendments, but dont work in that area nor do I speak for AWS.
- tpmx 6y ago
  The upside for the noob trying out/learning is huge. I'm no longer that person, but I think GCP/AWS are just being lazy about this - perhaps because they earn a lot of money from engineer mistakes. Of course it's possible to create an actual limit. There'll be some engineering cost, like 0.5%-1% extra per service? Edit: Being European I think legislation might be the fix, since both Amazon and Google have demonstrated an unwillingness to fix this issue, for a very long time.
- Alupis 6y ago
  Why not alert thresholds, configurable by the user? Email me when we cross $X amount in one day, Text when we cross $Y, and Call when we cross $Z. Additionally, allow the user to configure a hard cut-off limit if they desire. Just provide the mechanisms and allow users to make the call. Google et al would have a much stronger leg to stand on when enforcing delinquent account collections if they provided these mechanisms and the user chose to ignore them. Additionally, Google et al should protect _themselves_ by tracking usage patterns, and reaching out to customers that grossly surpass the average billable amount - just like OP with their near $100k bill in 1 day. Zero vetting to even have a reasonable guarantee the individual or company is even capable of paying such a large bill. And then what? Sue a company that doesn't have $100k for $100k? This makes zero sense.
  - patch_cable 6y ago
    Something like this for alerts? https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/monitor_estimated_charges_with_cloudwatch.html https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitori...
dijit 6y ago
> but man do I wish there was a way to put down a spending limit and simply disable anything that goes over that limit. Literally did this my first week when trying out GCP for my company. It is entirely possible and documented (with code): https://cloud.google.com/billing/docs/how-to/notify#cap_disable_billing_to_stop_usage https://cloud.google.com/billing/docs/how-to/notify#cap_disa...
- alasdair_ 6y ago
  The article says that they had a limit in place but that in practice the billing limit lags up to 24 hours behind the "real" number.
- dathinab 6y ago
  > Note: There is a delay of up to a few days between incurring costs and receiving budget notifications. Due to usage latency from the time that a resource is used to the time that the activity is billed, you might incur additional costs for usage that hasn't arrived at the time that all services are stopped. Following the steps in this capping example is not a guarantee that you will not spend more than your budget. Recommendation: If you have a hard funds limit, set your maximum budget below your available funds to account for billing delays. (Source link in parent post, emphasis mine). In this case they had a additional cost due to delay of $72k. Which, lets be honest means this feature kinda useless for anything but the somewhat harmless case. Only by combining this with resource limits in load balancers, instance and concurrency limits and similar can the maximal worst cost be limited. But tbh. this partially cripples auto-scaling functionality and it's really hard to find a good setting which doesn't allow to much "over" cost and at the same time doesn't hinder the intended auto-scaling use-case.
  - throwclowd 6y ago
    There's something uncanny about understanding the situation enough to turn on the budget alerts, while at the same time not realizing it's not going to help in time if your system runs amok.
  - donmcronald 6y ago
paxys 6y ago
This is already present and very easy to set up.
Retailer 6y ago
If you're on App Engine, we did an article about that https://retailingplatform.com/blog/how-to-control-costs-on-a-google-app-engine-project/ https://retailingplatform.com/blog/how-to-control-costs-on-a...
betwixthewires 6y ago
In my opinion, and maybe I'm an absolutist about this, the fact that there aren't these guardrails is opportunistic and predatory. Agile, iterative design and testing will inevitably lead to failures, that's the whole point. Marketing a cloud service to developers who need scalable and changing access to computing during that process should take that into consideration.
- KorematsuFred 6y ago
  I do not think the intentions here are to be opportunistic and predatory but inability to empathize with small developers. A large customer will very likely just pay off few hundred thousand dollars extra expenses. It is only individual developers who are at risk here and cloud operators do not have much interest in them.
  - betwixthewires 6y ago
    I don't know about that. Large customers can almost always find the capital to run their own infrastructure and save on cost. That's not to say that there aren't big customers of these types of services or that certain business models make using them more attractive than maintaining infrastructure yourself, but I would guess that revenue from these sorts of services are largely built on appealing to smaller customers, so their needs would be taken into consideration. Not taking potential cost overruns into consideration to me seems a bit deliberate. To me it looks very similar to the personal checking overdraft schemes banks were using up until a few years ago.
    - rightbyte 6y ago
      When I was a young and naive student I thought I could not charge my debit card under 0$. Got down to -3$ and had to pay a 40$ something fee when I already was out of money.
      - m1gu3l 6y ago
        Definitely drank a few 45 dollar lattes in my day. Sucks.
KorematsuFred 6y ago
There are already such features but a lot of indie developers are lazy to configure their infra properly. Now, default low limit does not make sense as it will piss off large customers. I run so many websites on Google Cloud Run that sometimes I feel I might be abusing them, but I have ensured each of my site has max limit of 2 hosts.
Aeolun 6y ago
Yeah, had that when I just started using it and it happily kept scaling like crazy. $200 bill in one day. I never used google again.

sudcha 6y ago

OP here. Thanks for sharing! I have no idea what they did internally, but something like this was my guess. I only communicated through customer support channel and replied to emails, and shared my doc (which cited all the loopholes) with them. It took them 10-15 days to get back and make a one-time good will contribution. The contribution didn't cover logging cost, so we did pay few hundred dollars.

salmonlogs 6y ago
Sounds like you found an amazing support rep and made a great case for it - good job!

vincentmarle 6y ago

I went through this very scary experience recently as well (although in our case it was $17K, not $72K). One of our devs accidentally caused an infinite loop with one of the Google Maps paid APIs on our dev account and within hours both our prod and dev accounts were suspended (pro tip: don't link your prod account to the billing account of your dev account). The worst part was that after removing the suspension, our cloud functions were broken and had to be investigated and fixed by Google engineers themselves resulting in our prod app being down for 24 hours... be very careful. Luckily we were able to get $11K refunded on our card and received $6K credits after spending all night with Google support.

lathiat 6y ago

By contrast I hear stories of AWS doing this quite often for one-off mistakes (crediting thousands of dollars). It doesn't make much sense to me not to consider well-intentioned requests for this sort of thing. Especially if you consider the dollar value of all those approvals and the business you might lose to some other platform and/or hesitance people will have to use those platforms for such things in the future.

xwdv 6y ago

If I were in this situation I would probably offer 10% of the bill to the employee as a reward for their help.

rightbyte 6y ago
Sounds like a bribe.
- xwdv 6y ago
  Right, better hope they help me out of the goodness of their heart instead.
sfkdjf9j3j 6y ago
That's too low. I usually tip my customer service reps 25% of whatever they save me.
- xwdv 6y ago
  So zero.