4 ms·

Got an unexpected $85K bill from Google Cloud

16 points by jiahengz 2y ago

As a small startup, we’re facing a major billing issue with u/GoogleCloud. An unintended error in our Translation API integration caused our bill to skyrocket to over $85K over a short period—five times our expected usage. This massive bill is something we simply cannot afford. Despite submitting support tickets weeks ago, reaching out to our account manager, and repeatedly pinging customer service (who keep telling us it will be resolved in 3-5 business days), we have yet to receive any response from the actual billing team. Because of the waiting, our account is now at risk of suspension due to overdue payment. Any support/tips on how we can accelerate the process would be greatly appreciated!

deleted 2y ago
[deleted]
dezb 2y ago
do what you're doing now, GO PUBLIC about it, do blogs, videos, youtube, twitter, linked in content / posts, just put this out far and wide so Google quickly realise you are about to cost them million in lost business compared to your small mistake clocking up a big bill they can afford to forgive most of.. name and shame them massively.. they will eventually notice..
- cetinsert 2y ago
  'shame' them? Google is not in the wrong here.
  - benterix 2y ago
    We are having this discussion continually since 2006. It was Amazon who, despite repeated promises that "they know it is a crucial issue for their customers and that they are working on this", ultimately refused to implement the hard cap on spending. Instead, they added a million features including alarms that basically warn you once it might be far too late, especially if you are not checking your mail every minute in the night. The other big public cloud providers followed their suit. So the unwritten ritual is when you make a mistake that makes you bankrupt for life is to beg, pray, and hope for their mercy.
invalidname 2y ago
Had a similar issue with app engine decades ago. Burned me on cloud/PaaS completely, the fact that something like that is possible is a huge vulnerability. But Google is particularly problematic in this regard. We were a gold support level customer and they just blew us off. They are an advertising company and nothing else. Everything else they do is a "hobby".
- OutOfHere 2y ago
  What do you use now? I'm guessing the pool is limited to: Linux virtual machines, S3-compatible object storage, and variants of Kubernetes.
  - invalidname 2y ago
    Since we used Java it was easy to pick Spring Boot and a VPS. Cheaper and amazingly much faster.
benterix 2y ago
This is a warning for all those who fell for the hype and are not using flat-rate services like Hetzner Cloud. Seriously, if you are not big and cannot afford losing tons of money by simple human mistakes, don't use a billing system that doesn't allow you to set up a hard cap on spending, period.
- znpy 2y ago
  > Seriously, if you are not big and cannot afford losing tons of money I would also set the threshold at "if you cannot afford premium support and a TAM allocated to you" A TAM being a Technical Account Manager, some representative from AWS who is assigned to you who should become your point of contact with AWS.
  - jiahengz 2y ago
    We do have an account manager, but they are pretty unresponsive and reply to emails very slowly. It would take a week for a round of messages.
    - znpy 2y ago
      you should escalate this to their manager i guess. you're probably paying a chunk of their salary. but then again, google is not really well known for their support experience.
      - jiahengz 2y ago
        How can I get their manager's contact? By asking a friend at Google?
cleepdar 2y ago
Had a similar issue with GCP (for a much larger amount) two years ago. Mistakes happen. In the end it took 3-4 months before the credit showed up in our billing account. In my experience GCP does not want to bankrupt customers for some out of control API calls. Keeping paying your invoices while excluding the huge amount you cannot afford. I think your fears of account suspension are overblown as long as you have an active billing dispute and are otherwise current in your billing account. Also you have to do your part to avoid repeating this mistake. In the absence of better cost-based controls -- make sure your quotas are brought _way_ down to control usage. Also add alerting including an escalation to something that will wake you up on nights/weekends like PagerDuty.
- jiahengz 2y ago
  Thanks for your info! Did you pay any bills during your 3-4 months dispute period? And what percentage of the credit did they provided you?
  - cleepdar 2y ago
    IIRC we paid 100% of our bills during the dispute period -- after subtracting the mistake over-spend. In the end we got 100% credit (forgiveness) You ultimately need to sell the vision to GCP that you are worth more to them alive than dead. I would include in your pitch the 37 specific measures you have taken so this doesn't happen again. And if another service goes wild you will catch it within an hour or two.
    - jiahengz 2y ago
      Now the problem I am facing is not really about pitching, which I could do really well, but about their unresponsiveness. I can not really find someone who's directly in charge of this...
lee101k 2y ago
[dead]

16 points by jiahengz 2y ago

deleted 2y ago

[deleted]

dezb 2y ago

do what you're doing now, GO PUBLIC about it, do blogs, videos, youtube, twitter, linked in content / posts, just put this out far and wide so Google quickly realise you are about to cost them million in lost business compared to your small mistake clocking up a big bill they can afford to forgive most of.. name and shame them massively.. they will eventually notice..

cetinsert 2y ago
'shame' them? Google is not in the wrong here.
- benterix 2y ago
  We are having this discussion continually since 2006. It was Amazon who, despite repeated promises that "they know it is a crucial issue for their customers and that they are working on this", ultimately refused to implement the hard cap on spending. Instead, they added a million features including alarms that basically warn you once it might be far too late, especially if you are not checking your mail every minute in the night. The other big public cloud providers followed their suit. So the unwritten ritual is when you make a mistake that makes you bankrupt for life is to beg, pray, and hope for their mercy.

invalidname 2y ago

Had a similar issue with app engine decades ago. Burned me on cloud/PaaS completely, the fact that something like that is possible is a huge vulnerability. But Google is particularly problematic in this regard. We were a gold support level customer and they just blew us off. They are an advertising company and nothing else. Everything else they do is a "hobby".

OutOfHere 2y ago
What do you use now? I'm guessing the pool is limited to: Linux virtual machines, S3-compatible object storage, and variants of Kubernetes.
- invalidname 2y ago
  Since we used Java it was easy to pick Spring Boot and a VPS. Cheaper and amazingly much faster.

benterix 2y ago

This is a warning for all those who fell for the hype and are not using flat-rate services like Hetzner Cloud. Seriously, if you are not big and cannot afford losing tons of money by simple human mistakes, don't use a billing system that doesn't allow you to set up a hard cap on spending, period.

znpy 2y ago
> Seriously, if you are not big and cannot afford losing tons of money I would also set the threshold at "if you cannot afford premium support and a TAM allocated to you" A TAM being a Technical Account Manager, some representative from AWS who is assigned to you who should become your point of contact with AWS.
- jiahengz 2y ago
  We do have an account manager, but they are pretty unresponsive and reply to emails very slowly. It would take a week for a round of messages.
  - znpy 2y ago
    you should escalate this to their manager i guess. you're probably paying a chunk of their salary. but then again, google is not really well known for their support experience.
    - jiahengz 2y ago
      How can I get their manager's contact? By asking a friend at Google?

cleepdar 2y ago

Had a similar issue with GCP (for a much larger amount) two years ago. Mistakes happen. In the end it took 3-4 months before the credit showed up in our billing account. In my experience GCP does not want to bankrupt customers for some out of control API calls. Keeping paying your invoices while excluding the huge amount you cannot afford. I think your fears of account suspension are overblown as long as you have an active billing dispute and are otherwise current in your billing account. Also you have to do your part to avoid repeating this mistake. In the absence of better cost-based controls -- make sure your quotas are brought _way_ down to control usage. Also add alerting including an escalation to something that will wake you up on nights/weekends like PagerDuty.

jiahengz 2y ago
Thanks for your info! Did you pay any bills during your 3-4 months dispute period? And what percentage of the credit did they provided you?
- cleepdar 2y ago
  IIRC we paid 100% of our bills during the dispute period -- after subtracting the mistake over-spend. In the end we got 100% credit (forgiveness) You ultimately need to sell the vision to GCP that you are worth more to them alive than dead. I would include in your pitch the 37 specific measures you have taken so this doesn't happen again. And if another service goes wild you will catch it within an hour or two.
  - jiahengz 2y ago
    Now the problem I am facing is not really about pitching, which I could do really well, but about their unresponsiveness. I can not really find someone who's directly in charge of this...

lee101k 2y ago

[dead]