5 ms·

As a former victim to the same issue as OP, I am furious every time I see a Googler promote that as a solution. In our case, we racked up a $10000 bill on BigQ

by ggthrowaway2020 6y ago

As a former victim to the same issue as OP, I am furious every time I see a Googler promote that as a solution. In our case, we racked up a $10000 bill on BigQuery in ~6 hours, when a job was failing and auto-retrying. We had set up every alert correctly and our reaction time was about 5 minutes (about $100 of usage, no big deal). So how did we get a $5000 bill? Google's alert was 6 hours late (according to them, this was root-caused to us, because we were submitting jobs continuously). They pointed to their TOS and said they don't guarantee on-time delivery of the alert. I had to write up a blog post with fancy graphs and prepare it for social media before they finally agreed to eat the bill.

FeistySkink 6y ago
Is there a public postmortem anywhere? Your message points to 'no', but just in case.
dathinab 6y ago
(By now) GCP documentation says billing alerts can be late by days. Yes days not just hours. Totally crazy. EDIT: Link (see blue info box a a bit below the anchor on which the page is opened): https://cloud.google.com/billing/docs/how-to/notify#cap_disable_billing_to_stop_usage https://cloud.google.com/billing/docs/how-to/notify#cap_disa...
- donmcronald 6y ago
  > Recommendation: If you have a hard funds limit, set your maximum budget below your available funds to account for billing delays. But if 1,000,000% lower doesn't work ($7 vs $70k) then...?
sandGorgon 6y ago
P.S. so im not a googler. you misunderstand the intent of this - you basically set this. even if it fails (because messages are delayed), google will refund. This has happened to us before - they do a refund - since you had set the limits correctly. In general, they are not super assholes. I actually dont know a case, where they have refused to refund. AWS is better here - since GCP doesnt have a support dashboard. So the "chasing them" experience is much worse.

FeistySkink 6y ago

Is there a public postmortem anywhere? Your message points to 'no', but just in case.

dathinab 6y ago

(By now) GCP documentation says billing alerts can be late by days. Yes days not just hours. Totally crazy. EDIT: Link (see blue info box a a bit below the anchor on which the page is opened): https://cloud.google.com/billing/docs/how-to/notify#cap_disable_billing_to_stop_usage https://cloud.google.com/billing/docs/how-to/notify#cap_disa...

donmcronald 6y ago
> Recommendation: If you have a hard funds limit, set your maximum budget below your available funds to account for billing delays. But if 1,000,000% lower doesn't work ($7 vs $70k) then...?

sandGorgon 6y ago

P.S. so im not a googler. you misunderstand the intent of this - you basically set this. even if it fails (because messages are delayed), google will refund. This has happened to us before - they do a refund - since you had set the limits correctly. In general, they are not super assholes. I actually dont know a case, where they have refused to refund. AWS is better here - since GCP doesnt have a support dashboard. So the "chasing them" experience is much worse.