4 ms·

GCP just cost me over $1k because of a small OSS bug

26 points by warent 5y ago

phendrenad2 5y ago
It's really annoying how user-hostile the cloud providers are. Why do cloud providers not by-default alert you when your bill increases dramatically? Why is it your personal responsibility to manually configure an obtuse billing alert system?
- nhinck 5y ago
  I don't think this is unique to cloud providers, can you name any utility provider that doesn't do this?
  - fckthisguy 5y ago
    The old service I have with rates is my (old) mobile subscription. They would text me when I reached certain reasonable milestones and again when I passed a user-defined milestone. My water utility will call me when my water usage spikes. That happened when I had a burst pipe but I'm not sure they're consistent with this so wouldn't count it as firmly as with the mobile carrier. Besides, not providing an alert by default is still user-hostile, even if other services don't provide them.
humanwhosits 5y ago
Why did a log line cost money? Does GCP charge for logging?
- dilyevsky 5y ago
  If you use their logging api ( stackdriver) they do. I assume it was probably automatically shipped from a gke cluster
- ren_engineer 5y ago
  >I had this running in GCP and it was logging about 500k times per minute it's in the issue. I'm not sure why you would run anything in the cloud without some sort of kill switch or major alerting. You are potentially risking bankruptcy unless you are able to get enough social media attention for them cloud provider to waive your bill. The cloud providers should be forced to add a "kill my services at this cost point" feature
  - thesandlord 5y ago
    https://cloud.google.com/billing/docs/how-to/budgets https://cloud.google.com/billing/docs/how-to/budgets You could argue that there should be some sane defaults though. GCP, AWS, etc are usually good about waving bills if it's very clearly an accident and the first time it happened, especially if you catch it early. Not always of course, but it's a good idea to create a billing support ticket if this happens. (Former GCP employee)
  - rixed 5y ago
    Or simply to offer prepaid subscriptions.
    - thesandlord 5y ago
      This won't work with most cloud platforms. What do you do when a customer goes past the prepaid amount? Stop everything? Delete everything? Charge them anyway?
      - _tom_ 5y ago
        Probably stop everything. The cloud situation is analogous to cell phones a decade ago, where you could easily run up a bill for thousands, with no way to prevent it. I can't recommend any service with open-ended pricing to any student or beginner. They risk bankruptcy.
NmAmDa 5y ago
"OpenReplay replays what users do" wow, invading users privacy sometimes comes at a high price.
- warent 5y ago
  Soapboxing your random opinion and thinly veiling it as if it's in context here is pretty tasteless. Any store you visit has cameras, websites might too. Recordings aren't necessarily associated with identity or used to track or invade privacy, it can all be anonymized and deleted.
  - archerx 5y ago
    He's not wrong though. Did you know you can be identified just by the gait of your work? If you think it can all be anonymized and then be deleted I have a few bridges to sell to you...
prirun 5y ago
The last time there was a post about this, I checked my GCP account and realized the credit card was about to expire so didn't worry about it. So I thought I'd be safe if something crazy happened. But no. Google somehow contacted my bank and got a new expiration date for my credit card, then sent me an email that they did it. How kind of them! This is the first time I've ever had my credit card expiration date extended without my permission. I mean, yeah, notifying me it's about to expire is great. But not doing it automatically. I only have a credit card on file with them so I can test HashBackup (I'm the author) to GCP. Consider it is Google after all, I'm afraid to try to remove my credit card because they might close down my gmail account too, which yeah, I am migrating off of, but don't want to lose it yet either. And if they do close my gmail account, I'm well aware of the recourse I have ... none!
- drost 5y ago
  > Google somehow contacted my bank and got a new expiration date for my credit card How is that even legal?
  - deleted 5y ago
    [deleted]

26 points by warent 5y ago

It's really annoying how user-hostile the cloud providers are. Why do cloud providers not by-default alert you when your bill increases dramatically? Why is it your personal responsibility to manually configure an obtuse billing alert system?

nhinck 5y ago
I don't think this is unique to cloud providers, can you name any utility provider that doesn't do this?
- fckthisguy 5y ago
  The old service I have with rates is my (old) mobile subscription. They would text me when I reached certain reasonable milestones and again when I passed a user-defined milestone. My water utility will call me when my water usage spikes. That happened when I had a burst pipe but I'm not sure they're consistent with this so wouldn't count it as firmly as with the mobile carrier. Besides, not providing an alert by default is still user-hostile, even if other services don't provide them.

humanwhosits 5y ago

Why did a log line cost money? Does GCP charge for logging?

dilyevsky 5y ago
If you use their logging api ( stackdriver) they do. I assume it was probably automatically shipped from a gke cluster
ren_engineer 5y ago
>I had this running in GCP and it was logging about 500k times per minute it's in the issue. I'm not sure why you would run anything in the cloud without some sort of kill switch or major alerting. You are potentially risking bankruptcy unless you are able to get enough social media attention for them cloud provider to waive your bill. The cloud providers should be forced to add a "kill my services at this cost point" feature
- thesandlord 5y ago
  https://cloud.google.com/billing/docs/how-to/budgets https://cloud.google.com/billing/docs/how-to/budgets You could argue that there should be some sane defaults though. GCP, AWS, etc are usually good about waving bills if it's very clearly an accident and the first time it happened, especially if you catch it early. Not always of course, but it's a good idea to create a billing support ticket if this happens. (Former GCP employee)
- rixed 5y ago
  Or simply to offer prepaid subscriptions.
  - thesandlord 5y ago
    This won't work with most cloud platforms. What do you do when a customer goes past the prepaid amount? Stop everything? Delete everything? Charge them anyway?
    - _tom_ 5y ago
      Probably stop everything. The cloud situation is analogous to cell phones a decade ago, where you could easily run up a bill for thousands, with no way to prevent it. I can't recommend any service with open-ended pricing to any student or beginner. They risk bankruptcy.

NmAmDa 5y ago

"OpenReplay replays what users do" wow, invading users privacy sometimes comes at a high price.

warent 5y ago
Soapboxing your random opinion and thinly veiling it as if it's in context here is pretty tasteless. Any store you visit has cameras, websites might too. Recordings aren't necessarily associated with identity or used to track or invade privacy, it can all be anonymized and deleted.
- archerx 5y ago
  He's not wrong though. Did you know you can be identified just by the gait of your work? If you think it can all be anonymized and then be deleted I have a few bridges to sell to you...

prirun 5y ago

The last time there was a post about this, I checked my GCP account and realized the credit card was about to expire so didn't worry about it. So I thought I'd be safe if something crazy happened. But no. Google somehow contacted my bank and got a new expiration date for my credit card, then sent me an email that they did it. How kind of them! This is the first time I've ever had my credit card expiration date extended without my permission. I mean, yeah, notifying me it's about to expire is great. But not doing it automatically. I only have a credit card on file with them so I can test HashBackup (I'm the author) to GCP. Consider it is Google after all, I'm afraid to try to remove my credit card because they might close down my gmail account too, which yeah, I am migrating off of, but don't want to lose it yet either. And if they do close my gmail account, I'm well aware of the recourse I have ... none!

drost 5y ago
> Google somehow contacted my bank and got a new expiration date for my credit card How is that even legal?
- deleted 5y ago
  [deleted]