6 ms·

Student Charged $14k on Stolen Google Cloud Credentials

71 points by nitins_jakta 7y ago

Hi, In 2017, I made a Google Cloud Account to use Google Maps API for a Computer Science student group project and put my debit card in. I naively put a $5 account notification in, thinking it was a cap. This project was defunct after 2017 and I should have just closed the Cloud account. All was fine up until January 2019 when the Google Cloud Credentials were somehow stolen and over the course of two days on Google Maps API, racked up enough API calls to generate over $14k invoice. I disabled the Google Cloud Account a day after I noticed an email from Google Cloud. Google Cloud did try to use debit card to deduct from checking account, but I don't leave thousands sitting around in it, so charge was declined. I talked to Google Cloud Billing and they have not been helpful, telling me to contact my bank. Today, I got a scary email from a collections agency demanding I login to my Google Cloud account and pay the bill! Worst part is, this API used to be free, until Google started charging exorbitant amounts for it. I know I did not make these API calls -- if you looked at the call volume history, there was nothing for well over a year, until those two days in 2019, it started going crazy (and the project is not running on any server or being used in any way). I suspect a group member might have accidentally leaked the credentials. I know AWS has waived costs[1] like this in the past, but Google is not known for customer support. I should have been more proactive in setting up a cap. Appreciate any advice or Google contacts to talk to an actual human. Should I see if Google is willing to actually verify this was unauthorized usage or just lower the bill? I'll eat a few thousand just to make this go away. To say GCP has left a sour taste in my mouth is an understatement! Thanks for reading. [1] https://dev.to/juanmanuelramallo/i-was-billed-for-14k-usd-on-amazon-web-services-17fn

foobarbazetc 7y ago
Make them prove you used it to generate the charges. Make them provide IPs etc. You need to say it was used fraudulently and you don’t agree to the charges.
- nodesocket 7y ago
  Can you get any information about the party who stole your credentials? What were the domains and urls? What is the domain whois (probably private), their IP's? Where are they hosting? This information might help you make a case of fraudent abuse of your GCP account.
  - nitins_jakta 7y ago
    Will try to ask, seems unlikely they have tools out of the box for billing support to do this. I bet they have to have an SWE grep logs. Thanks for the advice!
- nitins_jakta 7y ago
  Will do, thanks for the advice. The only thing they did was validate the API count, but I was not disputing this, only that I made the calls. The call volume itself looks suspicious. No calls for well over a year and then suddenly an incredible amount? You would think Google would have some algorithm to detect this?
  - foobarbazetc 7y ago
    Yeah exactly. But the key is not to say “lower my bill”, but rather “these charges aren’t mine”. If they send it to collections you can dispute it. A lot of laws and things around that. It sounds like it hasn’t been charged off yet though, since they want you to pay Google, not them. I wish you luck. :)
- qes 7y ago
  Google doesn't have any burden of proof here. You're responsible for your api keys and any use that occurs with them.
unknownkadath 7y ago
Before disputing the charge, be sure to back up all data and contact info from your Google accounts. Fighting charges has been known to trigger account lockouts with no appeal.
- nitins_jakta 7y ago
  This is even more terrible as an Android user. So much for "don't be evil". Thankful for the HN forum.
  - unknownkadath 7y ago
    Hope it works out with a minimum of hassle!
scarface74 7y ago
I’ve heard so many stories of something similar happening on AWS and after an email to support, all of the charges were dropped. This isn’t exactly helping Google to fight the narrative that it isn’t good with customer support and they can’t be trusted as a platform for business. So if you were a person deciding who to choose as your cloud provider, who are you going to choose? AWS - “No one ever got fired for choosing AWS” Microsoft - well known for their enterprise support and there are plenty of MS Shops out there. Or Google?
- nitins_jakta 7y ago
  I agree with you 100%. This has certainly left a sour taste in my mouth. Unfortunately Google Maps API is much better than the competition. AWS has none here. The google customer experience is just horrific.
segmondy 7y ago
While you're figuring this out, backup all your data on Google. Google is crazy and could possibly delete all your accounts and data.
- nitins_jakta 7y ago
  Backing up now, thanks. Even worse I'm an Android user. I don't have a good solution for the people that mail me every few months. Oh hey, Google might lock my account because I didn't fork over 14k in fraudulent charges, can you start emailing me from my Outlook email instead?
  - quickthrower2 7y ago
    The idea email is your own domain name. I’m yet to take my own advice though!
cjbprime 7y ago
Keep trying to talk to them and explain -- so far every instance like this I've heard about was refunded. Good luck.
londons_explore 7y ago
Google will typically waive charges in cases like this. The only time they won't is if (by looking at the logs) they decide you were probably scraping and storing all their data.
deleted 7y ago
[deleted]
applecrazy 7y ago
Did you check your Github repos and associated commit history for accidental push of secret files? There's an article on the HN front page describing secret leakage in Github repos (the most common is Google API keys, go figure)[1]. I imagine somebody out there has a bot to monitor pushes in realtime to extract secrets. You or a team member might have leaked keys in a similar manner. [1]: https://blog.acolyer.org/2019/04/08/how-bad-can-it-git-characterizing-secret-leakage-in-public-github-repositories/ https://blog.acolyer.org/2019/04/08/how-bad-can-it-git-chara...
samfisher83 7y ago
Next use a credit card. Basically thanks to credit card laws the bank will go tell google to f off and give you your money back. Debit cards don't have the same protection, but just call your bank or OCS (https://www.occ.treas.gov/ https://www.occ.treas.gov/). They have a little more bite.
- morituri 7y ago
  He was a student so he may not have had access to a credit card when he started the account. But yes, a credit card will have more protection.
- joecot 7y ago
  His debit card ended up declining it anyway, but that's not the point. The point is Google thinks he owes them thousands of dollars, and a collection agency is hounding him for the money. Disputing a charge with your credit card company doesn't change that. The credit card company might side in your favor and reverse the charges, but the original company can still pursue payment with you directly. Their claim can even show up on your credit report. I remember working with a payment processor in Europe that even automated sending a bill to collections if it was charged back by the consumer's credit card company.
  - motherofzappa 7y ago
    playing devils advocate, but taking peoples word prima facie is dangerous. Especially when it deals with financial issues. Pretty sure you can any contempt person could game the system if people found out “students” automatically get refunds for bs charges. I think AWS is different as they have their educate program and distinguish between students and free trial customers. AFAIK gcp doesn’t have an education system for students, so it’s harder to differentiate a legitimate request
    - quickthrower2 7y ago
      What will these system gamers do with computer power? Mining crypto will make them pennies, and running a business on stolen accounts isn’t sustainable.
boulos 7y ago
Sorry this happened to you! Feel free to send me your case number (email in profile), and I'll escalate it. The Support personnel have hopefully been helping out, as all Billing Issues are covered regardless of support tier. I obviously don't know the ins and outs of payment instrument refunds / do debit cards mean that you actually do have to contact your bank, but I'm sure people in Support do.
- travisglines 7y ago
  Thanks for responding to this and offering to escalate boulos. I certainly don't think a student just learning the ins and outs of a cloud provider's services should be able to spend 10k+ without warnings/thresholds that require configuration to exceed. It would be positive for platform adoption to make that process better.
  - motherofzappa 7y ago
    This is hilarious. Student doesn’t understand security in depth model, gets owned. Has a sour taste with said cloud provider. At what level do you accept responsibility for shoddy security practices. If the project was truly defunct then you should have closed the project or removed everyone’s access who isn’t project owner. Hindsight is 20/20.
    - duiker101 7y ago
      Nice victim shaming you got there. The fault of all of this is 100% on whoever stole the credentials and made those calls. OP could maybe have been more careful but that doesn't mean it's all his fault or that we should be shaming in oblivion. And Google can still try to help rather than just take advantage of the situation. Life is easier when we are not dicks to each other, a little empathy can go a long way.
      - jameskegel 7y ago
        I think a lot of the problems we face today are largely unacknowledged by those who create them. The delivery lacked any empathy but does not make it less true. Actions have consequences and ignorance of those outcomes doesn’t mean you can escape them. The world would be a little bit better of a place if we’d stop coddling those who float haphazardly through their own existence.
kkarakk 7y ago
I dunno if google lets you do this but amazon/azure will pretty reliably let you create new free tier accounts with fake emails and access them from the same IP. i just create a new debit/credit card every 6 months(it's pretty hassle free in india). i do pay for production instances, i just don't want to mess around on those production instances

71 points by nitins_jakta 7y ago

foobarbazetc 7y ago

Make them prove you used it to generate the charges. Make them provide IPs etc. You need to say it was used fraudulently and you don’t agree to the charges.

nodesocket 7y ago
Can you get any information about the party who stole your credentials? What were the domains and urls? What is the domain whois (probably private), their IP's? Where are they hosting? This information might help you make a case of fraudent abuse of your GCP account.
- nitins_jakta 7y ago
  Will try to ask, seems unlikely they have tools out of the box for billing support to do this. I bet they have to have an SWE grep logs. Thanks for the advice!
nitins_jakta 7y ago
Will do, thanks for the advice. The only thing they did was validate the API count, but I was not disputing this, only that I made the calls. The call volume itself looks suspicious. No calls for well over a year and then suddenly an incredible amount? You would think Google would have some algorithm to detect this?
- foobarbazetc 7y ago
  Yeah exactly. But the key is not to say “lower my bill”, but rather “these charges aren’t mine”. If they send it to collections you can dispute it. A lot of laws and things around that. It sounds like it hasn’t been charged off yet though, since they want you to pay Google, not them. I wish you luck. :)
qes 7y ago
Google doesn't have any burden of proof here. You're responsible for your api keys and any use that occurs with them.

unknownkadath 7y ago

Before disputing the charge, be sure to back up all data and contact info from your Google accounts. Fighting charges has been known to trigger account lockouts with no appeal.

nitins_jakta 7y ago
This is even more terrible as an Android user. So much for "don't be evil". Thankful for the HN forum.
- unknownkadath 7y ago
  Hope it works out with a minimum of hassle!

scarface74 7y ago

I’ve heard so many stories of something similar happening on AWS and after an email to support, all of the charges were dropped. This isn’t exactly helping Google to fight the narrative that it isn’t good with customer support and they can’t be trusted as a platform for business. So if you were a person deciding who to choose as your cloud provider, who are you going to choose? AWS - “No one ever got fired for choosing AWS” Microsoft - well known for their enterprise support and there are plenty of MS Shops out there. Or Google?

nitins_jakta 7y ago
I agree with you 100%. This has certainly left a sour taste in my mouth. Unfortunately Google Maps API is much better than the competition. AWS has none here. The google customer experience is just horrific.

segmondy 7y ago

While you're figuring this out, backup all your data on Google. Google is crazy and could possibly delete all your accounts and data.

nitins_jakta 7y ago
Backing up now, thanks. Even worse I'm an Android user. I don't have a good solution for the people that mail me every few months. Oh hey, Google might lock my account because I didn't fork over 14k in fraudulent charges, can you start emailing me from my Outlook email instead?
- quickthrower2 7y ago
  The idea email is your own domain name. I’m yet to take my own advice though!

cjbprime 7y ago

Keep trying to talk to them and explain -- so far every instance like this I've heard about was refunded. Good luck.

londons_explore 7y ago

Google will typically waive charges in cases like this. The only time they won't is if (by looking at the logs) they decide you were probably scraping and storing all their data.

deleted 7y ago

[deleted]

applecrazy 7y ago

Did you check your Github repos and associated commit history for accidental push of secret files? There's an article on the HN front page describing secret leakage in Github repos (the most common is Google API keys, go figure)[1]. I imagine somebody out there has a bot to monitor pushes in realtime to extract secrets. You or a team member might have leaked keys in a similar manner. [1]: https://blog.acolyer.org/2019/04/08/how-bad-can-it-git-characterizing-secret-leakage-in-public-github-repositories/ https://blog.acolyer.org/2019/04/08/how-bad-can-it-git-chara...

samfisher83 7y ago

Next use a credit card. Basically thanks to credit card laws the bank will go tell google to f off and give you your money back. Debit cards don't have the same protection, but just call your bank or OCS (https://www.occ.treas.gov/ https://www.occ.treas.gov/). They have a little more bite.

morituri 7y ago
He was a student so he may not have had access to a credit card when he started the account. But yes, a credit card will have more protection.
joecot 7y ago
His debit card ended up declining it anyway, but that's not the point. The point is Google thinks he owes them thousands of dollars, and a collection agency is hounding him for the money. Disputing a charge with your credit card company doesn't change that. The credit card company might side in your favor and reverse the charges, but the original company can still pursue payment with you directly. Their claim can even show up on your credit report. I remember working with a payment processor in Europe that even automated sending a bill to collections if it was charged back by the consumer's credit card company.
- motherofzappa 7y ago
  playing devils advocate, but taking peoples word prima facie is dangerous. Especially when it deals with financial issues. Pretty sure you can any contempt person could game the system if people found out “students” automatically get refunds for bs charges. I think AWS is different as they have their educate program and distinguish between students and free trial customers. AFAIK gcp doesn’t have an education system for students, so it’s harder to differentiate a legitimate request
  - quickthrower2 7y ago
    What will these system gamers do with computer power? Mining crypto will make them pennies, and running a business on stolen accounts isn’t sustainable.

boulos 7y ago

Sorry this happened to you! Feel free to send me your case number (email in profile), and I'll escalate it. The Support personnel have hopefully been helping out, as all Billing Issues are covered regardless of support tier. I obviously don't know the ins and outs of payment instrument refunds / do debit cards mean that you actually do have to contact your bank, but I'm sure people in Support do.

travisglines 7y ago
Thanks for responding to this and offering to escalate boulos. I certainly don't think a student just learning the ins and outs of a cloud provider's services should be able to spend 10k+ without warnings/thresholds that require configuration to exceed. It would be positive for platform adoption to make that process better.
- motherofzappa 7y ago
  This is hilarious. Student doesn’t understand security in depth model, gets owned. Has a sour taste with said cloud provider. At what level do you accept responsibility for shoddy security practices. If the project was truly defunct then you should have closed the project or removed everyone’s access who isn’t project owner. Hindsight is 20/20.
  - duiker101 7y ago
    Nice victim shaming you got there. The fault of all of this is 100% on whoever stole the credentials and made those calls. OP could maybe have been more careful but that doesn't mean it's all his fault or that we should be shaming in oblivion. And Google can still try to help rather than just take advantage of the situation. Life is easier when we are not dicks to each other, a little empathy can go a long way.
    - jameskegel 7y ago
      I think a lot of the problems we face today are largely unacknowledged by those who create them. The delivery lacked any empathy but does not make it less true. Actions have consequences and ignorance of those outcomes doesn’t mean you can escape them. The world would be a little bit better of a place if we’d stop coddling those who float haphazardly through their own existence.

kkarakk 7y ago

I dunno if google lets you do this but amazon/azure will pretty reliably let you create new free tier accounts with fake emails and access them from the same IP. i just create a new debit/credit card every 6 months(it's pretty hassle free in india). i do pay for production instances, i just don't want to mess around on those production instances