6 ms·

Google fails even here to reassure the market because this is all carefully worded corporate speak about “account management”. What they are really trying to s

Google fails even here to reassure the market because this is all carefully worded corporate speak about “account management”. What they are really trying to say is “Google Cloud is now less likely to delete your entire business computing systems over some minor credit card issues.” For this reason, this press release will not get the attention that google wants it to. Note however that google has included human reviewers in the process and googles humsn reviewers are notorious for being ruthless. It’s googles attitude to customers that is the ingrained and deep problem. Google has built into its DNA from the beginning that it does not want to be in contact with you, that you cannot get emergency attention, and that you are probably wrong.

TheIronYuppie 8y ago
To be clear: we're fixing the algorithms, adding extra humans, and showing customers how to take action on their own. We never want to let what happened here happen again, and we are definitely listening to any and all suggestions. Tl;Dr We screwed up. We're sorry. Disclosure: I work at Google on Google Cloud.
- hguhghuff 8y ago
  Then you should have have press release that says “We screwed up, we’re sorry, and we won’t delete your accounts any more”, instead of all this many-lawyered garbage about “account management policies”, words that no one is interested in. This post has 7 comments and 11 points after an hour. Compare that to the post that led to this outcome. If your trying to reassure the market that you don’t delete a companies entire back end infrastructure then yes this meets some internal management butt covering goal, but does not tell the market definitively that google cloud no longer deletes your entire computer systems.
  - TheIronYuppie 8y ago
    While we debated that exact thing for a long time, the reality was that we knew customers would want to dig deeper and understand HOW we wouldn't let it happen again. We tried to do that here. We tried not to make it sound too lawyerly, but I'll definitely take your feedback internally. Disclosure: I work at Google on Google Cloud.
    - hguhghuff 8y ago
      Amazon does not delete your account as far as I know until some extreme has been reached. You guys need to understand the scale of distrust this issue raises. If you want to fix it, give up on putting lawyers and spin doctors up front. Instead, give this disastrous issue the response it deserves and get the CEO and Chairman of google to make s loud, clear and focused apology and say loudly that we do not delete your entire computer systems over account payment issues until defcon 3 is reached. Why the F would I build my SAAS business on google cloud without that message being yelled repeatedly and loudly and clear from the top level of the company, as much so Google understands it as the market understands it. And by the way, I pay ALL my cloud accounts on credit card. I note carefully that at google this means I have a higher chance of having my entire computer systems deleted than if I use some other payment mechanism. Your policy here appears to me to be using systems deletion add leverage to get me off credit card payment. All of which says to me that google still just doesn’t get it.
      - roblabla 8y ago
        Amazon locked out a seller's account because they changed their Kindle's profile name to "Daddy" (in Chinese). Amazon asked for the seller's ID with that new name, and no amount of support escalation was enough to fix it[0]. It took an absurd amount of bad PR for them to come out and fix it. Google is bad, but Amazon is really just as bad. They both have absurd policies with no recourse, that work in the 80% case and fail very badly in the 20% left. [0] https://www.reddit.com/r/amazon/comments/5gvgdl/using_a_amazon_kindle_fire_destroyed_my/ https://www.reddit.com/r/amazon/comments/5gvgdl/using_a_amaz...
        saurik 8y ago
        Amazon Web Services is essentially unrelated as a business from Amazon.com. This is even less relevant than someone trying to compare the policies of some random end-user service like Google Photos with Google Cloud Platform. I personally had serious issues (not account suspension problems, but an inability to get things like basic tax paperwork to correct addresses) with Amazon Flexible Payments (which was part of some seemingly unrelated-to-everyone division), but I count absolutely none of them against AWS (which wasn't involved in running the service).
- some_account 8y ago
  You have a culture problem. Maybe because you are not a nice company, and you are selling people's private information to ad networks? That's one possible reason z another is that you probably have huge egos everywhere internally, and it's kind of tricky to actually care about customers then.
- protomyth 8y ago
  Why didn't the blog post have a link to the customer's blog post that is referenced in the first paragraph?
- notatoad 8y ago
  >We never want to let what happened here happen again This is basically the problem. You could say "this will never happen again", but instead it's "we don't want this to happen" as if it's somehow out of GCP's control. As long as you will still delete accounts that you suspect of fraud, based on some opaque definition of suspecting fraud, "we don't want this to happen" is the best you can do, and that's not very reassuring.
  - kajecounterhack 8y ago
    > As long as you will still delete accounts that you suspect of fraud, based on some opaque definition of suspecting fraud, "we don't want this to happen" is the best you can do, and that's not very reassuring. Having worked in fraud detection, this is true for every provider you use. The adversarial nature of fraud detection means making _your definition of fraud_ public is not an option because people will use it as a loss function for their fraud. I would just make sure I don't rely on a single provider and minimize single points of failure.
    - notatoad 8y ago
      i understand why fraud detection mechanisms need to be opaque. i don't understand why the consequence has to be "we will delete your account"
      - kajecounterhack 8y ago
        In the context of cloud instances, accounts allocate resources. If they never deleted fraudulent accounts, they couldn't release the allocated resources, which is a huge problem. Imagine bad guy x who steals 100k credit cards, then uses a botnet to spin up VMs on 100k fraudulent accounts; if each VM has 10gb of data on it, you've just frozen 1 petabyte. Say there's a 90 day appeals process; those resources can't be deleted or reclaimed for 90 days, which is already tremendously expensive. If that expanded to forever, you would just have to close up shop. In reality bad guys operate at even larger scales than this, doing all kinds of things. Depending on what company you are, you might not even be able to pull the compute resources away from them right away; so in addition to burning disk alloc they're now burning compute units. It depends on the product too -- what if they spin up pubsub instances and leave messages in the queue? You might have to freeze those queues so that if they're valid customers they don't lose data. The more offerings you have, the more potential loss. But yeah, they don't delete accounts right away -- there is indeed a grace period, and they do keep all your stuff (I suspect legislation may have something to do with it too). Last I remember it goes to appeal process (human review), and your data can't be deleted / resources reclaimed for quite a while. So it's not zero recourse for the user. Sounds like they're adding even more humans into the loop to ensure they don't unfortunately flag and then delete valid users. But it's non-trivial.
- shanghaiaway 8y ago
  Calling people humans is like calling women females. Neckbeard.
skmurphy 8y ago
This is a fantastic insight, it crystallizes a sense of foreboding about Google I have had for several years now since a friend got locked out of his GMail account by Google and had no redress. Dealing with Google is like dealing with an insect colony in contrast to Amazon or Apple support. Other large tech firms like Atlassian offer excellent support as well. I think hguhghuff is right: they don't seem to want a real conversation with their customers. No human touch.
- fjsolwmv 8y ago
  Atlassian is not a "large" tech firm. By any size metric it is 1% of the others you mentioned.
  - skmurphy 8y ago
    Fair point they are in the top 100 but not the top 50 Software/SaaS firms. They are at about a 600M run rate so I don't think of them as a "small" firm but but they not the behemoths that Apple, Google, Amazon, Facebook are.
ikeboy 8y ago
As an Amazon seller, I have to say this is an order of magnitude better response from Google than I'd ever expect from Amazon. Admittedly low standards. I've been locked out at times from my seller account because they wanted to verify a 10 euro purchase on a related account, and blocked the account for fraud until I faxed over a bank statement. A different time, I signed up for AWS, they couldn't charge the card for some reason, and so the account got blocked for fraud and I couldn't log into the seller account.
erichocean 8y ago
I no longer recommend Google Cloud after one of these insta-bans. The previous week, I had run a (literal) multi-million event on GC for Nike (which went fine, no complaints) and I realized immediately that there was literally nothing stopping Google from insta-banning an account in the future in the middle of a Nike event. No warning, no recourse, just "Fuck off, customer." These are live events, I simply cannot risk being shut off for "fraud". Seriously, fuck these guys. My trust level for Google: ZERO. I can't risk my business on that level of stupidity.
- renchap 8y ago
  This can happen with any provider, either due to policy, human error, malicious actor, ... The correct answer here is to use multiple providers and establish proper relationship with them.
- TheIronYuppie 8y ago
  I realize that you may never come back, but "insta-bans" are extremely unlikely in any case, but even less so if you take any of the actions in the post (all for free): - Provide a valid phone number where we can reach you in the event of suspicious activity on your account. - Provide one or more billing admins to your account. - Provide a secondary payment method in case there are problems charging your primary method. - Contact our sales team to see if you qualify for invoice billing instead of relying on credit cards. To be clear, if you do the last one (which you easily qualify for), you are under contract with us. In that case, the phrase "there was literally nothing stopping Google from insta-banning an account in the future in the middle of a Nike event" would be completely incorrect.
ebikelaw 8y ago
It’s odd that people think Google should keep their business running even when “minor credit card issues” result in them not being paid. If you don’t pay the meter your car gets towed. Simple.
- crooked-v 8y ago
  On the other hand, parking meters don't choose to reject your money halfway through your paid time and then immediately dispatch robot tow trucks.
tjpnz 8y ago
>It’s googles attitude to customers that is the ingrained and deep problem. Google has built into its DNA from the beginning that it does not want to be in contact with you, that you cannot get emergency attention, and that you are probably wrong. Of all the tech companies I've interacted with I've never dealt with one that has come off as uniquely elitist and tone-deaf as Google. Steve Yegge (former Googler) makes some very good points on this. https://medium.com/@steve.yegge/why-i-left-google-to-join-grab-86dfffc0be84 https://medium.com/@steve.yegge/why-i-left-google-to-join-gr...

TheIronYuppie 8y ago

To be clear: we're fixing the algorithms, adding extra humans, and showing customers how to take action on their own. We never want to let what happened here happen again, and we are definitely listening to any and all suggestions. Tl;Dr We screwed up. We're sorry. Disclosure: I work at Google on Google Cloud.

hguhghuff 8y ago
Then you should have have press release that says “We screwed up, we’re sorry, and we won’t delete your accounts any more”, instead of all this many-lawyered garbage about “account management policies”, words that no one is interested in. This post has 7 comments and 11 points after an hour. Compare that to the post that led to this outcome. If your trying to reassure the market that you don’t delete a companies entire back end infrastructure then yes this meets some internal management butt covering goal, but does not tell the market definitively that google cloud no longer deletes your entire computer systems.
- TheIronYuppie 8y ago
  While we debated that exact thing for a long time, the reality was that we knew customers would want to dig deeper and understand HOW we wouldn't let it happen again. We tried to do that here. We tried not to make it sound too lawyerly, but I'll definitely take your feedback internally. Disclosure: I work at Google on Google Cloud.
  - hguhghuff 8y ago
    Amazon does not delete your account as far as I know until some extreme has been reached. You guys need to understand the scale of distrust this issue raises. If you want to fix it, give up on putting lawyers and spin doctors up front. Instead, give this disastrous issue the response it deserves and get the CEO and Chairman of google to make s loud, clear and focused apology and say loudly that we do not delete your entire computer systems over account payment issues until defcon 3 is reached. Why the F would I build my SAAS business on google cloud without that message being yelled repeatedly and loudly and clear from the top level of the company, as much so Google understands it as the market understands it. And by the way, I pay ALL my cloud accounts on credit card. I note carefully that at google this means I have a higher chance of having my entire computer systems deleted than if I use some other payment mechanism. Your policy here appears to me to be using systems deletion add leverage to get me off credit card payment. All of which says to me that google still just doesn’t get it.
    - roblabla 8y ago
      Amazon locked out a seller's account because they changed their Kindle's profile name to "Daddy" (in Chinese). Amazon asked for the seller's ID with that new name, and no amount of support escalation was enough to fix it[0]. It took an absurd amount of bad PR for them to come out and fix it. Google is bad, but Amazon is really just as bad. They both have absurd policies with no recourse, that work in the 80% case and fail very badly in the 20% left. [0] https://www.reddit.com/r/amazon/comments/5gvgdl/using_a_amazon_kindle_fire_destroyed_my/ https://www.reddit.com/r/amazon/comments/5gvgdl/using_a_amaz...
      - saurik 8y ago
        Amazon Web Services is essentially unrelated as a business from Amazon.com. This is even less relevant than someone trying to compare the policies of some random end-user service like Google Photos with Google Cloud Platform. I personally had serious issues (not account suspension problems, but an inability to get things like basic tax paperwork to correct addresses) with Amazon Flexible Payments (which was part of some seemingly unrelated-to-everyone division), but I count absolutely none of them against AWS (which wasn't involved in running the service).
some_account 8y ago
You have a culture problem. Maybe because you are not a nice company, and you are selling people's private information to ad networks? That's one possible reason z another is that you probably have huge egos everywhere internally, and it's kind of tricky to actually care about customers then.
protomyth 8y ago
Why didn't the blog post have a link to the customer's blog post that is referenced in the first paragraph?
notatoad 8y ago
>We never want to let what happened here happen again This is basically the problem. You could say "this will never happen again", but instead it's "we don't want this to happen" as if it's somehow out of GCP's control. As long as you will still delete accounts that you suspect of fraud, based on some opaque definition of suspecting fraud, "we don't want this to happen" is the best you can do, and that's not very reassuring.
- kajecounterhack 8y ago
  > As long as you will still delete accounts that you suspect of fraud, based on some opaque definition of suspecting fraud, "we don't want this to happen" is the best you can do, and that's not very reassuring. Having worked in fraud detection, this is true for every provider you use. The adversarial nature of fraud detection means making _your definition of fraud_ public is not an option because people will use it as a loss function for their fraud. I would just make sure I don't rely on a single provider and minimize single points of failure.
  - notatoad 8y ago
    i understand why fraud detection mechanisms need to be opaque. i don't understand why the consequence has to be "we will delete your account"
    - kajecounterhack 8y ago
      In the context of cloud instances, accounts allocate resources. If they never deleted fraudulent accounts, they couldn't release the allocated resources, which is a huge problem. Imagine bad guy x who steals 100k credit cards, then uses a botnet to spin up VMs on 100k fraudulent accounts; if each VM has 10gb of data on it, you've just frozen 1 petabyte. Say there's a 90 day appeals process; those resources can't be deleted or reclaimed for 90 days, which is already tremendously expensive. If that expanded to forever, you would just have to close up shop. In reality bad guys operate at even larger scales than this, doing all kinds of things. Depending on what company you are, you might not even be able to pull the compute resources away from them right away; so in addition to burning disk alloc they're now burning compute units. It depends on the product too -- what if they spin up pubsub instances and leave messages in the queue? You might have to freeze those queues so that if they're valid customers they don't lose data. The more offerings you have, the more potential loss. But yeah, they don't delete accounts right away -- there is indeed a grace period, and they do keep all your stuff (I suspect legislation may have something to do with it too). Last I remember it goes to appeal process (human review), and your data can't be deleted / resources reclaimed for quite a while. So it's not zero recourse for the user. Sounds like they're adding even more humans into the loop to ensure they don't unfortunately flag and then delete valid users. But it's non-trivial.
shanghaiaway 8y ago
Calling people humans is like calling women females. Neckbeard.

skmurphy 8y ago

This is a fantastic insight, it crystallizes a sense of foreboding about Google I have had for several years now since a friend got locked out of his GMail account by Google and had no redress. Dealing with Google is like dealing with an insect colony in contrast to Amazon or Apple support. Other large tech firms like Atlassian offer excellent support as well. I think hguhghuff is right: they don't seem to want a real conversation with their customers. No human touch.

fjsolwmv 8y ago
Atlassian is not a "large" tech firm. By any size metric it is 1% of the others you mentioned.
- skmurphy 8y ago
  Fair point they are in the top 100 but not the top 50 Software/SaaS firms. They are at about a 600M run rate so I don't think of them as a "small" firm but but they not the behemoths that Apple, Google, Amazon, Facebook are.

ikeboy 8y ago

As an Amazon seller, I have to say this is an order of magnitude better response from Google than I'd ever expect from Amazon. Admittedly low standards. I've been locked out at times from my seller account because they wanted to verify a 10 euro purchase on a related account, and blocked the account for fraud until I faxed over a bank statement. A different time, I signed up for AWS, they couldn't charge the card for some reason, and so the account got blocked for fraud and I couldn't log into the seller account.

erichocean 8y ago

I no longer recommend Google Cloud after one of these insta-bans. The previous week, I had run a (literal) multi-million event on GC for Nike (which went fine, no complaints) and I realized immediately that there was literally nothing stopping Google from insta-banning an account in the future in the middle of a Nike event. No warning, no recourse, just "Fuck off, customer." These are live events, I simply cannot risk being shut off for "fraud". Seriously, fuck these guys. My trust level for Google: ZERO. I can't risk my business on that level of stupidity.

renchap 8y ago
This can happen with any provider, either due to policy, human error, malicious actor, ... The correct answer here is to use multiple providers and establish proper relationship with them.
TheIronYuppie 8y ago
I realize that you may never come back, but "insta-bans" are extremely unlikely in any case, but even less so if you take any of the actions in the post (all for free): - Provide a valid phone number where we can reach you in the event of suspicious activity on your account. - Provide one or more billing admins to your account. - Provide a secondary payment method in case there are problems charging your primary method. - Contact our sales team to see if you qualify for invoice billing instead of relying on credit cards. To be clear, if you do the last one (which you easily qualify for), you are under contract with us. In that case, the phrase "there was literally nothing stopping Google from insta-banning an account in the future in the middle of a Nike event" would be completely incorrect.

ebikelaw 8y ago

It’s odd that people think Google should keep their business running even when “minor credit card issues” result in them not being paid. If you don’t pay the meter your car gets towed. Simple.

crooked-v 8y ago
On the other hand, parking meters don't choose to reject your money halfway through your paid time and then immediately dispatch robot tow trucks.

tjpnz 8y ago

>It’s googles attitude to customers that is the ingrained and deep problem. Google has built into its DNA from the beginning that it does not want to be in contact with you, that you cannot get emergency attention, and that you are probably wrong. Of all the tech companies I've interacted with I've never dealt with one that has come off as uniquely elitist and tone-deaf as Google. Steve Yegge (former Googler) makes some very good points on this. https://medium.com/@steve.yegge/why-i-left-google-to-join-grab-86dfffc0be84 https://medium.com/@steve.yegge/why-i-left-google-to-join-gr...