17 ms·

Google Intrusion Detection Problems

374 points by K0nserv 10y ago

abstractbeliefs 10y ago
I don't understand why people are ever surprised about Google offering substandard support for really anything. Time and time again Google products prove to make the happy path happier, but as soon as anything should go wrong, you're on your own with no explanation, and no support. Additionally, and as an aside to the core issue here, I've found the recent Google Cloud UI updates to be a real pain. When I last used them, I did acknowledge that it was beta, but I seemed to lose an awful lot of oversight of an awful lot of things. It's put me seriously off it vs. AWS.
- blakeyrat 10y ago
  Well, if it's something like YouTube you can kind of "forgive" their awful customer service in exchange for the free hosting and search features. It's a heck of a lot less forgivable for a product you pay actual money to use. The support page telling you to use a widget that doesn't exist is really going above and beyond in the realm of user-hostility.
  - burgreblast 10y ago
    FWIW, Dropbox also points you to buttons that don't exist. When you delete an account, DB keeps charging you. There's a nice FAQ that explains to push a "review subscription" button and blah blah, but in reality that button doesn't exist. The button isn't there. Support will "fix"it for you one-off, but not change the button. Wonder if this dark pattern is worth 3-4% of revenue.
    - 0xmohit 10y ago
      > When you delete an account, DB keeps charging you. Maybe they expect you to cancel the card upon deleting an account. It can be said with certainty that this cannot be an oversight. Not many look at their credit card statement anyways.
      - Xorlev 10y ago
        > It can be said with certainty that this cannot be an oversight. It can be, especially when you're using something like Stripe. It only takes one failed call without retries (or a period of time where networking isn't working quite right) for an account to be orphaned. Arguably, those sorts of calls should be durably queued and retried until success, but it often doesn't happen. PM decides it's cheaper to have support field the one-off requests than to spend a few days making those changes. Later on, another developer accidentally disables the call and there isn't sufficient test coverage for disabling billing on account removal. It's a story that'll be repeated over and over.
    - x0x0 10y ago
      The easiest way to fix it is to click the fraud button on your cc account =P
      - burgreblast 10y ago
        Nice if it were that easy. But for business accounts you want to delete one (or more) licenses. Look at the form 2/3 of the way down and ask if this makes sense. https://www.dropboxforum.com/hc/en-us/community/posts/201886305-Deleting-licenses https://www.dropboxforum.com/hc/en-us/community/posts/201886...
  - jwatte 10y ago
    There should be integration tests for documentation. Maybe some machine learning could be applied to that? Maybe I should set that up on some cloud service. Let me check the documentation for how to do that...
    - ben_jones 10y ago
      That's biting off way too much. You have to realize the scope of the actual problem "making humans understand something they previously do not understand". If you can do that you might as well apply it to fixing education instead.
    - ocdtrekkie 10y ago
      One of the things sandstorm.io does which makes a lot of sense, is that they regularly commit changes to their docs within the PRs that change the actual software. It's not integration tested, mind you, but the general practice means they do a pretty good job keeping their docs updated.
    - blakeyrat 10y ago
      Maybe Google needs to have fewer robots involved and more human beings?
- chaostheory 10y ago
  Yeah, Google tends to rely on machines too much for customer support for paid services. I understand the rationale, but it's just not good enough compared to real people handling customer support... yet
- 0xmohit 10y ago
  > It's put me seriously off it vs. AWS. Can't really compare, but AWS isn't great either. I've experienced instances when an instance (1) shut down, (2) rebooted on it's own. AWS support would point you to the SLA if you were to report the incident.
  - sargun 10y ago
    EC2 instances are meant to be ephemeral.
    - bpicolo 10y ago
      That doesn't sound factual at all. Got docs which support that? They send us emails every time they plan on doing intentional maintenance on our instances.
      - 0xmohit 10y ago
        I wish it wasn't true. On all the occasions, I was told that there was "an issue with the underlying hardware".
        bpicolo 10y ago
        Sure, but that's very different than them being intended as ephemeral.
        0xmohit 10y ago
        I didn't suggest that those are ephemeral.
        jasonlotito 10y ago
        Yes, yes you did. Sargun posted: "EC2 instances are meant to be ephemeral" to which bpicolo replied: "That doesn't sound factual at all." You then replied with: "I wish it wasn't true." _it_ in this case refers to the ephemeral state of EC2 instances. You literally did suggest they were ephemeral.
- peteretep 10y ago
  FWIW, anecdotally the Google Fi support was pretty good
CaptSpify 10y ago
Lol Sorry to be so harsh, but Google has always been this way. Why do people keep putting essential stuff in someone else's sandbox? Your effectivly adding a SPoF, that you have no control over
- numbsafari 10y ago
  Is Amazon or Azure or Heroku really any different? I've had similar problems in the past with Amazon shutting down services and providing little to no support to get it remediated. Even after paying for technical support, it was impossible to recover (in our case, it was a billing issue that was entirely Amazon's fault). My general recommendations are: - be sure you have fully scripted your deployment process, - be sure you are making remote backups of critical data (often times this is as easy as setting up simple replication of S3 to GCS or Azure's Cloud Storage) - rely as little as possible on their "value added" features - if you do rely on their value added features, be sure you still have someone on staff who can quickly replace it with something minimal (e.g. know how to install your own mysql) At the very least, this will put you in a position where you aren't 100% locked out of your data, or have to completely rewrite your application in order to move to another provider. You don't have to go crazy with some kind of hot failover, or active-active deployment. Just have daily snapshots of your data and mitigate the risk of your PaaS provider. No matter who your hosting provider is, you should probably have this sketched out as part of a "disaster recovery" plan.
  - CaptSpify 10y ago
    > Is Amazon or Azure or Heroku really any different? Nope, and thats why I don't use them. There are situations where they make sense, but people should be much more skeptical about them. I don't have stats, but IME I've yet to see a company that actually saves money by using them. I've seen one company that did save money the first year, but got too addicted to "just spinning another instance up" and stopped optimizing their code. The next year they were bleeding cash. So, speaking from my experience: It's more expensive, less configurable, and unreliable. Why do people keep buying the cloud lie?
    - 10y ago
nokya 10y ago
The Nth story about that guy who runs a business depending on Google services and suddenly starts telling everyone else they shouldn't run their business on Google services because Google services flagged them and they realized it was a bad idea to run a business based on Google services...
- ocdtrekkie 10y ago
  Yup. This is note even a remotely surprising story. If you're building your business on Google, you're building your business to be arbitrarily shut down. You should not operate any business critical service or store any business critical data on any Google product or service. If anything, this is one of the more generous stories, because Google gave them a day or two of notice first. Usually you just find your Google account shut down one day.
  - pg_is_a_butt 10y ago
    building services on Google Cloud that could also be ported to AWS or Azure or Docker or private servers is incredibly easy. Google Cloud in many cases is nearly free, or very cheap... use them while you can, flip the switch and tell the world if they screw you over. using Google App Engine, Cloud SQL, and the built in Memcache services costs a few dollars a month. using AWS Elastic Beanstalk with RDS and ElastiCache can cost upwards of $40/month. Both of these services are meant to scale, and the costs scale as well. If you're building your business spending more money than you need to, you're building your business to arbitrarily go bankrupt.
  - Kadin 10y ago
    Well, I wouldn't go that far. If you run your business on Google Services, you definitely need to have a contingency plan for what you're going to do the day that Google decides to arbitrarily cut you off, though. But in some instances it might be financially worthwhile to take advantage of their services (particularly their free or very cheap ones) as long as you have a plan for standing up something else when that happens. For instance, I'm pretty comfortable using Gmail, even though I know they might someday pull the plug on my account for no particular reason. I'm comfortable doing that because all my mail is backed-up elsewhere, and in a pinch I could restore everything or stand up a new mailserver in a day. It would be obnoxious, sure, but it's not exactly breaking new technical ground or anything. The years of free service I've gotten out of them make that risk manageable. I could easily see a company taking that bargain for various other services. The problem becomes when you simply rely on a 3rd-party's infrastructure as if it was your own.
danpalmer 10y ago
Of course it's easy to pattern match this to previous stories of advertisers and publishers being banned from Google's ad products, people losing their email access on Gmail, etc, but this is an entirely different area of the company, and one where account managers are very much a thing. It strikes me as odd that one couldn't just email or phone their account manager to ask for clarity on a situation like this?
- hackcasual 10y ago
  Google tries to get away with as little human supervision as possible. I doubt they have an actual account manager
  - deleted 10y ago
    [deleted]
  - DashRattlesnake 10y ago
    They have an account manager, it's just a robot named Samantha West, programmed to insist she's a real person: http://newsfeed.time.com/2013/12/10/meet-the-robot-telemarketer-who-denies-shes-a-robot/ http://newsfeed.time.com/2013/12/10/meet-the-robot-telemarke...
- jwatte 10y ago
  It's part of Google culture, Google DNA. Even though the departments are different, the heritage will show.
- hehheh 10y ago
  The blog author was upset and confused that the email Google sent included an account number instead of account name. I don't think we're getting the whole story here.
jeffmould 10y ago
Not sure if he missed it on the support page, or simply is not willing to pay the added cost, but Google does offer a paid support option with both phone and email support options. Obviously the more you pay the quicker the response and more access you have to those options.
- ocdtrekkie 10y ago
  I suggest reading the article. "Google offers support solutions where you can talk to a person if you have a problem. We view it as problematic that interrupting an “allergic reaction” as a “support issue”. However, we would be willing to purchase top-tier support in order to get this resolved quickly. But there does not appear to be an option to purchase access to a human to get this resolved. Apparently, we should have thought about that before our project was suspended."
  - jeffmould 10y ago
    Hey thanks, but I did read the article, and saw that part. That is why I said that he may have missed that part of the page. And just because he says he would be "willing to purchase top-tier support" doesn't mean that he doesn't have a limit on how much he would pay for said support, thus my saying that maybe it was too much for him.
    - eridius 10y ago
      My impression from the article (and the quoted bit) is that you can't pay for support once your project is suspended.
      - DannyBee 10y ago
        As far as i can tell, this is false (I have a suspended test project, and the support options still seem to be there) I think he means "there is no 'pay to talk to a human'" option in the FAQ/etc for this issue. That is, he wants to pay to talk to a human about just this issue, and thinks somehow that paying for the support option will not give him that (or is too expensive for that).
redstripe 10y ago
This is the google M.O. - optimize all processes through cheap machine learning and heuristics and accept that some people will get screwed along the way. The number of people that can be served at very little cost without any human involvement is apparently very lucrative. Google just considers customer service as an archaic relic that predates the invention of behavioral algorithms.
- tlogan 10y ago
  And it works quite well from them. Yes, people on HN and on internet will complain... Also it could be that customer support is just overrated.... The problem is that 95% of support calls / issues are anyway just people complaining about things which are not at all related to the actual Google service. But, on the other hand, hackers which try Google Cloud and are bitten by lack of Google support do have influence on decision making process in big corporations.
  - Jordrok 10y ago
    Yeah, how dare all those little people complain when they're inadvertently crushed by the glorious Google behemoth. Don't they realize how insignificant they are? Their entire lives are simply rounding errors at Internet Scale. Sometimes the end result is less important than how you get there, and how you treat the ones who don't fit into your plans. It happens all the time with big companies, but Google even more than most. They KNOW that the potential for problems exists, but so long as it only affects 5%, who cares? Fuck you, got mine. I think that the attitude conveyed by Google's lack of support is what inspires a lot of people to post rants online, rather than the frequency of incidents.
pellej_s 10y ago
So, everyone's basically saying: > Google sucks > Don't rely on Google if you want your business to succeed > This is old news! Happens all the time > Google = SPoF ...you're all looking like trolls to me. Please, do tell how Spotify manages to service 60+ markets with some core pieces of infrastructure on Google's Cloud offerings.
- Guyag 10y ago
  I don't know that the comparison with Spotify is very fair - I would imagine they have a direct contact to employees.
- ocdtrekkie 10y ago
  Pretty sure if Google guts Spotify, Spotify can literally call Sundar Pichai himself on the phone to get it resolved. There's a scale issue there in terms of whether or not Google considers you worth bothering with.
- 1_2__3 10y ago
  By being Spotify-scale, thus forcing Google to care about them. See also: Snapchat.
- eadz 10y ago
  Spotify will have a phone number of someone they can call. I've had this issue with google too. They'll shut down your account, your whole account - google apps email and all - if an automated system detects something dodgy. In my case an automated system at ebay accused my site of phishing which it wasn't. There is a paid support option, but when your account is shut down you are unable to access the required code. I don't think you can appreciate how inaccessible Google is until they decide you're a bad actor.
- deleted 10y ago
  [deleted]
- CaptSpify 10y ago
  Google doesn't typically have bad products. Google is amazing when things go right, it's only when they go wrong that Google really shows their true nature. I'd wager a guess that most of the time things just don't go wrong. I also imagine that Spotify pays a lot of money and has a dedicated account manager.
tscanausa 10y ago
My name is Terrance and I work on Google's Cloud Support Team. Our team mission is to Reduce Customer Anxiety. We take that very seriously, and Fred's experience obviously shows that we fell short this time. The process should have been an easy flow to follow, and it was not. We are reviewing this incident in detail to ensure that we make the process less error prone and quicker in the future. Best, Terrance
- aluminussoma 10y ago
  Google Cloud publishes a Root Cause analysis after technical incidents. I wonder if this non-technical incident is worthy of a root cause analysis. As a potential customer, I know I'd feel more comfortable seeing this organizational "bug" fixed.
- ben_jones 10y ago
  Personally I really value comments like this and therefore feel the need to point out that the statement "Our team mission is to Reduce Customer Anxiety." seems like a bullet point for a status meeting and likely does not convey the desired effect of such a post: to have a somewhat personal response from a normally faceless company.
  - CaptSpify 10y ago
    It actually gives me the opposite feeling than probably intended: "We'd rather pay someone to post BS responses on social news sites, than dedicate someone to fix the problem." I'm not saying that's actually what is happening, but it feels like it to me.
  - blakeyrat 10y ago
    If people wanted their anxiety reduced, they'd sign up for a guided meditation group. People want their accounts reactivated.
- cocotino 10y ago
  As I was reading I thought this was one of those pawn shop copypastas.
aluminussoma 10y ago
Reaching a human being is a common problem with all Internet businesses. I've had similar frustrations with Google competitors. As an industry, we have to do much better with customer service. Our users are real human beings.
- ocdtrekkie 10y ago
  This is why when it comes to web hosting, I only do business with companies that have 24/7 phone support. It limits my options a lot. GoDaddy and HostGator both provide this, most others I've seen don't. But before I got my first HostGator account, I called their tech support line at 2 AM, and got a guy named Chris with a strong Texan accent. And I've had escalated issues at odd hours where I've actually spoken directly to sysadmins there. Ticket support only hosting simply can't compete with that service, no matter the cost.
colinbartlett 10y ago
If Google is ever going to fix their cloud market share problem[1], they need to reverse the deeply engrained perception that they don't support or maintain their services for the long term. The majority of Hacker News pundits will all have the same gut reaction to these kinds of anecdotes: complete and total lack of surprise. And although not all of us are in a position to control the vast budgets of the enterprises that drive much of this marketshare, the opinions of the rank and file do have a vast impact and I believe Google's unstable support and product commitment is one of the biggest things holding them back. 1. http://www.nytimes.com/2016/07/25/technology/google-races-to-catch-up-in-cloud-computing.html http://www.nytimes.com/2016/07/25/technology/google-races-to...
- abrookewood 10y ago
  You are soo right. I really want Google to succeed (more competition the better), but I hear things like this and I'm just blown away. They are already fighting a reputation for abandoning projects when it suits them (yes, mainly consumer applications, but the perception persists) and doing this just makes it worse. C'mon Google ... hire some marketing people that know what they are doing and sort out your customer perception issues.
- foxylad 10y ago
  I just want to say that Google's supposed lack of support is just a perception, despite all the usual comments below. Real example: we use Appengine, and the other day I spotted a bug in a request handler. I filed an issue on the Appengine issue tracker, and have been having a back and forth conversation with a a very real (and very skilled) engineer about how to resolve it. This is outside our support contract, so anyone with an software issue has exactly the same access. This doesn't cover situations such as the OP experienced, of course - but if they had a support contract, I think they'd have found the issue was resolved quickly and efficiently. We pay $150 a month for a silver support contract, and when we've needed help, we've got personal, prompt and effective support. At the end of the day, you get what you pay for. If your organisation depends on any service, you need effective support, and you need to pay for it. That goes for Microsoft, Amazon, Salesforce or anyone else - good support costs a lot, and they all have to cover costs. Each company uses a different model, and some include more support costs in their base charges than others. I happen to like Google's model where dipping your toe in the water costs very little, and when your service proves itself you can ramp up support as you need it. But it leaves Google open to criticism like this the OPs, who let their small project grow big without contingency planning and then complains when something goes wrong. Finally, the only reason I'm commenting here is that I was so impressed with the support from the engineer mentioned above that I said I would mention him next time Google's perceived lack of support came up on HN. Promise kept!
20years 10y ago
This is a perfect example of why I refuse to use Google Cloud. Google in the past has put me through similar automated loops with Adwords and Adsense. Even disabling an Adwords account for 2 weeks for reasons I still do not completely understand. After weeks of trying to get a hold of the right person and being pushed through a ton of different channels, it was finally re-instated with a generic apology and still no concrete reason as to why. I have learned that it is way too risky relying on Google services for crucial business operations.
Animats 10y ago
Intrusion detection false alarms can be a problem if you run anything like a web crawler. I have one running at "sitetruth.com", which is a site rating system hosted on a leased server. (Not a cloud service, a leased rackmount server). One of the things it does is to find the home page of a site by trying "example.com" and "www.example.com", with and without HTTPS. Some sites will block access for 30-45 seconds if those requests are made too fast. About once a year, there's a serious intrusion complaint, as the crawler, which obeys robots.txt, examines about 20 pages on a site in a few seconds. No more than three connections at once, but some sites are touchy. The server leasing company sends me a warning letter, I reply and call tech support, and there's no big problem. I can recommend leasing servers from Codero as an alternative to dealing with the Borg of Mountain View. I've been a customer for five years, and nothing bad has happened. They now have "cloud services" too, but I haven't used them.
- noja 10y ago
  > the crawler, which obeys robots.txt, examines about 20 pages on a site in a few seconds. That seems like something that could be slowed down slightly with little negative impact, and a big positive impact (no annoyed people, no interaction with support).
  - Animats 10y ago
    Our user is waiting. We rate sites on demand, and the user is looking at a rotating "busy" icon in their browser search results during the rating process.
paradox95 10y ago
Once had Google kill a Spark job after it was running for about 12 hours because they thought we were doing something nefarious. It cost us thousands of dollars in wasted time and Google spend because their automated system made a mistake. They never attempted to fix it, refund us or even seem concerned.
- bigiain 10y ago
  And you compare that to the many stories you hear about Amazon waiving invoices when people inadvertently ran up multi thousand dollar AWS bills by accident or when credentials were exposed. (Happened to an ex-colleague of mine years back - AWS creds committed to a public git repo - minutes later there's like 40 10xlarge or g2.8xlarge or whatever instances mining bitcoin. First thing he knew about it was Amazon ringing him up saying "this $10,000 spike in your typical use, that's not really you, right?" and shutting it all down for him and reversing the charge...) Then you consider this when you decide whether to base the next big business decision on an AWS or Google Cloud platform…
oconnore 10y ago
Just wait until you can buy one of their self driving cars.
pjjw 10y ago
Dollars to donuts these guys got hacked. Still pretty shitty the appeal process was busted.
deleted 10y ago
[deleted]
zippy786 10y ago
Google, said to have very high hiring bar, yet I've seen more than a few times now that their product lines are sub-par. I'm sure the explanation they give is that we are very big and we can ignore the 1% corner cases and have the mentality that "we are so big we don't really care about smaller parts of the internet". Also, the hypocrisy in using very logical questions to hire people but not sticking to similar logic in the product line is laughable.
- blakeyrat 10y ago
  It feels like they have an entire company of Mr. Spocks without a single Doctor McCoy to round things out.
  - PhantomGremlin 10y ago
    Mr. Spock would have approached the issue of support more logically and would have thought things through more carefully. It's more like they have an entire company of Sheldon Coopers, without a single Penny or Leonard Hofstadter to round things out.
    - zippy786 10y ago
      Sheldon would have never agreed to release a system with flawed design.
Safety1stClyde 10y ago
Google = rain man.
martinald 10y ago
We got the 100k credit which is nice. Noticed the actual technical side is excellent - vastly better than azure and AWS (AWS especially - I find the whole UI and way it works so clunky). However, despite using our credit to buy gold support when it came to using it I couldn't work their tickets UI at all. Turns out you need to change products from your Google apps support to cloud platform by some tiny link which is not at all obvious. I had to phone up and it was obvious people hit it all the time as the operator instantly knew what the issue was. Also related: azure by default didnt renew your free trial despite an active cc being on file. It just shuts everything off with no obvious warning. What other service does that? Honestly think these companies need to do some proper usability testing on their flows because there is so much clunky weirdness in cloud providers right now.
yladiz 10y ago
This makes me wonder, which do people dislike more: not having access to support in almost any capacity except "help desk tickets" which might not get answered for days, e.g. how Google handles non-free support tiers, or horribly long wait times on phones? My gut says that the former is worse, especially in this case, because you can just turn your phone on speaker and wait for the queue to finish up (or with some systems, have them call you back when the queue is to you). I almost can't believe that Google doesn't have a support line for its account holders for Google Cloud, when you have companies like Paypal and United Airlines, which have many more users (paying or not) and have support lines.
damm 10y ago
I admit to having a gmail.com account ---- I really don't understand how anyone can use Google for production. If you don't pay for support to have a way to actually get a hold of Google... ... Someday you may be in a world of hurt and have absolutely no way to get ahold of anyone. You can post a story and hope and pray that Google might read it and have someone reach out and remedy the situation.
yclept 10y ago
This sounds like a nightmare. One that I can't imagine happening at Rackspace Cloud...
ne01 10y ago
I hate that Google don't give a damn about its users. To them interacting with humans is a waste of time and money... They prefer to automate everything and manage servers, robots and programs instead. The same thing happened to us in 2013 luckily had backup servers at Linode.. took them 2 weeks to solve the problem it was a bug in their billing system. Google gave us $8k credit and I gave them another try and convinced myself that GCE was very young in 2013. Now I'm just worried! It's 2016 and our entire business depends on their platform and their support sucks! Even the $400/month version!! Google, why can't you have support like DigitalOcean and Linode?
- hga 10y ago
  Google, why can't you have support like DigitalOcean and Linode? Because it's totally not in their DNA? Because GCE and company are relatively low priority offerings? Would you agree these are higher: Ad Words etc., which make the bulk of their money. Search etc. which among other things has you seeing those ads. Android and all their other efforts to keep themselves from losing their access to the users of the above. For a while, Google+ was an anomalous offering they were pushing very hard (with this sort of brain damage with linked accounts that hit hard with their real names policy), but they might try such a stunt again. Compared to the two service providers your mentioned, which only do one thing. Even AWS is I gather rather like that, there's it + all the Amazon selling stuff which is significantly separate. Some of the DNA is shared, but that's not all bad, such as the focus on customers.
76bd 10y ago
The situation of having a project fully shut down without adequate information or process by which to correct it is clearly unacceptable. That said, I was involved in recently migrating to GCP and it has worked wonderfully for us thus far. We run a fairly standard java/MySQL web app, with paid silver support. Questions are answered promptly in our experience. We found compute engine VMs to be faster (and cheaper) than EC2, the developer console easier to use, and better quality documentation/APIs.
manigandham 10y ago
I usually defend GCP as they have better technology and platform architecture but I definitely agree that they have major communication problems. While there are plenty of individuals available through social channels, it does seem to be very informal most of the time which does not inspire confidence as a business looking for a solid foundation.
hehheh 10y ago
Bet you cash money the affected site is or was in a ddos botnet.
fhoffa 10y ago
Posting this 4 days later, what happened next: http://www.businessinsider.com/google-cloud-won-skeptic-after-shutting-site-down-2016-8 http://www.businessinsider.com/google-cloud-won-skeptic-afte... Google then shocked him again, in a good way. Within four hours of tweeting, someone from Google had contacted him and had restored access to his project. Trotter says that, it turns out, he and his team bear some responsibility. They had inadvertently set up a server wrong, exposing a hole, and a hacker was using his company's to conduct a "denial of service attack," which is when hackers overload another website or online service with so much traffic, it shuts down.

374 points by K0nserv 10y ago

I don't understand why people are ever surprised about Google offering substandard support for really anything. Time and time again Google products prove to make the happy path happier, but as soon as anything should go wrong, you're on your own with no explanation, and no support. Additionally, and as an aside to the core issue here, I've found the recent Google Cloud UI updates to be a real pain. When I last used them, I did acknowledge that it was beta, but I seemed to lose an awful lot of oversight of an awful lot of things. It's put me seriously off it vs. AWS.

blakeyrat 10y ago
Well, if it's something like YouTube you can kind of "forgive" their awful customer service in exchange for the free hosting and search features. It's a heck of a lot less forgivable for a product you pay actual money to use. The support page telling you to use a widget that doesn't exist is really going above and beyond in the realm of user-hostility.
- burgreblast 10y ago
  FWIW, Dropbox also points you to buttons that don't exist. When you delete an account, DB keeps charging you. There's a nice FAQ that explains to push a "review subscription" button and blah blah, but in reality that button doesn't exist. The button isn't there. Support will "fix"it for you one-off, but not change the button. Wonder if this dark pattern is worth 3-4% of revenue.
  - 0xmohit 10y ago
    > When you delete an account, DB keeps charging you. Maybe they expect you to cancel the card upon deleting an account. It can be said with certainty that this cannot be an oversight. Not many look at their credit card statement anyways.
    - Xorlev 10y ago
      > It can be said with certainty that this cannot be an oversight. It can be, especially when you're using something like Stripe. It only takes one failed call without retries (or a period of time where networking isn't working quite right) for an account to be orphaned. Arguably, those sorts of calls should be durably queued and retried until success, but it often doesn't happen. PM decides it's cheaper to have support field the one-off requests than to spend a few days making those changes. Later on, another developer accidentally disables the call and there isn't sufficient test coverage for disabling billing on account removal. It's a story that'll be repeated over and over.
  - x0x0 10y ago
    The easiest way to fix it is to click the fraud button on your cc account =P
    - burgreblast 10y ago
      Nice if it were that easy. But for business accounts you want to delete one (or more) licenses. Look at the form 2/3 of the way down and ask if this makes sense. https://www.dropboxforum.com/hc/en-us/community/posts/201886305-Deleting-licenses https://www.dropboxforum.com/hc/en-us/community/posts/201886...
- jwatte 10y ago
  There should be integration tests for documentation. Maybe some machine learning could be applied to that? Maybe I should set that up on some cloud service. Let me check the documentation for how to do that...
  - ben_jones 10y ago
    That's biting off way too much. You have to realize the scope of the actual problem "making humans understand something they previously do not understand". If you can do that you might as well apply it to fixing education instead.
  - ocdtrekkie 10y ago
    One of the things sandstorm.io does which makes a lot of sense, is that they regularly commit changes to their docs within the PRs that change the actual software. It's not integration tested, mind you, but the general practice means they do a pretty good job keeping their docs updated.
  - blakeyrat 10y ago
    Maybe Google needs to have fewer robots involved and more human beings?
chaostheory 10y ago
Yeah, Google tends to rely on machines too much for customer support for paid services. I understand the rationale, but it's just not good enough compared to real people handling customer support... yet
0xmohit 10y ago
> It's put me seriously off it vs. AWS. Can't really compare, but AWS isn't great either. I've experienced instances when an instance (1) shut down, (2) rebooted on it's own. AWS support would point you to the SLA if you were to report the incident.
- sargun 10y ago
  EC2 instances are meant to be ephemeral.
  - bpicolo 10y ago
    That doesn't sound factual at all. Got docs which support that? They send us emails every time they plan on doing intentional maintenance on our instances.
    - 0xmohit 10y ago
      I wish it wasn't true. On all the occasions, I was told that there was "an issue with the underlying hardware".
      - bpicolo 10y ago
        Sure, but that's very different than them being intended as ephemeral.
        0xmohit 10y ago
        I didn't suggest that those are ephemeral.
        jasonlotito 10y ago
        Yes, yes you did. Sargun posted: "EC2 instances are meant to be ephemeral" to which bpicolo replied: "That doesn't sound factual at all." You then replied with: "I wish it wasn't true." _it_ in this case refers to the ephemeral state of EC2 instances. You literally did suggest they were ephemeral.
peteretep 10y ago
FWIW, anecdotally the Google Fi support was pretty good

CaptSpify 10y ago

Lol Sorry to be so harsh, but Google has always been this way. Why do people keep putting essential stuff in someone else's sandbox? Your effectivly adding a SPoF, that you have no control over

numbsafari 10y ago
Is Amazon or Azure or Heroku really any different? I've had similar problems in the past with Amazon shutting down services and providing little to no support to get it remediated. Even after paying for technical support, it was impossible to recover (in our case, it was a billing issue that was entirely Amazon's fault). My general recommendations are: - be sure you have fully scripted your deployment process, - be sure you are making remote backups of critical data (often times this is as easy as setting up simple replication of S3 to GCS or Azure's Cloud Storage) - rely as little as possible on their "value added" features - if you do rely on their value added features, be sure you still have someone on staff who can quickly replace it with something minimal (e.g. know how to install your own mysql) At the very least, this will put you in a position where you aren't 100% locked out of your data, or have to completely rewrite your application in order to move to another provider. You don't have to go crazy with some kind of hot failover, or active-active deployment. Just have daily snapshots of your data and mitigate the risk of your PaaS provider. No matter who your hosting provider is, you should probably have this sketched out as part of a "disaster recovery" plan.
- CaptSpify 10y ago
  > Is Amazon or Azure or Heroku really any different? Nope, and thats why I don't use them. There are situations where they make sense, but people should be much more skeptical about them. I don't have stats, but IME I've yet to see a company that actually saves money by using them. I've seen one company that did save money the first year, but got too addicted to "just spinning another instance up" and stopped optimizing their code. The next year they were bleeding cash. So, speaking from my experience: It's more expensive, less configurable, and unreliable. Why do people keep buying the cloud lie?
  - 10y ago

nokya 10y ago

The Nth story about that guy who runs a business depending on Google services and suddenly starts telling everyone else they shouldn't run their business on Google services because Google services flagged them and they realized it was a bad idea to run a business based on Google services...

ocdtrekkie 10y ago
Yup. This is note even a remotely surprising story. If you're building your business on Google, you're building your business to be arbitrarily shut down. You should not operate any business critical service or store any business critical data on any Google product or service. If anything, this is one of the more generous stories, because Google gave them a day or two of notice first. Usually you just find your Google account shut down one day.
- pg_is_a_butt 10y ago
  building services on Google Cloud that could also be ported to AWS or Azure or Docker or private servers is incredibly easy. Google Cloud in many cases is nearly free, or very cheap... use them while you can, flip the switch and tell the world if they screw you over. using Google App Engine, Cloud SQL, and the built in Memcache services costs a few dollars a month. using AWS Elastic Beanstalk with RDS and ElastiCache can cost upwards of $40/month. Both of these services are meant to scale, and the costs scale as well. If you're building your business spending more money than you need to, you're building your business to arbitrarily go bankrupt.
- Kadin 10y ago
  Well, I wouldn't go that far. If you run your business on Google Services, you definitely need to have a contingency plan for what you're going to do the day that Google decides to arbitrarily cut you off, though. But in some instances it might be financially worthwhile to take advantage of their services (particularly their free or very cheap ones) as long as you have a plan for standing up something else when that happens. For instance, I'm pretty comfortable using Gmail, even though I know they might someday pull the plug on my account for no particular reason. I'm comfortable doing that because all my mail is backed-up elsewhere, and in a pinch I could restore everything or stand up a new mailserver in a day. It would be obnoxious, sure, but it's not exactly breaking new technical ground or anything. The years of free service I've gotten out of them make that risk manageable. I could easily see a company taking that bargain for various other services. The problem becomes when you simply rely on a 3rd-party's infrastructure as if it was your own.

danpalmer 10y ago

Of course it's easy to pattern match this to previous stories of advertisers and publishers being banned from Google's ad products, people losing their email access on Gmail, etc, but this is an entirely different area of the company, and one where account managers are very much a thing. It strikes me as odd that one couldn't just email or phone their account manager to ask for clarity on a situation like this?

hackcasual 10y ago
Google tries to get away with as little human supervision as possible. I doubt they have an actual account manager
- deleted 10y ago
  [deleted]
- DashRattlesnake 10y ago
  They have an account manager, it's just a robot named Samantha West, programmed to insist she's a real person: http://newsfeed.time.com/2013/12/10/meet-the-robot-telemarketer-who-denies-shes-a-robot/ http://newsfeed.time.com/2013/12/10/meet-the-robot-telemarke...
jwatte 10y ago
It's part of Google culture, Google DNA. Even though the departments are different, the heritage will show.
hehheh 10y ago
The blog author was upset and confused that the email Google sent included an account number instead of account name. I don't think we're getting the whole story here.

jeffmould 10y ago

Not sure if he missed it on the support page, or simply is not willing to pay the added cost, but Google does offer a paid support option with both phone and email support options. Obviously the more you pay the quicker the response and more access you have to those options.

ocdtrekkie 10y ago
I suggest reading the article. "Google offers support solutions where you can talk to a person if you have a problem. We view it as problematic that interrupting an “allergic reaction” as a “support issue”. However, we would be willing to purchase top-tier support in order to get this resolved quickly. But there does not appear to be an option to purchase access to a human to get this resolved. Apparently, we should have thought about that before our project was suspended."
- jeffmould 10y ago
  Hey thanks, but I did read the article, and saw that part. That is why I said that he may have missed that part of the page. And just because he says he would be "willing to purchase top-tier support" doesn't mean that he doesn't have a limit on how much he would pay for said support, thus my saying that maybe it was too much for him.
  - eridius 10y ago
    My impression from the article (and the quoted bit) is that you can't pay for support once your project is suspended.
    - DannyBee 10y ago
      As far as i can tell, this is false (I have a suspended test project, and the support options still seem to be there) I think he means "there is no 'pay to talk to a human'" option in the FAQ/etc for this issue. That is, he wants to pay to talk to a human about just this issue, and thinks somehow that paying for the support option will not give him that (or is too expensive for that).

redstripe 10y ago

This is the google M.O. - optimize all processes through cheap machine learning and heuristics and accept that some people will get screwed along the way. The number of people that can be served at very little cost without any human involvement is apparently very lucrative. Google just considers customer service as an archaic relic that predates the invention of behavioral algorithms.

tlogan 10y ago
And it works quite well from them. Yes, people on HN and on internet will complain... Also it could be that customer support is just overrated.... The problem is that 95% of support calls / issues are anyway just people complaining about things which are not at all related to the actual Google service. But, on the other hand, hackers which try Google Cloud and are bitten by lack of Google support do have influence on decision making process in big corporations.
- Jordrok 10y ago
  Yeah, how dare all those little people complain when they're inadvertently crushed by the glorious Google behemoth. Don't they realize how insignificant they are? Their entire lives are simply rounding errors at Internet Scale. Sometimes the end result is less important than how you get there, and how you treat the ones who don't fit into your plans. It happens all the time with big companies, but Google even more than most. They KNOW that the potential for problems exists, but so long as it only affects 5%, who cares? Fuck you, got mine. I think that the attitude conveyed by Google's lack of support is what inspires a lot of people to post rants online, rather than the frequency of incidents.

pellej_s 10y ago

So, everyone's basically saying: > Google sucks > Don't rely on Google if you want your business to succeed > This is old news! Happens all the time > Google = SPoF ...you're all looking like trolls to me. Please, do tell how Spotify manages to service 60+ markets with some core pieces of infrastructure on Google's Cloud offerings.

Guyag 10y ago
I don't know that the comparison with Spotify is very fair - I would imagine they have a direct contact to employees.
ocdtrekkie 10y ago
Pretty sure if Google guts Spotify, Spotify can literally call Sundar Pichai himself on the phone to get it resolved. There's a scale issue there in terms of whether or not Google considers you worth bothering with.
1_2__3 10y ago
By being Spotify-scale, thus forcing Google to care about them. See also: Snapchat.
eadz 10y ago
Spotify will have a phone number of someone they can call. I've had this issue with google too. They'll shut down your account, your whole account - google apps email and all - if an automated system detects something dodgy. In my case an automated system at ebay accused my site of phishing which it wasn't. There is a paid support option, but when your account is shut down you are unable to access the required code. I don't think you can appreciate how inaccessible Google is until they decide you're a bad actor.
deleted 10y ago
[deleted]
CaptSpify 10y ago
Google doesn't typically have bad products. Google is amazing when things go right, it's only when they go wrong that Google really shows their true nature. I'd wager a guess that most of the time things just don't go wrong. I also imagine that Spotify pays a lot of money and has a dedicated account manager.

tscanausa 10y ago

My name is Terrance and I work on Google's Cloud Support Team. Our team mission is to Reduce Customer Anxiety. We take that very seriously, and Fred's experience obviously shows that we fell short this time. The process should have been an easy flow to follow, and it was not. We are reviewing this incident in detail to ensure that we make the process less error prone and quicker in the future. Best, Terrance

aluminussoma 10y ago
Google Cloud publishes a Root Cause analysis after technical incidents. I wonder if this non-technical incident is worthy of a root cause analysis. As a potential customer, I know I'd feel more comfortable seeing this organizational "bug" fixed.
ben_jones 10y ago
Personally I really value comments like this and therefore feel the need to point out that the statement "Our team mission is to Reduce Customer Anxiety." seems like a bullet point for a status meeting and likely does not convey the desired effect of such a post: to have a somewhat personal response from a normally faceless company.
- CaptSpify 10y ago
  It actually gives me the opposite feeling than probably intended: "We'd rather pay someone to post BS responses on social news sites, than dedicate someone to fix the problem." I'm not saying that's actually what is happening, but it feels like it to me.
- blakeyrat 10y ago
  If people wanted their anxiety reduced, they'd sign up for a guided meditation group. People want their accounts reactivated.
cocotino 10y ago
As I was reading I thought this was one of those pawn shop copypastas.

aluminussoma 10y ago

Reaching a human being is a common problem with all Internet businesses. I've had similar frustrations with Google competitors. As an industry, we have to do much better with customer service. Our users are real human beings.

ocdtrekkie 10y ago
This is why when it comes to web hosting, I only do business with companies that have 24/7 phone support. It limits my options a lot. GoDaddy and HostGator both provide this, most others I've seen don't. But before I got my first HostGator account, I called their tech support line at 2 AM, and got a guy named Chris with a strong Texan accent. And I've had escalated issues at odd hours where I've actually spoken directly to sysadmins there. Ticket support only hosting simply can't compete with that service, no matter the cost.

colinbartlett 10y ago

If Google is ever going to fix their cloud market share problem[1], they need to reverse the deeply engrained perception that they don't support or maintain their services for the long term. The majority of Hacker News pundits will all have the same gut reaction to these kinds of anecdotes: complete and total lack of surprise. And although not all of us are in a position to control the vast budgets of the enterprises that drive much of this marketshare, the opinions of the rank and file do have a vast impact and I believe Google's unstable support and product commitment is one of the biggest things holding them back. 1. http://www.nytimes.com/2016/07/25/technology/google-races-to-catch-up-in-cloud-computing.html http://www.nytimes.com/2016/07/25/technology/google-races-to...

abrookewood 10y ago
You are soo right. I really want Google to succeed (more competition the better), but I hear things like this and I'm just blown away. They are already fighting a reputation for abandoning projects when it suits them (yes, mainly consumer applications, but the perception persists) and doing this just makes it worse. C'mon Google ... hire some marketing people that know what they are doing and sort out your customer perception issues.
foxylad 10y ago
I just want to say that Google's supposed lack of support is just a perception, despite all the usual comments below. Real example: we use Appengine, and the other day I spotted a bug in a request handler. I filed an issue on the Appengine issue tracker, and have been having a back and forth conversation with a a very real (and very skilled) engineer about how to resolve it. This is outside our support contract, so anyone with an software issue has exactly the same access. This doesn't cover situations such as the OP experienced, of course - but if they had a support contract, I think they'd have found the issue was resolved quickly and efficiently. We pay $150 a month for a silver support contract, and when we've needed help, we've got personal, prompt and effective support. At the end of the day, you get what you pay for. If your organisation depends on any service, you need effective support, and you need to pay for it. That goes for Microsoft, Amazon, Salesforce or anyone else - good support costs a lot, and they all have to cover costs. Each company uses a different model, and some include more support costs in their base charges than others. I happen to like Google's model where dipping your toe in the water costs very little, and when your service proves itself you can ramp up support as you need it. But it leaves Google open to criticism like this the OPs, who let their small project grow big without contingency planning and then complains when something goes wrong. Finally, the only reason I'm commenting here is that I was so impressed with the support from the engineer mentioned above that I said I would mention him next time Google's perceived lack of support came up on HN. Promise kept!

20years 10y ago

This is a perfect example of why I refuse to use Google Cloud. Google in the past has put me through similar automated loops with Adwords and Adsense. Even disabling an Adwords account for 2 weeks for reasons I still do not completely understand. After weeks of trying to get a hold of the right person and being pushed through a ton of different channels, it was finally re-instated with a generic apology and still no concrete reason as to why. I have learned that it is way too risky relying on Google services for crucial business operations.

Animats 10y ago

Intrusion detection false alarms can be a problem if you run anything like a web crawler. I have one running at "sitetruth.com", which is a site rating system hosted on a leased server. (Not a cloud service, a leased rackmount server). One of the things it does is to find the home page of a site by trying "example.com" and "www.example.com", with and without HTTPS. Some sites will block access for 30-45 seconds if those requests are made too fast. About once a year, there's a serious intrusion complaint, as the crawler, which obeys robots.txt, examines about 20 pages on a site in a few seconds. No more than three connections at once, but some sites are touchy. The server leasing company sends me a warning letter, I reply and call tech support, and there's no big problem. I can recommend leasing servers from Codero as an alternative to dealing with the Borg of Mountain View. I've been a customer for five years, and nothing bad has happened. They now have "cloud services" too, but I haven't used them.

noja 10y ago
> the crawler, which obeys robots.txt, examines about 20 pages on a site in a few seconds. That seems like something that could be slowed down slightly with little negative impact, and a big positive impact (no annoyed people, no interaction with support).
- Animats 10y ago
  Our user is waiting. We rate sites on demand, and the user is looking at a rotating "busy" icon in their browser search results during the rating process.

paradox95 10y ago

Once had Google kill a Spark job after it was running for about 12 hours because they thought we were doing something nefarious. It cost us thousands of dollars in wasted time and Google spend because their automated system made a mistake. They never attempted to fix it, refund us or even seem concerned.

bigiain 10y ago
And you compare that to the many stories you hear about Amazon waiving invoices when people inadvertently ran up multi thousand dollar AWS bills by accident or when credentials were exposed. (Happened to an ex-colleague of mine years back - AWS creds committed to a public git repo - minutes later there's like 40 10xlarge or g2.8xlarge or whatever instances mining bitcoin. First thing he knew about it was Amazon ringing him up saying "this $10,000 spike in your typical use, that's not really you, right?" and shutting it all down for him and reversing the charge...) Then you consider this when you decide whether to base the next big business decision on an AWS or Google Cloud platform…

oconnore 10y ago

Just wait until you can buy one of their self driving cars.

pjjw 10y ago

Dollars to donuts these guys got hacked. Still pretty shitty the appeal process was busted.

deleted 10y ago

[deleted]

zippy786 10y ago

Google, said to have very high hiring bar, yet I've seen more than a few times now that their product lines are sub-par. I'm sure the explanation they give is that we are very big and we can ignore the 1% corner cases and have the mentality that "we are so big we don't really care about smaller parts of the internet". Also, the hypocrisy in using very logical questions to hire people but not sticking to similar logic in the product line is laughable.

blakeyrat 10y ago
It feels like they have an entire company of Mr. Spocks without a single Doctor McCoy to round things out.
- PhantomGremlin 10y ago
  Mr. Spock would have approached the issue of support more logically and would have thought things through more carefully. It's more like they have an entire company of Sheldon Coopers, without a single Penny or Leonard Hofstadter to round things out.
  - zippy786 10y ago
    Sheldon would have never agreed to release a system with flawed design.

Safety1stClyde 10y ago

Google = rain man.

martinald 10y ago

We got the 100k credit which is nice. Noticed the actual technical side is excellent - vastly better than azure and AWS (AWS especially - I find the whole UI and way it works so clunky). However, despite using our credit to buy gold support when it came to using it I couldn't work their tickets UI at all. Turns out you need to change products from your Google apps support to cloud platform by some tiny link which is not at all obvious. I had to phone up and it was obvious people hit it all the time as the operator instantly knew what the issue was. Also related: azure by default didnt renew your free trial despite an active cc being on file. It just shuts everything off with no obvious warning. What other service does that? Honestly think these companies need to do some proper usability testing on their flows because there is so much clunky weirdness in cloud providers right now.

yladiz 10y ago

This makes me wonder, which do people dislike more: not having access to support in almost any capacity except "help desk tickets" which might not get answered for days, e.g. how Google handles non-free support tiers, or horribly long wait times on phones? My gut says that the former is worse, especially in this case, because you can just turn your phone on speaker and wait for the queue to finish up (or with some systems, have them call you back when the queue is to you). I almost can't believe that Google doesn't have a support line for its account holders for Google Cloud, when you have companies like Paypal and United Airlines, which have many more users (paying or not) and have support lines.

damm 10y ago

I admit to having a gmail.com account ---- I really don't understand how anyone can use Google for production. If you don't pay for support to have a way to actually get a hold of Google... ... Someday you may be in a world of hurt and have absolutely no way to get ahold of anyone. You can post a story and hope and pray that Google might read it and have someone reach out and remedy the situation.

yclept 10y ago

This sounds like a nightmare. One that I can't imagine happening at Rackspace Cloud...

ne01 10y ago

I hate that Google don't give a damn about its users. To them interacting with humans is a waste of time and money... They prefer to automate everything and manage servers, robots and programs instead. The same thing happened to us in 2013 luckily had backup servers at Linode.. took them 2 weeks to solve the problem it was a bug in their billing system. Google gave us $8k credit and I gave them another try and convinced myself that GCE was very young in 2013. Now I'm just worried! It's 2016 and our entire business depends on their platform and their support sucks! Even the $400/month version!! Google, why can't you have support like DigitalOcean and Linode?

hga 10y ago
Google, why can't you have support like DigitalOcean and Linode? Because it's totally not in their DNA? Because GCE and company are relatively low priority offerings? Would you agree these are higher: Ad Words etc., which make the bulk of their money. Search etc. which among other things has you seeing those ads. Android and all their other efforts to keep themselves from losing their access to the users of the above. For a while, Google+ was an anomalous offering they were pushing very hard (with this sort of brain damage with linked accounts that hit hard with their real names policy), but they might try such a stunt again. Compared to the two service providers your mentioned, which only do one thing. Even AWS is I gather rather like that, there's it + all the Amazon selling stuff which is significantly separate. Some of the DNA is shared, but that's not all bad, such as the focus on customers.

76bd 10y ago

The situation of having a project fully shut down without adequate information or process by which to correct it is clearly unacceptable. That said, I was involved in recently migrating to GCP and it has worked wonderfully for us thus far. We run a fairly standard java/MySQL web app, with paid silver support. Questions are answered promptly in our experience. We found compute engine VMs to be faster (and cheaper) than EC2, the developer console easier to use, and better quality documentation/APIs.

manigandham 10y ago

I usually defend GCP as they have better technology and platform architecture but I definitely agree that they have major communication problems. While there are plenty of individuals available through social channels, it does seem to be very informal most of the time which does not inspire confidence as a business looking for a solid foundation.

hehheh 10y ago

Bet you cash money the affected site is or was in a ddos botnet.

fhoffa 10y ago

Posting this 4 days later, what happened next: http://www.businessinsider.com/google-cloud-won-skeptic-after-shutting-site-down-2016-8 http://www.businessinsider.com/google-cloud-won-skeptic-afte... Google then shocked him again, in a good way. Within four hours of tweeting, someone from Google had contacted him and had restored access to his project. Trotter says that, it turns out, he and his team bear some responsibility. They had inadvertently set up a server wrong, exposing a hole, and a hacker was using his company's to conduct a "denial of service attack," which is when hackers overload another website or online service with so much traffic, it shuts down.