30 ms·

Tell HN: Google Cloud suspended our production projects at 1am on Saturday

• martin_a 4y ago
  Sorry, not really sorry. It has been well documented that any of the big tech companies will thankfully flush you and all you've build down the drain whenever they feel like that. You'll also probably never be able to talk to someone (high enough up the ladder) to figure things out. Nevertheless lots of people are building their tech and their lives based on the services provided by these companies, when a maliciously flagged YouTube comment for a linked account can kill _everything_ in seconds. Stop building stuff on their platforms. Now. edit: Yeah, probably not very "helpful" but how many of the "Google killed my X for no reason" posts will we have to see until either regulatory bodies step in (doesn't look like that) or people will just stop using those services. Yeah, it's cheap and it's easy to start with, but so are some drugs, too.
  • mrcartmeneses 4y ago
    Do we see much of this for AWS? Asking for a friend...
    • weird-eye-issue 4y ago
      AWS will email you for months and months about billing issues before they terminate anything
      • darkwater 4y ago
        This. In my previous company I was receiving almost every month some email from automated systems AND humans at AWS because either our finance dept screwed it up or AWS screwed it up but we had like hundred of thousands of $ in missing payments for months, and nothing happened (we always eventually paid AFAIK). Nowadays I think they are removing humans from the loop as much as possible though.
      • unity1001 4y ago
        Some with Digitalocean.
      • laegooose 4y ago
        I confirm this, had this multiple times
      • codegeek 4y ago
        And AWS support is much better.
      • doliveira 4y ago
        Indeed. My company forgot to warn the the credit card had been updated, emails were being sent to the work email of an employee that left... Took 4 months before they started turning off stuff, and even that was quite gradual, just enough to draw attention
    • 2143 4y ago
      With AWS the problems I heard was your bill getting really really high unexpectedly.
      • xmodem 4y ago
        TBF that can happen with any of the big cloud providers.
  • deleted 4y ago
    [deleted]
• thereare5lights 4y ago
  Out of curiosity, what level of support did you have? I'm referring to these https://cloud.google.com/support https://cloud.google.com/support What I hear is that unless you have at least standard support, they treat you like garbage.
  • holoduke 4y ago
    The only entities Google doesn't treat like garbage are advertisers. All others are part of the product
    • prox 4y ago
      Haha, I am an advertiser on Google and trust me, getting hold of someone is like doing a summoning ritual.
      • philipwhiuk 4y ago
        At least most rituals are well documented ;)
  • fruit2020 4y ago
    Thanks for this. I was checking the billing page recently and was surprised by the 2-3k cost for the support plan. Now I see it’s a % of the monthly spend
  • jeroenhd 4y ago
    They have the standard level of support, which means to get your situation resolved you need to post on HN and stay on the front page long enough for Googlers to notice. I swear this website is slowly starting to become a way to contact support for big tech more than anything else.
• PaywallBuster 4y ago
  Not exclusive to GCP. Backblaze has previously locked our account on a saturday. They claim we were putting too much pressure on their cluster. They could've locked the 1! token that was causing 90% of the load, but they locked the whole account which was used by 10 different apps for a veriety of purposes causing unecessary downtime in some production workloads. Also the workload has been quite consistent for days/weeks, so they could've detected it earlier and/or address it in a better way.
  • dman 4y ago
    Whoa - how much traffic were you generating?
  • dchia 4y ago
    If you are paying for the load/pressure, what's the problem?
    • jeroenhd 4y ago
      Backblaze is very cheap and has several "don't overload our system" clauses in their terms and conditions. They're cheap and reliable, that generally comes at the risk of early termination. I don't think you can have a cheap, reliable, unlimited service; pick two at most.
      • jsiepkes 4y ago
        Isn't everything metered with Backblaze (ie. no unlimited)? I think only things such as delete aren't metered. But in order to delete something you must have first uploaded it (which is metered). If everything is metered it seems kind of odd to complain about a lot of activity. Since it's literally the thing you make money with.
      • jquery 4y ago
        Indeed. If a customer tries to backup a petabyte of data on the base plan (because it’s “unlimited”) and they get the boot, they really shouldn’t be that surprised. Wouldn’t really compare it to OP’s situation, where they got an assurance in writing they were in good standing, and didn’t appear to be blocked for violating ToS.
• bilalq 4y ago
  I wonder what it would take to bring a culture change to Google so they can finally start having some empathy for customers and stop the casual cruelty. A new CEO?
  • coffeefirst 4y ago
    It’s probably too big for one strong personality to get it back to the Don’t Be Evil days. Regulation, litigation, or split it into pieces.
    • dzhiurgis 4y ago
      IMO we really need regulation on how support is done by big tech. It should be really hard for them to close accounts like this.
      • llanowarelves 4y ago
        Agreed. This issue seems to be popping up constantly on the front page. Not just Google but many of these companies. I should not be able to get customer service from a random McDonalds franchise over a chicken sandwich easier than a company that we spend thousands, tens, or even hundreds of thousands of dollars with. They do not get to use "We're too big with too many customers" as a defense. They can figure it out.
    • Croftengea 4y ago
      When was the last time any big company was split? AT&T 50 years ago, any newer examples? Nowadays big tech is too influential to allow politicians to make such decisions.
  • emsy 4y ago
    Regulation, full stop. The reason Big tech makes so much profits is because they have no accountability and at their scale, social shaming doesn’t work.
    • rglullis 4y ago
• DarthNebo 4y ago
  Try reaching out to their Developer Advocates on twitter & then maybe with any luck a 'person' can look into this.
• renewiltord 4y ago
  Google Cloud is a weird thing. It's got Google quality: - technically amazing: BigQuery and GKE are great - really half assed as a product: console is somehow slower than AWS despite the latter being the most tech debt of the lot for having invented everything, support half witted, and honestly I had a card declined and even Alibaba Cloud didn't suspend my shit If you have one of their premium support things you'll get faster service but I think I'm not hyper keen on them for general stuff.
  • xmodem 4y ago
    I'm always stunned whenever I have to search for something in our GCP audit logs, because the company that shipped Google search also shipped a log searching and filtering interface that's slower than our self-hosted kibana while searching over multiple orders of magnitude less data.
• danielx 4y ago
  It sucks that they gave you conflicting information that resulted in this but I wouldn't trust having my cloud infrastructure payments be handled using a single credit card. If you have a spend of over $2500/mo you can apply for invoiced billing directly from Google or if you have less than that you can contact a GCP reseller partner to get invoiced by them. Some GCP reseller partners also have a close relationship with Google and the contacts that come with that relationship.
  • booi 4y ago
    $2500 is honestly not that much. Google cloud dropped the ball on this and shouldn’t need to be handled with special relationships
    • Spare_account 4y ago
      All of the relationships I have with suppliers are handled by an account manager at the supplier who I can call to get things sorted out. This is normal. Why should a business relationship with Google (or any other cloud service) be any different? I don't use GCP but if I was going to start, I would ensure that I have a person to call or its a non-starter.
      • remus 4y ago
        > Why should a business relationship with Google (or any other cloud service) be any different? Depends on the size of the account surely? Maybe there's some support person to talk to, but there's plenty of services I subscribe to fo $10-100/month and I dont have an account manager for any of them.
        • mschuster91 4y ago
          Yeah but for almost all of this kind of service you have some sort of phone, email or 24/7 helpdesk system. Google however is infamous for not providing support at all on all their services, even if your complete digital identity got killed off because some AI can't recognize that the "CSAM" you sent was in fact communication with your child's doctor [1]. [1] https://www.heise.de/news/Nacktscanner-Unbedachte-Fotos-vom-Kind-fuer-den-Arzt-Google-Dienste-gesperrt-7238900.html https://www.heise.de/news/Nacktscanner-Unbedachte-Fotos-vom-...
• murrayb 4y ago
  I understand the cost and complexity of doing so but I think that production loads should be engineered across two or more service providers. That way any incident on a given provider is recoverable without that providers cooperation and you can mitigate against Oracle style lock-in.
  • carschno 4y ago
    I somewhat agree in principle. At the same time, the (implied) promise of the big providers like AWS and GCP is to provide everything needed for a reliable service while saving the time for setting it all up yourself. If you need multiple platform providers with fallback logic etc., it is hard to see a scenario in which setting up your own infrastructure would not be more effective. Perhaps the latter is the more practical solution indeed, though.
  • snotrockets 4y ago
    If you think that production loads should be designed as multi-cloud as redundancy, then you don't understand the cost of complexity and doing it.
• jamespwilliams 4y ago
  The same is true of DigitalOcean, fwiw. Shutting down machines used in production with no notice, outside working hours. I don’t know why these companies don’t care - they’re massively shooting themselves in their feet. Incidents like this turn customers off for life.
  • mro_name 4y ago
    they are big and you are small. It's like walking over ants.
    • arvindamirtaa 4y ago
      Ants pay the bill!
      • tinus_hn 4y ago
        Ants that don’t cause trouble pay the bill. Ants that cause trouble are a net loss. Customer service representation is one of the biggest expenses these companies have. That’s why they go to great lengths to make sure you can’t contact them and why they just close your account with no recourse as soon as you start causing trouble.
        • pid-1 4y ago
          Poor AWS must be close to bankruptcy then. When my account started spending 300-400 USD monthly, I was contacted by an account manager, who was at my office for a meeting two weeks later.
        • arvindamirtaa 4y ago
          Every time they do this, a few hundred ants decide, they may very well be next and move out. Supporting them through whatever they need may not be quite the net loss you claim it is.
  • Croftengea 4y ago
    Care to share more info please? Was it because card didn't go through? I've got some important stuff there.
• marban 4y ago
  never use google cloud systems for production is a strong statement based on one random incident. CGP customer since launch 2008 (when it was App Engine) and spend around $3k/m on it (Also using AWS and Heroku). Never had a single issue but CS responses vary a lot depending on how you funnel requests.
  • lovelearning 4y ago
    I don't think dismissing such incidents as random incidents is a good idea. Although crimes and thefts are rare, all developed societies invest millions into law enforcement and justice systems to measure and mitigate them. Although device and equipment failures are rare, both private and government systems exist to address them fairly. It seems like it's only in the tech industry that statistical insignificance of failures is assumed and then wielded as justification to avoid even minimal levels of accountability.
    • xmodem 4y ago
      > It seems like it's only in the tech industry that statistical insignificance of failures is assumed and then wielded as justification to avoid even minimal levels of accountability. Stealing this, it's far more broadly applicable than just to this sort of billing issue.
    • mkl95 4y ago
      > I don't think dismissing such incidents as random incidents is a good idea. If I had €1 for every time a politician has dismissed some incident as a random incident I could probably buy a high-end MacBook.
• atemerev 4y ago
  Hell is other people. Cloud is other people’s computers.
• Animats 4y ago
  Does business interruption insurance cover Google, or is that under the "acts of God" exception?
• KingOfCoders 4y ago
  Also: If you're on AWS/GCP and only backup to S3/GCS, you're doing it wrong. All your backups to setup a new system (even if it takes time) need to be with a different provider.
  • deelowe 4y ago
    Those are different providers. Why would you need a 3rd?
    • paragraft 4y ago
      Think they meant "AWS or GCP and only backup to S3 or GCS respectively"
    • KhalPanda 4y ago
      I think they’re saying only backing up _data_ is not enough (S3, GCS). You need to be able to roll out infrastructure elsewhere to, in order to have somewhere to restore said data.
  • londons_explore 4y ago
    Backups to another provider are very costly for most users due to egress bandwidth pricing. Most backup systems can't do efficient incremental diffs either - so you end up sending your companies entire database every 12/24 hours, burning through a lot of bandwidth.
    • quickthrower2 4y ago
      Would have through incremental backup (replication) would be a core feature of most databases.
      • londons_explore 4y ago
        I think the problem is that replication alone isn't enough. You also need to support snapshots for backups because you don't want to replay years worth of replication logs to do a restore. And when you support snapshots in your backup software, it's very tempting to then drop support for replication... Replication logs also have another problem - since they include every data change, certain types of database updates, for example a large field that changes hundreds of times per second, can work out far less bandwidth to just snapshot periodically. Overall, all this stuff is a lot of complexity most businesses don't want to have. They just want to have a simple backup solution and get back to building their product. And that usually involves using some managed database where backups are just an option in the control panel. Or a self-managed database where backups are disk snapshots. Or a script that copies a database dump to S3 every day. But it hardly ever involves streaming change logs...
• paganel 4y ago
  The law should come hard on this sort of shenanigans, but hitting on Big Tech has already become a political issue in the US, Europe is, in the end, too afraid to do it (bar some relatively minor extra taxes that Big Tech is all to happy to pay as long as it's not made legally liable in other ways), and here we are, with crazy stories like this one (one of many).
  • raverbashing 4y ago
    The law can only come if people sue And yes I think people should engage lawyers in cases like this. Even in small cases court if it's the case
    • eru 4y ago
      > The law can only come if people sue That's not true in the long run. Legislators can make arbitrary laws that ban or prescribe arbitrary things. See eg how there are special laws in Spain and Australia that (try to) force specific behaviour from Google News. Behaviour that you can't override with a contract.
  • cuteboy19 4y ago
    Counterpoint: GDPR
  • sofixa 4y ago
    > Europe is, in the end, too afraid to do it Take a look at the Digital Markets and Digital Services Acts on top of the already active GDPR.
    • yoro46 4y ago
      Well, the GDPR isn't there to fine companies for doing the wrong thing. A company's first violation rarely results in fines or disciplinary action. In any case, I do agree that the EU seems to move quicker and act stricter than other regions in regulating big tech companies.
      • 4y ago
• silisili 4y ago
  Seeing all these horror stories, I can't help but wonder when people are going to switch back to on prem, or at least some hybrid. It's pretty apparent we can't trust cloud providers...
  • reeeeee 4y ago
    I'm not so sure. Aren't we having a bit of negativity bias here, as we mostly hear about the negative events while there are probably countless successful integrations with cloud providers.
    • oars 4y ago
      Exactly. This is truly negativity bias. Look at all the banks shifting their infra to the cloud. Haven't heard anything bad happen from that. And their uptime and business requirements are as stringent as anyone else, even further scrutinized by regulators and auditors.
    • dustinmoris 4y ago
      Yes, if you'd compare all horror stories of people having huge downtimes because their on-prem system collapsed vs. people having an issue with their cloud provider then we'd very quickly realise again why the cloud has taken us by storm.
  • fxtentacle 4y ago
    I'd guess that a large percentage of SMBs are running on-prem, so you don't hear about switching back because they avoided the cloud in the first place.
  • cbg0 4y ago
    Expect to have issues with colo as well, unless you own the building.
  • 12ian34 4y ago
    I think it's a vocal minority. The vast majority of GCS customers won't have this problem. These kind of posts typically do very well on HN.
• euph0ria 4y ago
  The company I work for has forbidden the use of Google and their services (if possible) such as GCP etc. due to how they treat their customers. In particular, there never seems to be any human being that you can talk to and find out what you need to do to fix the issue. We do not want the same to occur to our production systems if there is an overflow from a Play Store ban to GCP etc. The business risk is too high when you rely on Google's services. Some examples: Suspended GCP even when everything in order - https://news.ycombinator.com/item?id=32547912 https://news.ycombinator.com/item?id=32547912 Ban app even when competitors are similar - https://www.vice.com/en/article/z3vdpj/google-play-store-ban https://www.vice.com/en/article/z3vdpj/google-play-store-ban... Play Association ban - https://old.reddit.com/r/androiddev/comments/ts6jfg/google_has_terminated_our_developer_account_says/ https://old.reddit.com/r/androiddev/comments/ts6jfg/google_h... Ban due to wrong wording - https://twitter.com/hermaritz/status/1371383715381805061 https://twitter.com/hermaritz/status/1371383715381805061 Terraria banned - https://twitter.com/Demilogic/status/1358661840402845696 https://twitter.com/Demilogic/status/1358661840402845696 New project banned - https://medium.com/@amton15127/why-you-should-not-use-fireba https://medium.com/@amton15127/why-you-should-not-use-fireba... Google bans company - https://www.reddit.com/r/tifu/comments/8kvias/tifu_by_gettin https://www.reddit.com/r/tifu/comments/8kvias/tifu_by_gettin... Google bans mail - https://talkingpointsmemo.com/edblog/a-serf-on-googles-farm https://talkingpointsmemo.com/edblog/a-serf-on-googles-farm Ban app for communicating changes during covid - https://news.ycombinator.com/item?id=23221447 https://news.ycombinator.com/item?id=23221447 Adwords ban - https://news.ycombinator.com/item?id=23224791 https://news.ycombinator.com/item?id=23224791 Serverpunch bad support - https://news.ycombinator.com/item?id=17431609 https://news.ycombinator.com/item?id=17431609 Delete app - https://news.ycombinator.com/item?id=20826618 https://news.ycombinator.com/item?id=20826618 Google bans game with pandemic - https://news.ycombinator.com/item?id=23229073 https://news.ycombinator.com/item?id=23229073 Google bans dev with no recourse - https://news.ycombinator.com/item?id=15197357 https://news.ycombinator.com/item?id=15197357
  • chucky123 4y ago
    These are the ones that got the limelight. Multiply this list x100 to get the true number of fuckups.
  • 4y ago
• oivoodoo_ 4y ago
  I don't use google ads now because of suspended account for the last 2 years. No luck to get the response about the issue from their support. I would say never use google products. They are not people oriented. They charge for the digital products without having reliable communication. if you want to grow it's definitely better to keep using the other services. I don't recommend anything else but have you heard any cases about digitalocean or amazon or fly or other platforms? also it's possible to write to support and get information about any kind of the issues that you are getting for any kind of the reasons.
  • jsemrau 4y ago
    My ad campaign was suspended every three months because I had an exclamation mark in it. After explaining it for the third time I gave up on it and started to build for the post-advertising world.
    • candiddevmike 4y ago
      What does "build for the post-advertising world" entail? I'm extremely interested in alternatives to the Google Ads protection racket. I've been focusing on outstanding customer service/product to drive word of mouth and it seems to have a better ROI.
  • deleted 4y ago
    [deleted]
  • deleted 4y ago
    [deleted]
  • lubien 4y ago
    Fly.io employee here. Account suspensions are manual. If you have billing issues at most you will be limited on what you can do (deploy, scale...).
    • oivoodoo_ 4y ago
      exactly. I heard only the good words about fly actually :) You don't break things like google services with their clients.
• lizardactivist 4y ago
  Using Google services for anything important is like tossing it into the wind and hope for the best.
• mkl95 4y ago
  Cloud computing platforms are a bit of a double edged sword. I find it kind of ironic that big companies are moving their stuff to those services but they often have to rely on some consultancy firm that can deal with these problems. Congratulations, you had a problem and now you have two.
  • ilc 4y ago
    That's a feature not a bug. They went from: (Remember, companies are people!) We have to make choices about big expensive things. - 0 people to blame besides them. Vendors / Consultants make choices about bigger and more expensive things for them - 1 person to blame besides them. Consultants + "The Cloud" to blame - 2 people to blame besides them. It is a feature to have many people to blame, in these type of environments.
• invalidname 4y ago
  This calls for a "me too" moment where everyone shares their Google is terrible and I'll never use them again. Here are a couple of my favorites from our company: https://medium.com/hackernoon/why-and-how-we-left-app-engine-after-it-almost-destroyed-us-40ac2fc0b1a8 https://medium.com/hackernoon/why-and-how-we-left-app-engine... https://dev.to/codenameone/google-play-kafkaesque-experience-mp3 https://dev.to/codenameone/google-play-kafkaesque-experience... There's this story from a couple of days ago which is shocking: https://news.ycombinator.com/item?id=32538805 https://news.ycombinator.com/item?id=32538805 Google is a terrible, dystopian company.
  • ramraj07 4y ago
    The most interesting part is how all this awfulness seems to stem purely from apathy.
  • RedShift1 4y ago
    Warning for that Medium link, that background color burned my eyes...
    • armchairguy 4y ago
    • deleted 4y ago
      [deleted]
    • cabbagesauce 4y ago
      Underrated commend.
  • deleted 4y ago
    [deleted]
• andrewinardeer 4y ago
  I know I'm going to come off as a Google apologist, however, I'm leaning towards there being more to the story than what you are saying. The previous month's billing didn't 'went through' and you are unable to zero in on whether this is a Google issue or a you issue. I get that it is easier to complain on the internet and often cathartic do so as a coping mechanism. Having an account suspended because of a billing issue is not exclusive to Google. Have you moved your projects off GCP now as voting with your wallet would be the sensible thing to do. Until you encounter a billing issue with AWS and move again to Azure and then to DO.
  • moonchrome 4y ago
    >Having an account suspended because of a billing issue is not exclusive to Google. I'm sorry but : >Despite reassurances "your account will not be suspended" while communicating with billing support, all the projects were suspended at 1am on Saturday. If he contacted someone at customer support, after making sure the mistake was corrected (on whatever side it was) and they assure him that the account will not be suspended - what the fuck is he supposed to do about it beyond that ? Imagine having your business/job fucked up over an incompetent customer support system.
    • __d 4y ago
      GCP is a SPoF. If you care about your availability, avoid single points of failure. Then it doesn't matter if what support tells you turns out to be not true. How you do that can vary, but ... billing issues are always a risk. Having two cards, with two different banks, and two different names, is a pretty basic precaution to take. Having multiple cloud providers closes out a bunch more existential risks. DNS is the one unavoidable SPoF for most Internet businesses: you cannot have a "backup registrar". The best you can do here is pay for a quality service provider: someone who will call you if there's a problem, before they cut you off. That's going to cost a lot more than GoDaddy, btw.
  • londons_explore 4y ago
    No human working at Google will do an account suspension on a Saturday. So it must have been automation. And there is a very good chance that automation simply hadn't been blocked by the support agent which had been giving reassurances.
• devmor 4y ago
  Google has the worst support, bar none, of any infrastructure company I've dealt with in my 13 years in the industry. IF you can get ahold of someone... and IF they comprehend your issue... and IF they know how to solve it... and IF they can communicate with other departments involved... and IF they remember to respond to you... and IF their internal documentation is up to date... then you MIGHT stand a chance of getting your problem solved. It's at the point where you will get better support for Google's own services from companies that resell them, because they have on-call internal contacts at Google and actually care about their customers.
• pontifier 4y ago
  Murfie died because of Google cloud costs. It's a bit of a long story, but from what I was able to piece together, a Google cloud service provider recommended some changes to the way the storage was done to save money. Mostly, because of those changes they got hit by a surprise $30,000 bill for data egress. It was basically the last straw for a company that wasn't profitable due mostly to cloud costs. It killed the company. That's why I had to step in to try to save them.
  • unity1001 4y ago
    > data egress Google, AWS are still charging people mid-2000s bandwidth prices. Its outrageous. Digitalocean's egress price is 1/10th of Google's cheaper egress pricing. Same with Hetzner. And with Hetzner dedicated servers? Its unmetered.
    • mrtksn 4y ago
      If I recall correctly, there was a story on HN about Hetzner threatening to shut down an "unmetered" server for using too much bandwidth. here it is: https://news.ycombinator.com/item?id=32242987 https://news.ycombinator.com/item?id=32242987
      • unity1001 4y ago
        From the discussion: > - The link says "my friend got this", so even the person who authored that post isn't claiming to have received this notice. This is a second hand report, at best. For all we know, their friend received this notice years ago, and mentioned it to the OP, who didn't realize it was old news. There have been people confirming in recent HN discussions that they were using full 1gbps of their dedicated box for a while since Hetzner declared their true-unmetered policy a year or so ago.
        • nijave 4y ago
          There was a long discussion with multiple people getting emails on LowEndBox forums using around 250TB/month
• ggm 4y ago
  Is there no model of service where Google can't make real human support possible at scale for profit?
  • nikanj 4y ago
    Probably, but there is more profit in not having real human support. The savings are massive, and the cost is just a few dropped customers here and there
• herpderperator 4y ago
  Were you able to get back online? Have they responded since the suspension? It's early Monday morning now.
• intellectronica 4y ago
  Generally, all cloud providers have two ways of engaging with and billing customers. One is using a credit card, which is very easy for getting started. The other one is having an established relationship with a sales / account team and setting up billing by invoice. Smaller companies sometimes don't understand that and stick with credit card billing for too long. Once you move to production and have your own customers that you can't let down, you should really make the extra effort and move to invoice billing (with all the upfront overhead that entails).
  • baobabKoodaa 4y ago
    Ah yes, it was the customer's fault. Stupid customer using method recommended by service provider.
    • intellectronica 4y ago
      • baobabKoodaa 4y ago
        For the record, I was being sarcastic and do not actually believe it was the customer's fault. I believe this is an abhorrent way of running business by Google.
        • rb666 4y ago
          And then you get hit by the car, even though the light was green. Doesn't help you, does it?
          • baobabKoodaa 4y ago
            More like: the driver stops and gestures me to walk over the road. As I step in front of the car, they suddenly jump on the gaspedal and drive over me.
          • 4y ago
• hazbo 4y ago
  A few years ago I was working at a startup and we had just started moving some stuff over from AWS to GCP. Things were going pretty good with GCP until one day they pulled the plug on everything with no explanation. It turns out that our company credit card details had been fraudulently used without our knowledge. The criminal had decided to use the card to buy Google Ad Words or something like that - this is the same card we use on our GCP billing account. Anyway Google just took prod down with no notice on grounds of fraud. Could happen to anyone.
  • belter 4y ago
    Sounds like those Whiteboard interviews need to incorporate Graph Deterministic Algorithms for Highly-Dynamic Networks...
    • gumby 4y ago
      Need another Leetcode section on heartlessly screwing your SaaS customers.
      • HeavyStorm 4y ago
        Wasn't there an algorithm for that?
        • doctor_eval 4y ago
          It sounds like they need a whiteboard session on something like “policy engine to decouple billing status from service status”.
        • hn_version_0023 4y ago
          Yes, It’s called the Google hiring process.
        • samstave 4y ago
          https://i.imgur.com/ctKRD80.png https://i.imgur.com/ctKRD80.png
• noisy_boy 4y ago
  I keep hearing about this "automation" and sure, I'm probably too dumb to understand FAANG-level automation but aren't there switches that the automated workflows are expected to check against, atleast before running high-impact actions are like account suspension? Switches that can be pushed by the customer service groups with appropriate entitlement? I have written components with such overrides (with proper auditing to ensure review) and that worked pretty well as a safety-valve. Surely a bunch of PHDs with research papers can't be all wrong so what am I missing here?
• mattbee 4y ago
  This is why my old hosting company changed its auto disconnection code to work at 11am on a Monday morning, never at a weekend. That way it's the bill-payers that notice and get in touch, not the poor tech on the out-of-hours rota. Also, it only firewalled the customer's public IPs from the Internet - it didn't stop servers. That way when the account is settled there was often nothing for the techs to do. Also it was only ever done after the bank reconciliation was marked as complete, just in case... We relied on goodwill and recommendation of engineers for our growth, so were keen not to make their lives harder for someone else's bureaucratic (or cynical) mistake.
  • czbond 4y ago
    The method you describe seems like the proper way to achieve this thorny issue.
  • designium 4y ago
    Which hosting company are you referring to?
    • softawre 4y ago
      From his profile: > Cofounder and former MD of Bytemark, a British hosting company.
  • kyriakos 4y ago
    That's how hetzner handles late payments and only does this after 2 warning emails.
    • Gibbon1 4y ago
      In the normal BnB world anyone with a account gets a phone call before they get cut off.
    • bingo-bongo 4y ago
      That may be true for billing, but Hetzner killed all access to my dedicated server on a weekday afternoon and when I reached out to them within 30 min, the response was “it was done by the networking department, I can’t do anything about it, but they’ll look into it tomorrow..” .. :| wtf?! not okay Hetzner! (My “mistake” was a interface mis-configuration, nothing abusive or otherwise problematic)
• Linda703 4y ago
• nikanj 4y ago
  There's a TL;DR like this at least once per month on HN. How do people keep making the same mistake? Google doesn't care about you. Escalate enough, and they will also block your personal account, deleting all your emails+photos in the process. Friends don't let friends run production software on Google services.
• anonymous344 4y ago
  This is exactly my nightmare. I have several google cloud apps, all stupidly under one account that I use for other things also than just to gcp. Maybe it's time to start to migrate away from gcp, because how can you even reach them ?? and how on earth any provider cuts anything off during the weekend?
• tastemykungfu 4y ago
  Really sad Google just continues to do this BS. At the company I work for, we are seriously evaluating moving off of AWS (about ~$10k/mo bill). This is not going to fly.
  • unity1001 4y ago
    Try digitalocean. Their offerings are competitive, with a fraction of the price. With that kind of low cost, you can have failovers elsewhere. Like Linode, or even Hetzner.
• julianwachholz 4y ago
  > "We're fully redundant across AZs, regions, and even cloud providers!" crows the engineer with a single corporate credit card backing the entire house of cards. https://twitter.com/QuinnyPig/status/1288275701389389825?t=FVRQlZPYIz9yn97n0XASBg&s=19 https://twitter.com/QuinnyPig/status/1288275701389389825?t=F...
• buzzwords 4y ago
  Reading a lot of comments here makes me think that GCP has built up a very bad reputation for customer support. Does anyone know if they are responded to this?
• niuzeta 4y ago
  This sort of stuff genuinely scares me to try Google cloud, ever. Even if I were to try something on the aside, it's scary how much I rely on Google services, and how monumentally screwed I would be, if my Google account would get disabled somehow without any recourse.
• gorbachev 4y ago
  Tangential question. Has anyone compiled a list of these Google customer service nightmares anywhere? I would love to give a Google sales person a link to such as an explanation why I will never buy services from them.
  • shortlived 4y ago
    There is a rough start here https://news.ycombinator.com/item?id=32548441 https://news.ycombinator.com/item?id=32548441
• mberning 4y ago
  This would be a situation where government should step in with some regulatory authority. The public utilities in general cannot cut off your electric or gas service for a simple billing issue. And in some cases, such as during a harsh winter, than cannot disconnect you at all. We need something like this for cloud providers.
• tomxor 4y ago
  > They were "investigating" the issue and assured the project will not be suspended. In my experience, and from everything I have read of others, Google's customer support is mostly a facade - Google will have given them a few buttons to press but they have absolutely no autonomy or authority, they are disconnected from Google and when things go wrong they cannot help you no matter how much they promise.
• trasz 4y ago
  That's another case of reliability of cloud solutions compared to bare metal.
• mrjin 4y ago
  This reminds someone's assertion on cloud long time ago: "Cloud is just someone else's computer".
• alphaomegacode 4y ago
  For a company that operates at such massive scale for their own services, Google has some glaring oversights when it comes to enterprise services. I have a friend who volunteers as a webmaster for a charity org and they kept getting notices about their impending shut down of G Suite and the need to move to Google Workspace. The problem seemed to be a credit card and yet, when my friend was showing me on his laptop, there was no place for them to enter credit card information. They ended up losing all their email and Drive & Docs access for days and couldn't even get an email or phone address - paid or not - to address the issue. After getting burned at a mid-sized business I used to work at with Angular 1.x and Google Glass, I learned that Google apparently sees business services as experiments rather than long-term investments.
  • christophilus 4y ago
    Was Angular 1.x an official Google thing? I always thought it was just a Google employee’s project.
    • skybrian 4y ago
      I doubt it was ever an official Google product because open source packages generally aren't. But there was a team at Google working on it when Angular 1.x was big.
    • alphaomegacode 4y ago
      I'm not sure if it was ever an official thing but I remember accessing their pages and speaking to their reps at conferences regarding Angular. For example, if you look at the Angular 1.2 docs, look at the bottom of the page here and you'll see Google included their copyright on those docs https://code.angularjs.org/1.2.27/docs/guide/ie https://code.angularjs.org/1.2.27/docs/guide/ie
• djklanac 4y ago
  Googler here. I'd like to help. DM me https://twitter.com/david_klanac https://twitter.com/david_klanac
  • lupire 4y ago
    Are you someone with authority over Billing, or a nice but powerless employee in a different department?
    • djklanac 4y ago
      It doesn't take authority to effect change. You'd be surprised at what you can accomplish once you figure out how who owns the various fragments of the solution. OP – My offer to help still stands. I'm sorry to hear about your bad experience. Sorry doesn't fix it. I realize that. But I want to make it right.
• pcdoodle 4y ago
  The leader in search folks. This level of incompetence. I moved all my domains from google when they starting picking political sides. Even their search product is becoming useless.
• T3RMINATED 4y ago
• latexr 4y ago
  > TLDR; never use google cloud systems for production. By this point the TLDR should be “don’t use a Google account”. It doesn’t matter if it’s business or personal, the horror stories of suspension without recourse are the same.
• rattray 4y ago
  Out of curiosity, how long was the suspension active, how quickly was it fixed once you notified them, and how large is your account (very roughly)?
• HeavyStorm 4y ago
  There are other cloud providers out there that are be trusted. Rest assured that these events are not common.
• throw_m239339 4y ago
  I'll repeat it again and again and again, Google is not a trustworthy partner and scammed me a lot of money out of my adsense gains, only to slap me in the face by sending a 50 bucks voucher to my physical address to "buy ads". And it was all 15 years ago. Google are a bunch of thieves in my book and nobody should ever trust Google Cloud. Since then I worked in many big corporations and have had responsibilities and everytime I could weight against using Google cloud, I did. They must have lost millions because of me, all that because they kicked me out of adsense for bullshit reasons without the possibility of appeal. Google can go to hell.
• hd99 4y ago
• czbond 4y ago
  On the other side of the coin, AWS communicated they were shutting down EC2-Classic. With years of notice, and then on the cut off date, is still slowly winding it down. :) Love AWS
• tlogan 4y ago
  I was really under impression that Thomas Currian will make Google cloud a little more reliable. Less cool (even worse tech ) but more reliable: good documentation, reliable servers, good management console, and good support (think Rackspace in olden days). I guess that is not happening. AWS is good enough and Azure is a mess but at least support personal is trained.
• hd99 4y ago
• mk89 4y ago
  So as of today, mid 2022 ... there is still again ONE only reliable cloud? How can a company fight the so called "vendor lock-in" without opening their own data center? The only alternative I see is just really going against the flow - use simpler hosting services, etc.... but who does that nowadays?
• BrainVirus 4y ago
  What most engineers fail to understand is that when you calculate true reliability of a service you should factor in all such incidents. Most people are somehow convinced that outages due to billing issues "don't count". Service outages due to overzealous automated enforcement of some security rule do not count. Being banned from a service because of your politics doesn't count. The end result, however, is the same - your service is down.
• rufius 4y ago
  Google Cloud is the Clown Shoes offering amongst major providers. I’d much rather bet my business on DigitalOcean or Vultr before I ever touched Google Cloud.
• luka-birsa 4y ago
  A suggestion for engineers that lack business experience. As soon as your project grows beyond the non-hobby scale and starts making real money, invest some time to reach out to company providing the services that keep it alive and find an account manager that will take care of you. We're a Google customer and of course we had our fair share of issues (btw: it's the same with Azure, based on our experience), and we always escalate to our account manager. That usually starts turning the wheels much faster. Also, do not be afraid to ask your account manager for additional discounts. Everybody in sales has some wiggle room to get you a better deal. With Google their account manager actually suggested that we work through a reseller and we're getting % off the list prices. Dealing with Google, Amazon, etc.. is not like dealing with code. It's not only transactional and you need to invest some time in the relationship even if you're the customer.
  • jnsaff2 4y ago
    I found the Azure reps to be completely useless for any of the actual problems. I worked for couple of really big enterprises, one of which had such a high commitment that Azure sent two engineers to sit with the teams working on their cloud. Highest level Enterprise support .. and yet for actual non-obvious problems it took them a month to reply with: can't help you with this, you must be doing it wrong. This was techsupport tho, I'm sure the billing for multi-million commitments was just fine and dandy including wining and dining. One other company was a big AWS user (and some Office365 so there were Azure pitches too) and GCP tried to get in with the business as well. When we went to their offices the level of patronizing smug that came from their reps was astonishing, so off-putting that I refused all further interaction with them as I wanted to barf.
    • dharmab 4y ago
      I worked for one of Azure's biggest customers and our account management teams were fantastic. A few individuals were some of my favorite people I've worked with. That's where all the talent in the Azure support org goes- promoted to the biggest contracts.
      • 4y ago
• jrm4 4y ago
  Getting ready to start reaching my IT for Professionals class this semester. I've basically shifted to making the final project something involving "a machine on which you (genuinely and completely) have root." This would be why.
• jwaldrip 4y ago
  In terms of customer experience, GCP is amazing and its why we chose Google. When you work with an account manager, the experience is far better than any other service. If you only work with support, you aren't going to have a good advocate for your account.
• markdog12 4y ago
  We've also received the same threatening emails, even though it was an error on their end, and everything was paid. We were also assured it was an issue on their end, and to ignore the emails. This is wildly irresponsible stuff.
• upupandup 4y ago
  Not only that they stringed me along for the free credit thing and I realized that I started to incur expenses for running some other instances. When I protested they assured me I billing will handle it. Except my account manager just ghosted me and I found out that if I do a chargeback all of my google services could be shut down, not just my GCP related billing. I've never returned and have been with AWS since. Nobody in their right mind uses GCP and the only good thing they had was Firebase but already Supabase has pretty much rendered that useless. Fck google cloud and I say that with confidence. If you are one of those shops that require engineers to use both GCP and AWS, start choosing, because new and old are on AWS and will remain on AWS.
• saje_585 4y ago
  Have you try contacting Google Cloud support teams? What did they say? I found this path https://cloud.google.com/support/billing https://cloud.google.com/support/billing
• twistedpair 4y ago
  Running your whole company cloud off a credit card is OK pre-seed round, but then you should get a proper business account setup w/ ACH payments/invocing. Better still, after you grow a bit, use a GCP reseller (SADA and DoIT are great) or a usage commitment with your account manager for a nice discount. I've never heard of one of those accounts getting turned off in the middle of the night. Sure, it sucks to hear of account lockouts like this, but productionizing your finances is part of the startup lifecycle.
• H8crilA 4y ago
  This reminds me about a criticism of quoting 99.999...% reliability of storage systems: the number of 9s kind of doesn't matter because you're more likely to experience an administrative problem, such as an unnoticed payments issue :)
• midislack 4y ago
  It's not just Google cloud that's the problem here. It's using somebody else's computer. Now that you've learned your lesson, you need to run out and buy a nice Threadripper and I guarantee it will run your whole workload eith enough juice to spare so when you're pulling 1 million paying visitors per day or whatever you'll still be able to run your whole operation on one machine. You may have to fire your "responsive" front end people but they'll kill your whole business sooner or later anyhow.
• anotherrandom 4y ago
  Running a hobby project on Oracle Cloud (that or a provider like DigitalOcean/Linode are the only ways it is profitable due to bandwith expenses) that I may try to turn into a business eventually and worried about that same scenario happening.
• salmonlogs 4y ago
  I used to work at Google Cloud, I left about 2 years ago so my advice might be slightly outdated so hopefully it helps someone. I highly recommend anyone running production workloads to get in touch with sales and meet your account manager, and customer engineer. You also need to swap to invoice billing and sign the offline payment terms. Credit card payments leave you in a MUCH higher risk category at higher risk of suspension. You can ask them to put a flag on your accounts so that you won't get suspended in the event of billing issues. Iirc it gives you something like 180 days and blocks any of the automated systems. The downside is you could get a crazy huge bill if you have an account compromised or some rogue IaC code. But you can manage that with quotas mostly. And with some good negotiation you can get a rogue bill written off. YMMV.
• kyrra 4y ago
  Googler, opinions are my own. I know nothing about this case, but I have a few thoughts about this story. 1) The "full story" is actually rather sparse on exact details. The author quoted just a few words of "Your Projects are at risk of suspension", but doesn't provide more details about what was actually in the email. Did something else potentially cause this shutoff, not just billing? (I agree, based on his wording that this likely was a billing issue, but it would be good to know the full story). 2) The OP has not replied to any comments on here. 3) If the OP is reading this, they should go through their front-line support people at GCP saying this story is about you, so they can help you look into it. (it'll likely get the PR people to know who you are). If there was really a screw up where you paid and were still shut off for billing non-payment, I bet that would lead to a larger postmortem internally.
  • chucky123 4y ago
    Some really good points in here.
• f4ncyp4ntz 4y ago
• rochak 4y ago
  OP’s lack of replies to countless comments here is making it hard to trust the credibility of this post.
• splix 4y ago
  Same happened to us the last Thursday, but in our case they suspended our GCP account without a prior notification. Just learned about this from our external monitoring. Turns out they wanted to check my IDs, and it's after I had been their customer for 14 years. And it took them 5 days to verify that ID. An hour ago they activated it back. I've been running a few projects on GCP, with active customers, which I lost now. Those 5 days of outrage is basically a catastrophe for us. And no apology was given by Google. There was barely any communication from their side. Never use GCP for production.