14 ms·

Our production servers were suspended by Google Cloud

209 points by ayewo 4y ago

nova22033 4y ago
If your servers were suspended, how is your website functional? https://console.onvoard.com/register/ https://console.onvoard.com/register/
- greenyoda 4y ago
  In the second paragraph of the article: "I managed to get our servers back online few hours ago with a workaround by adding a new billing account."
steponlego 4y ago
A nice big Threadripper in the corner of your office is all you need, I can almost certainly guarantee it. There are very very few businesses with less than 1000 employees that can't be run on one modern high end server.
- leesalminen 4y ago
  What happens when the power goes out? Internet goes out? Someone spills a cup of coffee on the server? Landlord needs to do some work and drywall dust flies around everywhere? It's not that most CRUD apps could run on a single machine, it's that they probably shouldn't. Customers want redundancy, automated failover, etc. Datacenters are much better at offering these things than your office park.
  - re-thc 4y ago
    Interestingly enough the chances of that happening are actually less than an AWS region outage :).
  - sourcecodeplz 4y ago
    You invest in 1-2 UPSes and a 4/5G mobile modem data plan that kicks in when needed. If you really want to be sure, get a diesel generator also. Will take some time to setup but it is worth it. - No one can see your code - No one can shut you down
  - CyberDildonics 4y ago
    Someone spills a cup of coffee on the server? Are people spilling coffee on your servers at work?
  - steponlego 4y ago
    You can do automated failover to Raspberry Pi, it'll hold while the main server is cared for. If you go cloud, you deserve what you get. And what you get is what you get, because it's not your computer, and it's not your data.
  - margorczynski 4y ago
    Using a colo where each instance has a backup in another colo would probably be sufficient for most.
- _cenw 4y ago
  This always assumes these applications are built by competent people who value efficiency. The job market is full of people that think they need high scalability but then underestimate the cost of an HTTP request or the memory footprint of keeping an entire dataset in memory. It has happened several times in my career building platforms that I have to double check that packet capture I just took from that single Kubernetes node is accurate (my favorite is 22k DNS queries, all A IN s3.amazonaws.com, in a 0.2s span). Similar things happen with memory and CPU.
- cantSpellSober 4y ago
  You lose the benefit of a CDN though (caching content on a server closer to where each individual user is accessing it)
mjburgess 4y ago
In many ways, Google continues to expose the failure of automation. The 2010s dream of "megacorp serving everyone with centralised programming" ever more exposed as an unethical pipedream.
- michaelteter 4y ago
  > failure of automation Many of the Google horror stories are less a failure of automation and more just a failure of well designed business processes. This same event could/would have occurred if it were humans sending the "additional information required" emails and also following a protocol of "shut them down in 10 days if they don't provide". The fact that this process was (most likely) automated is not the issue. Honestly, as an outsider, it would appear that Google is a tech company that isn't actually very good at business. They do some smart tech things with their huge number of smart workers, but the business side seems to make a lot of very poor decisions (or simply doesn't even consider some part of the business or doesn't prioritize business processes well).
  - eastbound 4y ago
    Every time I’m told it was an automatic system at Google, I doubt it. I can’t imagine all those heavy decisions aren’t actually reviewed by humans, who validate the automatic suggestions. Would anyone imagine that the decision to cut off a customer entirely, is not checked by a clerk? by a lawyer? The defense “It was automatic” doesn’t even pass the basic sanity test.
    - shadowgovt 4y ago
      At Google's scale, cutting one paying customer isn't a heavy decision. That's what makes doing business with them so weird; it may be mission-critical to your business, but to them it's a Tuesday.
  - ethbr0 4y ago
    > it would appear that Google is a tech company that isn't actually very good at business Their primary product (ads) sells itself. So as a company they haven't ever had to become good at business -- in the interacting-with-other-business-entities sense.
fwsgonzo 4y ago
You expect not to be treated the same way by AWS? Am I missing something here?
- social_quotient 4y ago
  AWS would be easy to get on the phone near instantly.
  - deleted 4y ago
    [deleted]
- miyuru 4y ago
  AWS don't do shutdowns over a email for missing details. Even when they have issue over non payment, they wait at least 3 months, gives calls, sends follow up emails and for the actual shutdown they disable the network. the servers will continue to run for sometime. so if the payment is made it comes back without many issues.
  - shadowgovt 4y ago
    True, but this wasn't a missed-payment shutdown; it's a KYC-compliance shutdown. Still, I'd bet even money Amazon does the messaging better than Google on something like that.
jsiepkes 4y ago
It's really annoying that companies like Google bombard you with all kinds of "cover our ass" e-mails about every change they make. Most about features I don't even use. Making it even harder to understand what the impact is and know if I can safely ignore the mail. So I can fully understand people missing some "important" Google communication because of that.
Ayesh 4y ago
This is one of those reasons why I prefer prepaid cloud services. There are not many providers in this field (Vultr, Digital Ocean, etc). You can generally estimate the length of the runway based in the funds you have in your account. It also helps that there are multiple funding sources, so I can split the bill between multiple cards or accounts. I travel the major part of the year, and it is quite common to get my cards lost (often at my own fault) or compromised (Mastercard tends to suspend cards when they detect charges from multiple countries in a short time frame). I can wait for the new cards to arrive worry-free that my cloud services don't take a hit like this.
- sieabahlpark 4y ago
  [dead]
- daenney 4y ago
  Unfortunately, this wasn't a payment issue so this wouldn't have saved you. It's a Know-Your-Customer issue, i.e prove you're not doing business with someone on the US embargo list. DO, Vultr etc. would have the same issue depending on the timeframe they were given to prove compliance by the feds.
  - TedDoesntTalk 4y ago
    I’ve had Vultr and Digital Ocean accounts for years. Never got a KYC request from them. This is driven by the lawyers at Google.
iamben 4y ago
Emails are a difficult thing, aren't they? Was the 10 day countdown the first time Google mentioned "we'll suspend the account unless..."? Generally on these things - and unless it's an absolute emergency - you'd expect lots of prior warning and at least a couple of "YOUR SERVICES ARE DUE TO BE SUSPENDED" emails in the 48 hours before. That said, I've rightly called my bank out for sending an email with the subject line "Your account was compromised" - and then the body began "...is what you could be reading if you didn't have adequate security", followed by a sales pitch for some cyber security software. When you get rubbish like this (or absolutely banal "important information regarding" emails) so often, it's hard to know when to bother.
- ar9av 4y ago
  It's frustrating when emails don't categorize as critical and we end up missing something important. And don't even get me started on those misleading subject lines that turn out to be nothing but a sales pitch. It's like we have to play detective just to figure out which emails are worth our attention. It would be great if companies could be more transparent and upfront about the urgency of their messages.
  - GrinningFool 4y ago
    Every message is urgent to the company that sent it.
    - zamnos 4y ago
      Get a better vendor? They should at least have separate categories for marketing vs "important" account information, vs actually important account information (that last category can't be unsubscribed from).
- chrisandchris 4y ago
  True. AWS at least prefixes mails with "[ACTION REQUIRED]" if there _might_ be some action on your side required. And by coubting the mails that are coming in today with "important information", I can fully understand if you'll miss one.
pirsquare 4y ago
Hi everyone, I'm the customer in this case. Wanted to let everyone know that this sort of abrupt suspension for production apps has been a recurring issue that has happened time and again with Google Clown Platform. Similar horror stories: https://news.ycombinator.com/item?id=32547912 https://news.ycombinator.com/item?id=32547912 https://news.ycombinator.com/item?id=33737577 https://news.ycombinator.com/item?id=33737577 ----- [Update 1] Yes I agree I have some fault for the initial suspension, but that's not the biggest issue here. The biggest issue for me is how they "react" to help a customer get things back to normal when something goes wrong. First, I had to wait for 8 hrs for their Trust and Safety team. Then it was 24-48 hrs. See https://imgur.com/a/i0OHXcI https://imgur.com/a/i0OHXcI Until now I still haven't had a resolution. And the most ridiculous part is they could still tell me "I just received an email from the Trust and Safety team and they mentioned that they have sent you a separate email." when I had replied 3 times within an hour after they requested for more details. See proof https://imgur.com/a/1ZSmGBH https://imgur.com/a/1ZSmGBH
- apple4ever 4y ago
  If you have any fault it's very tiny. Those type of "important" emails are too common. And no warning is ridiculous too.
- philliphaydon 4y ago
  [dead]
- m00dy 4y ago
  I rent one of dedicated machines in Hetzner. Cheap and performant.
  - twawaaay 4y ago
    It is also a different product from Google Cloud or AWS.
- sourcecodeplz 4y ago
  > Then it was 24-48 hrs. See https://imgur.com/a/i0OHXcI https://imgur.com/a/i0OHXcI Usually I don't care much for grammatical errors, as long as the point gets across clearly. In this case I don't know why it irks me.
meibo 4y ago
"I missed an email that had important in the title because I didn't think it was important, now my servers are suspended and it's all Google's fault". Makes sense, I'm sure you'll fare better with Amazon.
- phphphphp 4y ago
  I’d normally agree — if you ignore an important email, the consequences are on you — but the Google verification process is very hostile and Google do constantly send emails about “important” but actually innocuous things which trains users to consider these emails as unimportant. The OP certainly has some culpability but having bad experience of this process myself multiple times, it does feel as if Google designed it to be hostile — the emails being just one part of it. Google Cloud is great in many ways, I love the product, but the billing is such a pain, top to bottom.
  - delfinom 4y ago
    >Google do constantly send emails about “important” but actually innocuous things which trains users to consider these emails as unimportant. Haha yea. Like the post notes, the Data Processor notifications have gotten so irritating that I had to setup a filter to junk them immediately. GDPR caused spam being foisted on the rest of us not in the EU.
- kryptiskt 4y ago
  My payment info needed to updated at AWS recently, they sent increasingly frequent emails about it for weeks (what can I say? I'm a bit lazy) and no services were affected while I dawdled. So, yes, AWS does it better.
- arunkant 4y ago
  More like "we gonna mess up customer's entire business because we can't bother to give a call on phone (even robocall would have been fine)". I get phone calls/sms for my phone bill/internet bill/electricity bill or even amazon deliveries. I guess a customer's business is not that important for google cloud
- 4y ago
DeathArrow 4y ago
Better use colocated servers. You own the hardware, the app and the data.
- deleted 4y ago
  [deleted]
permo-w 4y ago
yes this is fairly unreasonable behaviour by Google, but also, come on man. if this is your business and livelihood, then you should be at the very least scanning over all emails from your main service provider, never mind ones clearly marked as “important”
- ceejayoz 4y ago
  I have nothing running on GCP and still have hundreds of "important", "alert", etc. emails from their platform. Tax changes in Belarus! Advisory Notifications will become Generally Available! Spike in user reported spam! Same reason I've almost thrown out important communications from my credit card company; they send so much "important!" mail that's actually an offer to sign up for paid credit card monitoring it starts to be a spam flag.
  - permo-w 4y ago
    I personally have 2 cloud compute instances running, one with a specially requested GPU, and going back 3 months I have no emails from them marked "important", and only two that haven't directly resulted from my actions, each an update - clearly marked "[Update]" - to some element of their legal infrastructure which I suspect they're required to inform of have you actually checked this or are you just assuming it's the case?
saurabhnanda 4y ago
Please move to Hetzner. If you miss payments even they might pull the plug, but at least they have humans who respond to emails and can get you back online quickly once you pay.
- lbriner 4y ago
  This isn't really helpful because it's true until it isn't true or you get locked out in a way that you can't prove ownership or something. All businesses need to at least consider BC and DR, even if they eventually decide they have to risk it. There are loads of reasons why a dependency goes down which you have no control over so even basic spreading between providers or having a less-capable but simple DR system can go miles towards having a business not get killed in this way.
- re-thc 4y ago
  Hetzner works the complete opposite. You might not even get to open an account :p
- _cenw 4y ago
  As much as I love Hetzner and can recommend their legal department, they are very risk averse especially with new customers and might not even let you open an account. Any sort of weirdness from your server will also result in a very harsh email and a threat to suspend your services in way less than 10 days. I had the pleasure to experience their grumpy network admins when IPFS decided that node discovery included dialing 900+ RFC1918 IPs.
michaelteter 4y ago
With many online services, if there is something that needs verification (like confirming your email address), every time you use the app you will see a banner at the top reminding you of this need. Anything that is a countdown to doom should be a persistent banner on the app, reminding you of the importance and maybe even showing how many days left you have to deal with it.
- arunkant 4y ago
  Or maybe, make a phone call?
  - gardnr 4y ago
    I got a phone call from a domain registrar a few months ago. They let me know one of my domains was expiring and they didn’t have a card on file. I thanked them, hung up, and have since been slowly moving all my domains over to them. I really appreciated that phone call.
- londons_explore 4y ago
  To be fair, Google cloud does this. But most people don't log in the Google cloud dashboard every day for fun.
- 8organicbits 4y ago
  With cloud orchestration tools it's pretty common to log in to the provider's web console infrequently. A banner is easy to miss, even if it's persistent.
- shadowgovt 4y ago
  Google is actually uniquely bad at this. The system that displays notify banners for Cloud is a too-many-cooks-in-the-kitchen mechanism that requires multiple layers of authorizations, signoff, scrutiny, and binary changes to their UI. For that reason, it isn't (at least, didn't used to be) blocking to roll a feature out for that pipeline to work, so a lot of things that should be top-banner messages weren't.
Havoc 4y ago
Is it just me or is google getting a disproportionate amount of locked out issues?
- lbriner 4y ago
  Proptional to their prices, probably very proportionate. If you want better service, choose another provider and pay the price that it costs to pay 1000s of customer support staff, sometimes to only answer things like "why can't I see my EC2 instance" or "Where can I find my invoices". Personally, I have found both Azure and AWS to be pretty responsive to Customer Service, although AWS can be quite blunt with their replies like when we tried to get them to authorise allowing us to use EC2 instances as mail relays. "No, use SES" was the long-and-short of it. ymmv
  - shadowgovt 4y ago
    Scale also. If this happens in just 0.00001% of cases, that's plenty of users to get one or two events a day.
harry8 4y ago
bait and switch. It's so easy, all you need is x. Then when you're far enough in they demand a through w or you're shutdown with extreme prejudice. Do they really need it? Is that what you agreed to? Did you negotiate the change in terms. "I am altering the deal, pray I do not alter the deal further..." They don't blush. They don't flinch they don't even blink. Good luck if you really don't have another choice. Is the whole "cloud" just a con to trap people with predatory pricing into a massive overspend?
- shadowgovt 4y ago
  TBF, Google definitely doesn't collect KYC data for fun. They'd much prefer not to. But the Feds are pretty solid on the whole "Aiding and and abetting terrorism is treason" thing.
  - harry8 4y ago
    Tell them to take a hike until they come back with a warrant or at least a reasonable justifiable suspicion not dragnet surveillance. You know like google is a goddamn american company? What is this "Oh it's ok, we're just going along with the government's presumption of guilt to justify yet another power grab so that's fine." TBF Google are big enough to display some spine if they'd "prefer not to" I'm pretty solid on that. So I flat out don't believe you when you say they'd prefer not to because if that were true, they wouldn't. Are you sure I'm wrong to think that way? Looks to me like they're quite happy to.
    - shadowgovt 4y ago
      > Tell them to take a hike until they come back with a warrant or at least a reasonable justifiable suspicion not dragnet surveillance The Patriot Act doesn't work that way. Congress passed laws placing some reasonable minimum requirements on financial institutions that they have some sense of the actual identity of the people they are doing business with. Those are part of the "You're incorporated and do business in the financial sector" laws, so they're not a warrant kinda thing; a company's continued authorization to do business by law is contingent upon following them. If you want to get into the philosophy of the Patriot Act, feel free, but if you're an individual US citizen, you have more power to change it than Google does. Google can push back, but that's a good way to get hauled in front of a Senate subcommittee or have their business practices curtailed unless the law itself changes.
londons_explore 4y ago
Google don't care what your name is... They just want to be paid. However, the law requires them to not do business with people on the US embargo list. And to verify that, they need your name. That is probably why there was no way to ignore this paperwork, and why customer service couldn't override it.
- shadowgovt 4y ago
  ... and why the time window was so short. Google isn't triggering this because they want to lose customers; they're triggering it because the Feds breathed down their neck about a suspected violation.
- newaccount74 4y ago
  The problem isn't that they are suspending the account due to missing information. The problem is that they apparently consider a single email (that looks very similar to dozens of other emails that they send out) sufficient warning. Especially since most devs probably get dozens of spam emails a month that look more urgent ("Account Suspension" is a popular subject line for phishing emails). At the very least they should try calling the account holder before suspending the account. If they can't reach the account holder, they should re-enable the account very quickly after they reach them, and give them an extension with enough time to provide all the required information.
  - zamalek 4y ago
    Not to mention all the useless spam that Google send themselves. The SNR is abysmal.
andrewstuart 4y ago
“Moving Forward. Starting today, I will be making plans to move everything to AWS.” I laughed when I read this because it’s presenting moving to AWS as a completely different future. The industry has moved beyond depending on a single cloud computing vendor. It’s negligent to put all your eggs in one basket. I would say don’t put your business in the hands of a single cloud vendor. Either exit the cloud or go multi cloud or run plain old Linux machines on hetzner or ionos. Design your systems to withstand your credit card expiring, or missing an email.
- ceejayoz 4y ago
  > The industry has moved beyond depending on a single cloud computing vendor. Given how much of the industry struggles when an AWS region goes down, this seems... unlikely.
  - andrewstuart 4y ago
    How about “the industry should have moved beyond depending on a single cloud computing vendor.”?
    - shadowgovt 4y ago
      It's pretty cost-prohibitive. (... yes, more cost-prohibitive than losing one or two days of business to a SNAFU).
    - quadrifoliate 4y ago
      That is a big difference from what you said. The reality is that all serious outfits either host on AWS, or keep some level of AWS account at the ready, in addition to bare metal Linux servers in colos or hosting providers that they can move onto if needed. Your statement is confusing simplicity and flexibility with multi-cloud. No one wants to deal with the confusing mix of random hosted services that 3 different cloud providers try to sell you, with one of them (Google Cloud) being this opaque horror story that the article is talking about. Host on AWS, and mostly host your services on EC2. It's dead simple to migrate elsewhere if needed. You will likely never need to.
brian_cunnie 4y ago
Only one email from Google Cloud? That's weird: I got four when my Wells Fargo debit card was compromised on 2/28/2023. - 2023-03-04 "Your Project: nono is at risk of suspension" - 2023-03-04 "Action required: your billing account 0028C9-xx-yy is past due or has invalid payment information" - 2023-03-04 "Google Cloud Platform & APIs: Update your payment method to keep services" - 2023-03-04 "Google Cloud Platform & APIs: Payment was declined for 0028C9-xx-yy" Similarly, I got four emails from Digital Ocean, but only one from AWS.
- op00to 4y ago
  This isn’t past due, this is a KYC thing.
PaulHoule 4y ago
“Important” is a magic word that causes letters from the credit card company to go straight to the trash.
avsteele 4y ago
I hate how the internet age has conditioned customers to expect less. There is no way a non-internet business would cut off a client of 4-years without actually talking (phone) to them.
- looping__lui 4y ago
  You mean like airlines, banks, insurances, utility companies etc.? If anything standards and expectations went up in a lot of areas - e.g., think one day shipping, returning stuff whenever you feel like etc.
- kkielhofner 4y ago
  Same with uptime. I come from a healthcare and telephony background where "five nines" (roughly five minutes and 32 seconds of cumulative downtime in a year) is actually still a thing. I'm absolutely amazed at the number of companies that just accept each region/service/control plane/networking/etc outage in the big clouds. First you wait an extended period of time for them to even acknowledge the issue (rarely even updating their status dashboard) and then you stand around powerless waiting for your business to come back online. Time and time again. Yes, multiple region deployments are encouraged (still not even five nines) but when AWS and others give you the ability to create really convoluted architectures (which they love and encourage - it's sticky) using at least a half dozen of their products (IAM, VPC, RDS, S3, etc) for the simplest solution that make it virtually impossible to deploy, debug, and maintain multi-region without a dedicated "AWS Architect" role. Downstream outages for solutions/companies building in clouds because of region or service issues demonstrate just how difficult this approach is in practice. Then you sit back and wait for the almost inevitable "where did this $50k invoice come from", "we ran through our monthly budget overnight", "Oh they don't like us anymore and are closing our account", "What's the spend going to be this month - no one knows" and other scenarios discussed in this thread. Can't figure out how to wrangle costs? Sick of the outages? Good luck getting your cloud Rube Goldberg machine migrated out (especially because by now you've hired based on cloud of choice). Depending on a variety of factors any of these scenarios can kill a company. This isn't normal. This is regressive. It is unacceptable.
- _cenw 4y ago
cube00 4y ago
> Google Cloud have zero empathy for customers Common to any large company, if you want better service, support small business.
eigenvalue 4y ago
It seems like Google has burned up nearly all the goodwill it used to have years ago. They now have a nearly broken search engine that needs the term “Reddit” appended to get anything useful, and you routinely hear horror stories about them acting badly without any regard to users. I hope they get disrupted and replaced by bing chat and others asap. The only thing that will possibly teach them a lesson is if their business results start rolling over in a big way and large activist investors can cause the board to realize they need to replace Sundar with someone who gets that radical change is needed.
- danpalmer 4y ago
  These are memes that are popular on HN, but don't reflect reality outside of the tech bubble. There are of course elements of truth to them, hence why they're so pervasive, but they are not representative of reality.
  - igortg 4y ago
    Partially agree, but ... the tech bubble is pretty good on dictate trends and set standards that the general public will be in 2-3 years in advance.
    - danpalmer 4y ago
      I'm not sure that's accurate. We'd like to think so, but most tech trends don't take off and go mainstream.
    - ROTMetro 4y ago
      Is that still true? I think a group of tech influencers existed in the past but I don't see any evidence that continues to be the case. I guess they could all exist in discords I don't have site into. Previously you could see trends migrate through tech communities and out into the world. I don't really see that now.
  - NineStarPoint 4y ago
    It's for this reason that Google's ad business isn't really in danger at the moment, yes. People are going to continue to use google search and gmail and chrome. But having this reputation in the tech bubble negatively effects things like google cloud that tech people get to make decisions on. This is one of the reasons that google cloud is so far behind AWS and Azure in adoption.
Mizoguchi 4y ago
Make sure your decision to migrate to AWS is not an emotional one. I had my own feud with AWS over a $60,000 bill and an enormous amount of hours spent by my team fighting crypto hackers after they took over one of our accounts and use very advanced scripts to create hundreds of large instances. AWS has become very complicated to manage and at the same time the attacks to their infrastructure have increased exponentially. If your company doesn't have a dedicated team to stay on top of AWS you'll be vulnerable to attacks like the one I experienced. You simply can't count on the baseline configuration of your account and instances provided by AMS, you need advanced knowledge on how to secure your resources beyond basic stuff like MFA and monitoring. And if you think AWS will have your back when shit hits the fan, lol good luck with that. They will try to collect the bills while providing absolutely no support, reminding you about their Shared Responsibility Agreement. They also expect you to fight the attacks. I ended up spending months of work securing my services and cleaning my debt with AWS billing, to the point we almost went to court. So again, you may have good reasons to migrate, just beware that AWS won't come free of hassle.
- tasuki 4y ago
  > very advanced scripts to create hundreds of large instances. What was very advanced about those scripts? As an outsider, I'd think that perhaps a very simple script might be enough to create hundreds of large instances.
  - Mizoguchi 4y ago
    Sure, but that was only one part of the attack and not the clever one. The ability to bypass security, even after following AWS support instructions is what made it so difficult to fight, to the point that only after AWS security team took over the case it did not stop. The problem is AWS security doesn't interact with customers directly (as far as I know), you have to go through tech support and these guys just don't have that level of knowledge.
- jen20 4y ago
  Did you get to a root cause of how the “crypto hackers” “took over” your account, or of what the “very advanced” scripts did?
fredsmith219 4y ago
I am not a fan of "The Cloud" for ongoing production. I understand the case for it but I never liked the risk of someone else, with a different agenda and incentives than me, running my critical infrastructure. The Cloud does alleviate some risks with hosting your own servers, but it introduces new risks. This reminds me of a slogan I saw many years ago: There is no cloud, it's just somebody else's computer. That somebody else can complete shut you down when their interests no longer align with your own. No thank you.
- gspencley 4y ago
  I sort of tend to agree with you, but the counter-point is that there is always going to be some critical piece of the infrastructure that you are dependent upon that someone else controls. If you colocate, it's the data centre. If you host on-prem, it's the physical telco infrastructure. There are no shortages of incompetent people and institutions that will screw you. It's typically incompetence, not malice. But you will run into malicious actors as well. The reason I sort of agree with you is that, being someone who is very comfortable with bare metal solutions and came from the 90s mentality of writing portable code, I have started to see "The Cloud" as a return to mainframe programming. There is nothing wrong with that, but you are programming for "a machine" that often leads to code that is exceedingly hard to port. "Cloud Agnosticism" is the industry buzzword for not locking yourself into one vendor, but you need to make that decision on day one and set engineering policies enforcing a cloud services abstraction layer to be maintained. Most startups in the process of scaling are not going to see the immediate business value in incurring that cost. (though, with more horror stories like this one maybe that will change... but I doubt it).
  - kkielhofner 4y ago
    I've done large colo deployments. When you're provisioning your own hardware it's cloud-init, ansible, K8s, whatever. Nothing even knows you're in colos operated by different providers in different regions. Nothing knows or cares you're using five different upstream internet providers (whether your own - complicated or IP blends from the colo). Have a falling out or big issue? No reputable colo provider is going to hold your hardware hostage (it's the physical world and legal and reputational issues heat up quickly). They'll be happy to ship it out (get rid of you) to another provider, who will be even happier to re-rack. Meanwhile if your architecture is even remotely well designed (load balancer, multiple facilities) you're never down and it's not even that much of a "fire drill". With the capabilities of modern hardware this is often a single 1U redundant HW machine. All of this, by the way, is a fraction of the cost of cloud solution (a $25k bonkers 1U is ~$700 a month with a three year lease and 1U hosting + bandwidth + power is ~$200 a month). So for $2k a month (and it's actually predictable) you're up in two facilities and have capabilities an order of magnitude greater than a big cloud at equivalent spend. Companies screaming on twitter and profusely apologizing to their customers due to the most recent "name a cloud" outage demonstrate that multi-cloud (and even multi-region) is extremely difficult if not impossible to do in practice.
oliwarner 4y ago
It's a shame the industry can't agree on a basic SLA: - Solicit a hard limit when you open an account. People can only afford so much, don't bankrupt them. - As soon as we see your account heading for a 10,000% increase on last month, we'll send a SMS, GIVE YOU A PHONE CALL, even send a letter and confirm everything is okay. - If you're about to trigger a suspension, we'll send you an automated SMS, and if you're a paying customer, we'll do you a solid and spend $5 calling you to make sure you understand what's about to happen to your business. - You get one 7-day extension. People get sick, go on holiday, have life in their way and often haven't figured out how to juggle business critical credentials amongst people. Just imagine the quality of life improvements if people felt safe using a cloud provider. Instead we see stories like this and withdraw back to colocation and flat-rate VPSes.
- apple4ever 4y ago
  That's the human and courtesy way of doing it. Not sure why it's so hard.
- sharemywin 4y ago
  That's the thing right at some point you should have an account manager. if your spending maybe 4 but definitely 5 figures you should have an account manager. I would argue same on the publisher side.
- coldpie 4y ago
  That all sounds like it costs money. This is a capitalist society: companies operating correctly should provide as little service as possible for as much money as possible. Nothing you suggest aligns with that goal: -A hard limit? Why limit your income to $600 when your customers can accidentally rack up a $6000 bill? You can even be magnanimous and negotiate it down to a $2000 bill and everyone's happy. -Why would we pay money in monitoring software, SMS & phone call fees to stop a 10,000% revenue increase? -$5 to try to tell a customer not to give us more money? Are you crazy? -People getting sick under capitalism are drags on the economy and should be derided and made as miserable as possible so it doesn't happen again. > Just imagine the quality of life improvements if people felt safe using a cloud provider. We're not here to make people feel safe. We're here to make incredibly rich people even richer. Get on board, man. You'll never operate a successful company with your approach, everyone else will beat you on price.
ragnarsson 4y ago
I got a bill on GC that was less than a cent and my bank couldn't process that. GC closed my project (free tier) and then I paid the minimum amount that my bank allowed.
ranting-moth 4y ago
I remember when Google was the best thing because they were saving us from MS. Now they behave like this. I doubt even MS would have done a thing like this back in the day when they were the baddies.
danpalmer 4y ago
Reminder: if your level of interaction with a service is putting your credit card in, the service is likely to interact with you at the level of automated emails, and also likely to trust you as much as someone who filled out a web form. If it's business critical, talk to a human. All the cloud providers have account managers just waiting to talk to you. If this sort of thing happens and you have an account manager, they call you! Even better, get a contract that prevents them from cutting you off like this, or sets an SLA for getting back online. Again, all the cloud providers will offer this, even for relatively low spends.
- SomeHacker44 4y ago
  How do you get such a contract at the big providers?
  - re-thc 4y ago
    Show them the invoice of your current cloud provider. If it's a big enough number then...
  - danpalmer 4y ago
    You email them and say you're looking to chat about your plans for the next year. They'll set up a meeting in no time. This is the norm! Running a business on services purchased semi-anonymously is not.
  - timetofly2023 4y ago
    I work in gcp as account manager for enterprise, you won’t believe I chase deal for as low as 2k per month, tough time, every dollar counts
xyst 4y ago
The author mentions they will switch to AWS. But is Amazon really any better? Probably still face this same risk. You don’t own the hardware. You are just leasing it.
nova22033 4y ago
> 1) Google Cloud dropped us an email 10 days back that I missed out You missed a KYC e-mail and they locked your account. If you miss a "Important information about your bank account", what do you think your bank is going to do?
- albertopv 4y ago
  I would try to put me in customer shoes before shutting down their business because of missing paperwork. Even a single missed payment shouldn't be enough. Assume that counterpart may e didn't mean to do wrong, it almost always works.
- hectormalot 4y ago
  As someone working at a bank: you’re going to get a phone call and a bunch of letters (on actual paper). It’s exceedingly unlikely a bank would suspend your account on missing one KYC e-mail and no further follow ups.
- kevincox 4y ago
  I agree that this was partly the OPs fault, but I think that Google did the very least. 1. Send more than one email. Three would probably be reasonable for an account shutdown action maybe at 10, 4 and 2 days. 2. Use a better subject. That subject sounds like they are giving you information. At least something like "Required Important Information about your Bank Account". But something even better would probably be like "Must Provide Information about your Bank Account to avoid Account Closure" and maybe some escalating severity in the later warnings. 3. Do they have a phone number on file? Give that a call if there is no response to the original email in 3 business days. It could even be an automated message. "Hello, this is Google Cloud. We have sent you an email about required action on your account. Please read the information in that email and act appropriately otherwise your account will be closed. Remember that all Google Cloud emails come from @google.com addresses." 4. A big banner in the Cloud Console. After 4 business days show it to all users in the account. We are talking account shutdown of a paying customer's likely production services. They should take a bit of care.
thdespou 4y ago
It looks like a knee- jerk reaction. Not sure if Amazon will provide any more benefits. At least they'll get some alternative hosting
KomoD 4y ago
Suspended by valid reason, and your fault, as I could clearly see 23 words into the post.
mrslave 4y ago
Why does everyone want a copy of our identity documents? It's creepy and unsafe.
- cantSpellSober 4y ago
  Compliance with KYC https://support.google.com/googlepay/answer/7644078#verify https://support.google.com/googlepay/answer/7644078#verify
deleted 4y ago
[deleted]
CodeWriter23 4y ago
Actual subject line from AWS email when account about to be terminated. Got my attention. “Action required – Your AWS account is past due”
absentmoon 4y ago
Reading this makes me wonder if there is room for something similar to "The Global Chubby Planned Outage". In addition to good comms, they could gradually degrade your service up until suspension as a way of getting your attention.
breakingrules 4y ago
They're big on shutting you down for missing an email and then not talking to you. They did it to me over 10 years ago with adsense. Also, im a big fan of you calling them out and vowing to never use them again. Thats how it works.

209 points by ayewo 4y ago

nova22033 4y ago

If your servers were suspended, how is your website functional? https://console.onvoard.com/register/ https://console.onvoard.com/register/

greenyoda 4y ago
In the second paragraph of the article: "I managed to get our servers back online few hours ago with a workaround by adding a new billing account."

steponlego 4y ago

A nice big Threadripper in the corner of your office is all you need, I can almost certainly guarantee it. There are very very few businesses with less than 1000 employees that can't be run on one modern high end server.

leesalminen 4y ago
What happens when the power goes out? Internet goes out? Someone spills a cup of coffee on the server? Landlord needs to do some work and drywall dust flies around everywhere? It's not that most CRUD apps could run on a single machine, it's that they probably shouldn't. Customers want redundancy, automated failover, etc. Datacenters are much better at offering these things than your office park.
- re-thc 4y ago
  Interestingly enough the chances of that happening are actually less than an AWS region outage :).
- sourcecodeplz 4y ago
  You invest in 1-2 UPSes and a 4/5G mobile modem data plan that kicks in when needed. If you really want to be sure, get a diesel generator also. Will take some time to setup but it is worth it. - No one can see your code - No one can shut you down
- CyberDildonics 4y ago
  Someone spills a cup of coffee on the server? Are people spilling coffee on your servers at work?
- steponlego 4y ago
  You can do automated failover to Raspberry Pi, it'll hold while the main server is cared for. If you go cloud, you deserve what you get. And what you get is what you get, because it's not your computer, and it's not your data.
- margorczynski 4y ago
  Using a colo where each instance has a backup in another colo would probably be sufficient for most.
_cenw 4y ago
This always assumes these applications are built by competent people who value efficiency. The job market is full of people that think they need high scalability but then underestimate the cost of an HTTP request or the memory footprint of keeping an entire dataset in memory. It has happened several times in my career building platforms that I have to double check that packet capture I just took from that single Kubernetes node is accurate (my favorite is 22k DNS queries, all A IN s3.amazonaws.com, in a 0.2s span). Similar things happen with memory and CPU.
cantSpellSober 4y ago
You lose the benefit of a CDN though (caching content on a server closer to where each individual user is accessing it)

mjburgess 4y ago

In many ways, Google continues to expose the failure of automation. The 2010s dream of "megacorp serving everyone with centralised programming" ever more exposed as an unethical pipedream.

michaelteter 4y ago
> failure of automation Many of the Google horror stories are less a failure of automation and more just a failure of well designed business processes. This same event could/would have occurred if it were humans sending the "additional information required" emails and also following a protocol of "shut them down in 10 days if they don't provide". The fact that this process was (most likely) automated is not the issue. Honestly, as an outsider, it would appear that Google is a tech company that isn't actually very good at business. They do some smart tech things with their huge number of smart workers, but the business side seems to make a lot of very poor decisions (or simply doesn't even consider some part of the business or doesn't prioritize business processes well).
- eastbound 4y ago
  Every time I’m told it was an automatic system at Google, I doubt it. I can’t imagine all those heavy decisions aren’t actually reviewed by humans, who validate the automatic suggestions. Would anyone imagine that the decision to cut off a customer entirely, is not checked by a clerk? by a lawyer? The defense “It was automatic” doesn’t even pass the basic sanity test.
  - shadowgovt 4y ago
    At Google's scale, cutting one paying customer isn't a heavy decision. That's what makes doing business with them so weird; it may be mission-critical to your business, but to them it's a Tuesday.
- ethbr0 4y ago
  > it would appear that Google is a tech company that isn't actually very good at business Their primary product (ads) sells itself. So as a company they haven't ever had to become good at business -- in the interacting-with-other-business-entities sense.

fwsgonzo 4y ago

You expect not to be treated the same way by AWS? Am I missing something here?

social_quotient 4y ago
AWS would be easy to get on the phone near instantly.
- deleted 4y ago
  [deleted]
miyuru 4y ago
AWS don't do shutdowns over a email for missing details. Even when they have issue over non payment, they wait at least 3 months, gives calls, sends follow up emails and for the actual shutdown they disable the network. the servers will continue to run for sometime. so if the payment is made it comes back without many issues.
- shadowgovt 4y ago
  True, but this wasn't a missed-payment shutdown; it's a KYC-compliance shutdown. Still, I'd bet even money Amazon does the messaging better than Google on something like that.

jsiepkes 4y ago

It's really annoying that companies like Google bombard you with all kinds of "cover our ass" e-mails about every change they make. Most about features I don't even use. Making it even harder to understand what the impact is and know if I can safely ignore the mail. So I can fully understand people missing some "important" Google communication because of that.

Ayesh 4y ago

This is one of those reasons why I prefer prepaid cloud services. There are not many providers in this field (Vultr, Digital Ocean, etc). You can generally estimate the length of the runway based in the funds you have in your account. It also helps that there are multiple funding sources, so I can split the bill between multiple cards or accounts. I travel the major part of the year, and it is quite common to get my cards lost (often at my own fault) or compromised (Mastercard tends to suspend cards when they detect charges from multiple countries in a short time frame). I can wait for the new cards to arrive worry-free that my cloud services don't take a hit like this.

sieabahlpark 4y ago
[dead]
daenney 4y ago
Unfortunately, this wasn't a payment issue so this wouldn't have saved you. It's a Know-Your-Customer issue, i.e prove you're not doing business with someone on the US embargo list. DO, Vultr etc. would have the same issue depending on the timeframe they were given to prove compliance by the feds.
- TedDoesntTalk 4y ago
  I’ve had Vultr and Digital Ocean accounts for years. Never got a KYC request from them. This is driven by the lawyers at Google.

iamben 4y ago

Emails are a difficult thing, aren't they? Was the 10 day countdown the first time Google mentioned "we'll suspend the account unless..."? Generally on these things - and unless it's an absolute emergency - you'd expect lots of prior warning and at least a couple of "YOUR SERVICES ARE DUE TO BE SUSPENDED" emails in the 48 hours before. That said, I've rightly called my bank out for sending an email with the subject line "Your account was compromised" - and then the body began "...is what you could be reading if you didn't have adequate security", followed by a sales pitch for some cyber security software. When you get rubbish like this (or absolutely banal "important information regarding" emails) so often, it's hard to know when to bother.

ar9av 4y ago
It's frustrating when emails don't categorize as critical and we end up missing something important. And don't even get me started on those misleading subject lines that turn out to be nothing but a sales pitch. It's like we have to play detective just to figure out which emails are worth our attention. It would be great if companies could be more transparent and upfront about the urgency of their messages.
- GrinningFool 4y ago
  Every message is urgent to the company that sent it.
  - zamnos 4y ago
    Get a better vendor? They should at least have separate categories for marketing vs "important" account information, vs actually important account information (that last category can't be unsubscribed from).
chrisandchris 4y ago
True. AWS at least prefixes mails with "[ACTION REQUIRED]" if there _might_ be some action on your side required. And by coubting the mails that are coming in today with "important information", I can fully understand if you'll miss one.

pirsquare 4y ago

Hi everyone, I'm the customer in this case. Wanted to let everyone know that this sort of abrupt suspension for production apps has been a recurring issue that has happened time and again with Google Clown Platform. Similar horror stories: https://news.ycombinator.com/item?id=32547912 https://news.ycombinator.com/item?id=32547912 https://news.ycombinator.com/item?id=33737577 https://news.ycombinator.com/item?id=33737577 ----- [Update 1] Yes I agree I have some fault for the initial suspension, but that's not the biggest issue here. The biggest issue for me is how they "react" to help a customer get things back to normal when something goes wrong. First, I had to wait for 8 hrs for their Trust and Safety team. Then it was 24-48 hrs. See https://imgur.com/a/i0OHXcI https://imgur.com/a/i0OHXcI Until now I still haven't had a resolution. And the most ridiculous part is they could still tell me "I just received an email from the Trust and Safety team and they mentioned that they have sent you a separate email." when I had replied 3 times within an hour after they requested for more details. See proof https://imgur.com/a/1ZSmGBH https://imgur.com/a/1ZSmGBH

apple4ever 4y ago
If you have any fault it's very tiny. Those type of "important" emails are too common. And no warning is ridiculous too.
philliphaydon 4y ago
[dead]
m00dy 4y ago
I rent one of dedicated machines in Hetzner. Cheap and performant.
- twawaaay 4y ago
  It is also a different product from Google Cloud or AWS.
sourcecodeplz 4y ago
> Then it was 24-48 hrs. See https://imgur.com/a/i0OHXcI https://imgur.com/a/i0OHXcI Usually I don't care much for grammatical errors, as long as the point gets across clearly. In this case I don't know why it irks me.

meibo 4y ago

"I missed an email that had important in the title because I didn't think it was important, now my servers are suspended and it's all Google's fault". Makes sense, I'm sure you'll fare better with Amazon.

phphphphp 4y ago
I’d normally agree — if you ignore an important email, the consequences are on you — but the Google verification process is very hostile and Google do constantly send emails about “important” but actually innocuous things which trains users to consider these emails as unimportant. The OP certainly has some culpability but having bad experience of this process myself multiple times, it does feel as if Google designed it to be hostile — the emails being just one part of it. Google Cloud is great in many ways, I love the product, but the billing is such a pain, top to bottom.
- delfinom 4y ago
  >Google do constantly send emails about “important” but actually innocuous things which trains users to consider these emails as unimportant. Haha yea. Like the post notes, the Data Processor notifications have gotten so irritating that I had to setup a filter to junk them immediately. GDPR caused spam being foisted on the rest of us not in the EU.
kryptiskt 4y ago
My payment info needed to updated at AWS recently, they sent increasingly frequent emails about it for weeks (what can I say? I'm a bit lazy) and no services were affected while I dawdled. So, yes, AWS does it better.
arunkant 4y ago
More like "we gonna mess up customer's entire business because we can't bother to give a call on phone (even robocall would have been fine)". I get phone calls/sms for my phone bill/internet bill/electricity bill or even amazon deliveries. I guess a customer's business is not that important for google cloud
4y ago

DeathArrow 4y ago

Better use colocated servers. You own the hardware, the app and the data.

deleted 4y ago
[deleted]

permo-w 4y ago

yes this is fairly unreasonable behaviour by Google, but also, come on man. if this is your business and livelihood, then you should be at the very least scanning over all emails from your main service provider, never mind ones clearly marked as “important”

ceejayoz 4y ago
I have nothing running on GCP and still have hundreds of "important", "alert", etc. emails from their platform. Tax changes in Belarus! Advisory Notifications will become Generally Available! Spike in user reported spam! Same reason I've almost thrown out important communications from my credit card company; they send so much "important!" mail that's actually an offer to sign up for paid credit card monitoring it starts to be a spam flag.
- permo-w 4y ago
  I personally have 2 cloud compute instances running, one with a specially requested GPU, and going back 3 months I have no emails from them marked "important", and only two that haven't directly resulted from my actions, each an update - clearly marked "[Update]" - to some element of their legal infrastructure which I suspect they're required to inform of have you actually checked this or are you just assuming it's the case?

saurabhnanda 4y ago

Please move to Hetzner. If you miss payments even they might pull the plug, but at least they have humans who respond to emails and can get you back online quickly once you pay.

lbriner 4y ago
This isn't really helpful because it's true until it isn't true or you get locked out in a way that you can't prove ownership or something. All businesses need to at least consider BC and DR, even if they eventually decide they have to risk it. There are loads of reasons why a dependency goes down which you have no control over so even basic spreading between providers or having a less-capable but simple DR system can go miles towards having a business not get killed in this way.
re-thc 4y ago
Hetzner works the complete opposite. You might not even get to open an account :p
_cenw 4y ago
As much as I love Hetzner and can recommend their legal department, they are very risk averse especially with new customers and might not even let you open an account. Any sort of weirdness from your server will also result in a very harsh email and a threat to suspend your services in way less than 10 days. I had the pleasure to experience their grumpy network admins when IPFS decided that node discovery included dialing 900+ RFC1918 IPs.

michaelteter 4y ago

With many online services, if there is something that needs verification (like confirming your email address), every time you use the app you will see a banner at the top reminding you of this need. Anything that is a countdown to doom should be a persistent banner on the app, reminding you of the importance and maybe even showing how many days left you have to deal with it.

arunkant 4y ago
Or maybe, make a phone call?
- gardnr 4y ago
  I got a phone call from a domain registrar a few months ago. They let me know one of my domains was expiring and they didn’t have a card on file. I thanked them, hung up, and have since been slowly moving all my domains over to them. I really appreciated that phone call.
londons_explore 4y ago
To be fair, Google cloud does this. But most people don't log in the Google cloud dashboard every day for fun.
8organicbits 4y ago
With cloud orchestration tools it's pretty common to log in to the provider's web console infrequently. A banner is easy to miss, even if it's persistent.
shadowgovt 4y ago
Google is actually uniquely bad at this. The system that displays notify banners for Cloud is a too-many-cooks-in-the-kitchen mechanism that requires multiple layers of authorizations, signoff, scrutiny, and binary changes to their UI. For that reason, it isn't (at least, didn't used to be) blocking to roll a feature out for that pipeline to work, so a lot of things that should be top-banner messages weren't.

Havoc 4y ago

Is it just me or is google getting a disproportionate amount of locked out issues?

lbriner 4y ago
Proptional to their prices, probably very proportionate. If you want better service, choose another provider and pay the price that it costs to pay 1000s of customer support staff, sometimes to only answer things like "why can't I see my EC2 instance" or "Where can I find my invoices". Personally, I have found both Azure and AWS to be pretty responsive to Customer Service, although AWS can be quite blunt with their replies like when we tried to get them to authorise allowing us to use EC2 instances as mail relays. "No, use SES" was the long-and-short of it. ymmv
- shadowgovt 4y ago
  Scale also. If this happens in just 0.00001% of cases, that's plenty of users to get one or two events a day.

harry8 4y ago

bait and switch. It's so easy, all you need is x. Then when you're far enough in they demand a through w or you're shutdown with extreme prejudice. Do they really need it? Is that what you agreed to? Did you negotiate the change in terms. "I am altering the deal, pray I do not alter the deal further..." They don't blush. They don't flinch they don't even blink. Good luck if you really don't have another choice. Is the whole "cloud" just a con to trap people with predatory pricing into a massive overspend?

shadowgovt 4y ago
TBF, Google definitely doesn't collect KYC data for fun. They'd much prefer not to. But the Feds are pretty solid on the whole "Aiding and and abetting terrorism is treason" thing.
- harry8 4y ago
  Tell them to take a hike until they come back with a warrant or at least a reasonable justifiable suspicion not dragnet surveillance. You know like google is a goddamn american company? What is this "Oh it's ok, we're just going along with the government's presumption of guilt to justify yet another power grab so that's fine." TBF Google are big enough to display some spine if they'd "prefer not to" I'm pretty solid on that. So I flat out don't believe you when you say they'd prefer not to because if that were true, they wouldn't. Are you sure I'm wrong to think that way? Looks to me like they're quite happy to.
  - shadowgovt 4y ago
    > Tell them to take a hike until they come back with a warrant or at least a reasonable justifiable suspicion not dragnet surveillance The Patriot Act doesn't work that way. Congress passed laws placing some reasonable minimum requirements on financial institutions that they have some sense of the actual identity of the people they are doing business with. Those are part of the "You're incorporated and do business in the financial sector" laws, so they're not a warrant kinda thing; a company's continued authorization to do business by law is contingent upon following them. If you want to get into the philosophy of the Patriot Act, feel free, but if you're an individual US citizen, you have more power to change it than Google does. Google can push back, but that's a good way to get hauled in front of a Senate subcommittee or have their business practices curtailed unless the law itself changes.

londons_explore 4y ago

Google don't care what your name is... They just want to be paid. However, the law requires them to not do business with people on the US embargo list. And to verify that, they need your name. That is probably why there was no way to ignore this paperwork, and why customer service couldn't override it.

shadowgovt 4y ago
... and why the time window was so short. Google isn't triggering this because they want to lose customers; they're triggering it because the Feds breathed down their neck about a suspected violation.
newaccount74 4y ago
The problem isn't that they are suspending the account due to missing information. The problem is that they apparently consider a single email (that looks very similar to dozens of other emails that they send out) sufficient warning. Especially since most devs probably get dozens of spam emails a month that look more urgent ("Account Suspension" is a popular subject line for phishing emails). At the very least they should try calling the account holder before suspending the account. If they can't reach the account holder, they should re-enable the account very quickly after they reach them, and give them an extension with enough time to provide all the required information.
- zamalek 4y ago
  Not to mention all the useless spam that Google send themselves. The SNR is abysmal.

andrewstuart 4y ago

“Moving Forward. Starting today, I will be making plans to move everything to AWS.” I laughed when I read this because it’s presenting moving to AWS as a completely different future. The industry has moved beyond depending on a single cloud computing vendor. It’s negligent to put all your eggs in one basket. I would say don’t put your business in the hands of a single cloud vendor. Either exit the cloud or go multi cloud or run plain old Linux machines on hetzner or ionos. Design your systems to withstand your credit card expiring, or missing an email.

ceejayoz 4y ago
> The industry has moved beyond depending on a single cloud computing vendor. Given how much of the industry struggles when an AWS region goes down, this seems... unlikely.
- andrewstuart 4y ago
  How about “the industry should have moved beyond depending on a single cloud computing vendor.”?
  - shadowgovt 4y ago
    It's pretty cost-prohibitive. (... yes, more cost-prohibitive than losing one or two days of business to a SNAFU).
  - quadrifoliate 4y ago
    That is a big difference from what you said. The reality is that all serious outfits either host on AWS, or keep some level of AWS account at the ready, in addition to bare metal Linux servers in colos or hosting providers that they can move onto if needed. Your statement is confusing simplicity and flexibility with multi-cloud. No one wants to deal with the confusing mix of random hosted services that 3 different cloud providers try to sell you, with one of them (Google Cloud) being this opaque horror story that the article is talking about. Host on AWS, and mostly host your services on EC2. It's dead simple to migrate elsewhere if needed. You will likely never need to.

brian_cunnie 4y ago

Only one email from Google Cloud? That's weird: I got four when my Wells Fargo debit card was compromised on 2/28/2023. - 2023-03-04 "Your Project: nono is at risk of suspension" - 2023-03-04 "Action required: your billing account 0028C9-xx-yy is past due or has invalid payment information" - 2023-03-04 "Google Cloud Platform & APIs: Update your payment method to keep services" - 2023-03-04 "Google Cloud Platform & APIs: Payment was declined for 0028C9-xx-yy" Similarly, I got four emails from Digital Ocean, but only one from AWS.

op00to 4y ago
This isn’t past due, this is a KYC thing.

PaulHoule 4y ago

“Important” is a magic word that causes letters from the credit card company to go straight to the trash.

avsteele 4y ago

I hate how the internet age has conditioned customers to expect less. There is no way a non-internet business would cut off a client of 4-years without actually talking (phone) to them.

looping__lui 4y ago
You mean like airlines, banks, insurances, utility companies etc.? If anything standards and expectations went up in a lot of areas - e.g., think one day shipping, returning stuff whenever you feel like etc.
kkielhofner 4y ago
Same with uptime. I come from a healthcare and telephony background where "five nines" (roughly five minutes and 32 seconds of cumulative downtime in a year) is actually still a thing. I'm absolutely amazed at the number of companies that just accept each region/service/control plane/networking/etc outage in the big clouds. First you wait an extended period of time for them to even acknowledge the issue (rarely even updating their status dashboard) and then you stand around powerless waiting for your business to come back online. Time and time again. Yes, multiple region deployments are encouraged (still not even five nines) but when AWS and others give you the ability to create really convoluted architectures (which they love and encourage - it's sticky) using at least a half dozen of their products (IAM, VPC, RDS, S3, etc) for the simplest solution that make it virtually impossible to deploy, debug, and maintain multi-region without a dedicated "AWS Architect" role. Downstream outages for solutions/companies building in clouds because of region or service issues demonstrate just how difficult this approach is in practice. Then you sit back and wait for the almost inevitable "where did this $50k invoice come from", "we ran through our monthly budget overnight", "Oh they don't like us anymore and are closing our account", "What's the spend going to be this month - no one knows" and other scenarios discussed in this thread. Can't figure out how to wrangle costs? Sick of the outages? Good luck getting your cloud Rube Goldberg machine migrated out (especially because by now you've hired based on cloud of choice). Depending on a variety of factors any of these scenarios can kill a company. This isn't normal. This is regressive. It is unacceptable.
_cenw 4y ago

cube00 4y ago

> Google Cloud have zero empathy for customers Common to any large company, if you want better service, support small business.

eigenvalue 4y ago

It seems like Google has burned up nearly all the goodwill it used to have years ago. They now have a nearly broken search engine that needs the term “Reddit” appended to get anything useful, and you routinely hear horror stories about them acting badly without any regard to users. I hope they get disrupted and replaced by bing chat and others asap. The only thing that will possibly teach them a lesson is if their business results start rolling over in a big way and large activist investors can cause the board to realize they need to replace Sundar with someone who gets that radical change is needed.

danpalmer 4y ago
These are memes that are popular on HN, but don't reflect reality outside of the tech bubble. There are of course elements of truth to them, hence why they're so pervasive, but they are not representative of reality.
- igortg 4y ago
  Partially agree, but ... the tech bubble is pretty good on dictate trends and set standards that the general public will be in 2-3 years in advance.
  - danpalmer 4y ago
    I'm not sure that's accurate. We'd like to think so, but most tech trends don't take off and go mainstream.
  - ROTMetro 4y ago
    Is that still true? I think a group of tech influencers existed in the past but I don't see any evidence that continues to be the case. I guess they could all exist in discords I don't have site into. Previously you could see trends migrate through tech communities and out into the world. I don't really see that now.
- NineStarPoint 4y ago
  It's for this reason that Google's ad business isn't really in danger at the moment, yes. People are going to continue to use google search and gmail and chrome. But having this reputation in the tech bubble negatively effects things like google cloud that tech people get to make decisions on. This is one of the reasons that google cloud is so far behind AWS and Azure in adoption.

Mizoguchi 4y ago

Make sure your decision to migrate to AWS is not an emotional one. I had my own feud with AWS over a $60,000 bill and an enormous amount of hours spent by my team fighting crypto hackers after they took over one of our accounts and use very advanced scripts to create hundreds of large instances. AWS has become very complicated to manage and at the same time the attacks to their infrastructure have increased exponentially. If your company doesn't have a dedicated team to stay on top of AWS you'll be vulnerable to attacks like the one I experienced. You simply can't count on the baseline configuration of your account and instances provided by AMS, you need advanced knowledge on how to secure your resources beyond basic stuff like MFA and monitoring. And if you think AWS will have your back when shit hits the fan, lol good luck with that. They will try to collect the bills while providing absolutely no support, reminding you about their Shared Responsibility Agreement. They also expect you to fight the attacks. I ended up spending months of work securing my services and cleaning my debt with AWS billing, to the point we almost went to court. So again, you may have good reasons to migrate, just beware that AWS won't come free of hassle.

tasuki 4y ago
> very advanced scripts to create hundreds of large instances. What was very advanced about those scripts? As an outsider, I'd think that perhaps a very simple script might be enough to create hundreds of large instances.
- Mizoguchi 4y ago
  Sure, but that was only one part of the attack and not the clever one. The ability to bypass security, even after following AWS support instructions is what made it so difficult to fight, to the point that only after AWS security team took over the case it did not stop. The problem is AWS security doesn't interact with customers directly (as far as I know), you have to go through tech support and these guys just don't have that level of knowledge.
jen20 4y ago
Did you get to a root cause of how the “crypto hackers” “took over” your account, or of what the “very advanced” scripts did?

fredsmith219 4y ago

I am not a fan of "The Cloud" for ongoing production. I understand the case for it but I never liked the risk of someone else, with a different agenda and incentives than me, running my critical infrastructure. The Cloud does alleviate some risks with hosting your own servers, but it introduces new risks. This reminds me of a slogan I saw many years ago: There is no cloud, it's just somebody else's computer. That somebody else can complete shut you down when their interests no longer align with your own. No thank you.

gspencley 4y ago
I sort of tend to agree with you, but the counter-point is that there is always going to be some critical piece of the infrastructure that you are dependent upon that someone else controls. If you colocate, it's the data centre. If you host on-prem, it's the physical telco infrastructure. There are no shortages of incompetent people and institutions that will screw you. It's typically incompetence, not malice. But you will run into malicious actors as well. The reason I sort of agree with you is that, being someone who is very comfortable with bare metal solutions and came from the 90s mentality of writing portable code, I have started to see "The Cloud" as a return to mainframe programming. There is nothing wrong with that, but you are programming for "a machine" that often leads to code that is exceedingly hard to port. "Cloud Agnosticism" is the industry buzzword for not locking yourself into one vendor, but you need to make that decision on day one and set engineering policies enforcing a cloud services abstraction layer to be maintained. Most startups in the process of scaling are not going to see the immediate business value in incurring that cost. (though, with more horror stories like this one maybe that will change... but I doubt it).
- kkielhofner 4y ago
  I've done large colo deployments. When you're provisioning your own hardware it's cloud-init, ansible, K8s, whatever. Nothing even knows you're in colos operated by different providers in different regions. Nothing knows or cares you're using five different upstream internet providers (whether your own - complicated or IP blends from the colo). Have a falling out or big issue? No reputable colo provider is going to hold your hardware hostage (it's the physical world and legal and reputational issues heat up quickly). They'll be happy to ship it out (get rid of you) to another provider, who will be even happier to re-rack. Meanwhile if your architecture is even remotely well designed (load balancer, multiple facilities) you're never down and it's not even that much of a "fire drill". With the capabilities of modern hardware this is often a single 1U redundant HW machine. All of this, by the way, is a fraction of the cost of cloud solution (a $25k bonkers 1U is ~$700 a month with a three year lease and 1U hosting + bandwidth + power is ~$200 a month). So for $2k a month (and it's actually predictable) you're up in two facilities and have capabilities an order of magnitude greater than a big cloud at equivalent spend. Companies screaming on twitter and profusely apologizing to their customers due to the most recent "name a cloud" outage demonstrate that multi-cloud (and even multi-region) is extremely difficult if not impossible to do in practice.

oliwarner 4y ago

It's a shame the industry can't agree on a basic SLA: - Solicit a hard limit when you open an account. People can only afford so much, don't bankrupt them. - As soon as we see your account heading for a 10,000% increase on last month, we'll send a SMS, GIVE YOU A PHONE CALL, even send a letter and confirm everything is okay. - If you're about to trigger a suspension, we'll send you an automated SMS, and if you're a paying customer, we'll do you a solid and spend $5 calling you to make sure you understand what's about to happen to your business. - You get one 7-day extension. People get sick, go on holiday, have life in their way and often haven't figured out how to juggle business critical credentials amongst people. Just imagine the quality of life improvements if people felt safe using a cloud provider. Instead we see stories like this and withdraw back to colocation and flat-rate VPSes.

apple4ever 4y ago
That's the human and courtesy way of doing it. Not sure why it's so hard.
sharemywin 4y ago
That's the thing right at some point you should have an account manager. if your spending maybe 4 but definitely 5 figures you should have an account manager. I would argue same on the publisher side.
coldpie 4y ago
That all sounds like it costs money. This is a capitalist society: companies operating correctly should provide as little service as possible for as much money as possible. Nothing you suggest aligns with that goal: -A hard limit? Why limit your income to $600 when your customers can accidentally rack up a $6000 bill? You can even be magnanimous and negotiate it down to a $2000 bill and everyone's happy. -Why would we pay money in monitoring software, SMS & phone call fees to stop a 10,000% revenue increase? -$5 to try to tell a customer not to give us more money? Are you crazy? -People getting sick under capitalism are drags on the economy and should be derided and made as miserable as possible so it doesn't happen again. > Just imagine the quality of life improvements if people felt safe using a cloud provider. We're not here to make people feel safe. We're here to make incredibly rich people even richer. Get on board, man. You'll never operate a successful company with your approach, everyone else will beat you on price.

ragnarsson 4y ago

I got a bill on GC that was less than a cent and my bank couldn't process that. GC closed my project (free tier) and then I paid the minimum amount that my bank allowed.

ranting-moth 4y ago

I remember when Google was the best thing because they were saving us from MS. Now they behave like this. I doubt even MS would have done a thing like this back in the day when they were the baddies.

danpalmer 4y ago

Reminder: if your level of interaction with a service is putting your credit card in, the service is likely to interact with you at the level of automated emails, and also likely to trust you as much as someone who filled out a web form. If it's business critical, talk to a human. All the cloud providers have account managers just waiting to talk to you. If this sort of thing happens and you have an account manager, they call you! Even better, get a contract that prevents them from cutting you off like this, or sets an SLA for getting back online. Again, all the cloud providers will offer this, even for relatively low spends.

SomeHacker44 4y ago
How do you get such a contract at the big providers?
- re-thc 4y ago
  Show them the invoice of your current cloud provider. If it's a big enough number then...
- danpalmer 4y ago
  You email them and say you're looking to chat about your plans for the next year. They'll set up a meeting in no time. This is the norm! Running a business on services purchased semi-anonymously is not.
- timetofly2023 4y ago
  I work in gcp as account manager for enterprise, you won’t believe I chase deal for as low as 2k per month, tough time, every dollar counts

xyst 4y ago

The author mentions they will switch to AWS. But is Amazon really any better? Probably still face this same risk. You don’t own the hardware. You are just leasing it.

nova22033 4y ago

> 1) Google Cloud dropped us an email 10 days back that I missed out You missed a KYC e-mail and they locked your account. If you miss a "Important information about your bank account", what do you think your bank is going to do?

albertopv 4y ago
I would try to put me in customer shoes before shutting down their business because of missing paperwork. Even a single missed payment shouldn't be enough. Assume that counterpart may e didn't mean to do wrong, it almost always works.
hectormalot 4y ago
As someone working at a bank: you’re going to get a phone call and a bunch of letters (on actual paper). It’s exceedingly unlikely a bank would suspend your account on missing one KYC e-mail and no further follow ups.
kevincox 4y ago
I agree that this was partly the OPs fault, but I think that Google did the very least. 1. Send more than one email. Three would probably be reasonable for an account shutdown action maybe at 10, 4 and 2 days. 2. Use a better subject. That subject sounds like they are giving you information. At least something like "Required Important Information about your Bank Account". But something even better would probably be like "Must Provide Information about your Bank Account to avoid Account Closure" and maybe some escalating severity in the later warnings. 3. Do they have a phone number on file? Give that a call if there is no response to the original email in 3 business days. It could even be an automated message. "Hello, this is Google Cloud. We have sent you an email about required action on your account. Please read the information in that email and act appropriately otherwise your account will be closed. Remember that all Google Cloud emails come from @google.com addresses." 4. A big banner in the Cloud Console. After 4 business days show it to all users in the account. We are talking account shutdown of a paying customer's likely production services. They should take a bit of care.

thdespou 4y ago

It looks like a knee- jerk reaction. Not sure if Amazon will provide any more benefits. At least they'll get some alternative hosting

KomoD 4y ago

Suspended by valid reason, and your fault, as I could clearly see 23 words into the post.

mrslave 4y ago

Why does everyone want a copy of our identity documents? It's creepy and unsafe.

cantSpellSober 4y ago
Compliance with KYC https://support.google.com/googlepay/answer/7644078#verify https://support.google.com/googlepay/answer/7644078#verify

deleted 4y ago

[deleted]

CodeWriter23 4y ago

Actual subject line from AWS email when account about to be terminated. Got my attention. “Action required – Your AWS account is past due”

absentmoon 4y ago

Reading this makes me wonder if there is room for something similar to "The Global Chubby Planned Outage". In addition to good comms, they could gradually degrade your service up until suspension as a way of getting your attention.

breakingrules 4y ago

They're big on shutting you down for missing an email and then not talking to you. They did it to me over 10 years ago with adsense. Also, im a big fan of you calling them out and vowing to never use them again. Thats how it works.