6 ms·
"Finally, we are in planning to remove Google Cloud services from our data plane’s hot path, and keeping them only for secondary/failover." That's pretty clear
by Animats 5mo ago
"Finally, we are in planning to remove Google Cloud services from our data plane’s hot path, and keeping them only for secondary/failover."
That's pretty clear. Google can no longer be trusted as a B2B service provider.
- nikanj 5mo agoNever could. Google might block your entire company because one of your workers did something nasty on their personal account, and their ban hammer is mighty and blocks all related accounts to the Nth degree
- dpkirchner 5mo agoI've been wondering if I should be interrogating my friends before allowing them access to my wifi. "Have you or any of your family members ever been banned by Google?"
- Zamicol 5mo agoMore businesses need to hear this message. Google has proven time and time again they cannot be trusted as a service provider, exactly because of this problem.
- shimman 5mo agoThey trust them enough to still give them money, just goes to show how entrenched big tech is and why they need to be broken up into dozens of pieces.
- jimbokun 5mo agoIn the meantime start by migrating away from them for anything serious.
- tjwebbnorfolk 5mo agoThey have not explained WHY their account was suspended. That's the most important part, imo. Cloud Providers don't suspend entire accounts for no reason.
- JCTheDenthog 5mo agoUnfortunately the cloud providers also rarely if ever tell you the reason.
- londons_explore 5mo agoMy guess would be the credit card expired.... If it were something out of Railways hands, I think they would say something like "We have not yet identified the reason for the suspension, and are awaiting a response from Google".
- coreylane 5mo agoNo one pays $2m invoices with credit cards.
- tyrshand 5mo agoit surprised me a lot when I first encountered it, but some organisations do :)
- stackskipton 5mo agoAt any company doing Enterprise work, you don't cut off someone for non payment without Account Manager doing multiple phone calls to whoever you have contact information for, emailing everyone listed on the account and whoever opened a support ticket and maybe even putting a banner in the panel with "ACCOUNT OVERDUE, CALL US TO SORT IT OUT!" Generally it takes 30 days past due and complete no contact for anyone before suspension.
- Animats 5mo agoThat was normal US B2B behavior until recently. A supplier that cut off their customers for no good reason would lose their reputation in the industry and have real problems selling. Then we got down to a very small number of major suppliers in some areas. Many have moats that deter switching. This gives the big suppliers too much power. A century ago, this happened with railroads. Which is why railroads were forced to become common carriers. They're required to transport goods for anybody who can pay. Read up on antitrust activity of a century ago. We've been here before.
- nthypes 5mo agoMeta is no different. I know a company that had their OAuth app on Meta rendered completely unusable just because one of their employees (a dev) had their personal Facebook account banned by Meta for no reason. They tried to escalate it multiple times but got nowhere, lol. Meta is even worse because accounts need to be 'personal'; if you have a Business Manager, the users added to it are all tied to their personal Meta/Facebook accounts. This is ludicrous.
- nthypes 5mo agoMeta and Google B2B are both horrible. Their ad account bans are constant, and they have no real escalation process to get help. These companies are monopolies that should treat businesses more seriously, especially in these situations.
- malfist 5mo ago[flagged]
- lotsoweiners 5mo agoSeems relevant to me as it is still a service that their company relied on.
- Kiro 5mo agoSure you're not misreading Metal?
- skullone 5mo agoThe is in context of B2B, which meta has a huge ecosystem and often rips away a companies revenue for hidden reasons
- budoso 5mo agoCrazy considering this was their primary argument against the App Store's revenue share model. Not that they're wrong, but you'd think they would at least be consistent.
- sa46 5mo agoRailway has an overwhelming incentive to pin the blame on Google. This report doesn't answer why Google suspended Railway's account. I'd wait for more details before adjudicating.
- jerf 5mo agoIn principle, I agree with you. In practice, Google has earned the way my priors are ready to believe it's 100% their fault with mighty and sustained effort. Or lack thereof, depending on your point of view.
- CPLX 5mo agoThey said it was automated and affected a bunch of other customers, which gives at least some hint. And in general Google lost any immediate benefit of the doubt status many years ago. Many such stories.
- sophacles 5mo agoTo quote the article: > Google Cloud placed Railway’s production account into a suspended status incorrectly, as part of an automated action. This action extended to many accounts within Google Cloud. As this was a platform-wide action, there was no proactive outreach to individual customers prior to the restriction.
- subscribed 5mo agoThat would be approximately 6365262822 time Google suspended someone for no good reason. So no, Google doesn't get the benefit of the doubt.
- daninsea 5mo agoRailway don't have a great reputation for building scalable systems (effects of vibe coding?). It's worth waiting for Google's response before jumping to conclusions. They can move to Azure/AWS/own datacenter, but there's a good chance this will repeat in a few months.
- Citizen_Lame 5mo agoThere is always one bootlicker, fresh 1 day account no less.
- lightedman 5mo agoThen let me be the not day-one account to say Railway is utterly bearing some responsibility here. "However, in this ring, there was still a hard dependency on workload discoverability being tied to the network control plane API that was hosted on the machines running in Google Cloud." They've gotta be joking me that they deliberately left something so critical under the control of any other entity than themselves. That demonstrates a lack of critical planning and a lack looking at their configuration from a first-principles approach.
- Citizen_Lame 5mo agoThere is always responsibility with Railway, that's given. But also taking into account how many big websites went down when AWS was down, building critical redundancy at such large scale is not cheap, and not many companies do it. Same as security theatre, we have redundancy theatre because they needed to sell the CLOUD.
- daninsea 5mo agoBeen a passive reader here at HN for too long, finally registered today. Instead of viewing this incident objectively, you choose to insult me (?). I know multiple startup founders personally (2 of them are in the current YC batch), and the sheer callousness with which they look at infra, especially from security/scalability/reliability angle is shocking. I'll personally reserve judgement against GCP (replace with AWS/Azure/OCI/whatever) until we know more.
- FrustratedMonky 5mo agoHasn't every cloud provider had issues? Is the enshitification of servces really isolated to Google, or are we all doomed.
- jimbokun 5mo agoBanning accounts for no stated reason is kind of a Google speciality. They have a long well documented history of this sort of thing.
- Schiendelman 5mo agoHas AWS ever done this? I think their lack of social network nonsense is a benefit here...
- jimbokun 5mo agoThere is a history going back many years of Google suspending or terminating accounts with no explanation, often having to backtrack after users published their frustration and the incident went viral. Google has always acted as if they have no obligations whatsoever to their paying customers.
- r0m4n0 5mo agoI'm not sure that's the lesson to learn from this outage. Hell Google resolved the problem in 7 minutes which is as good as you could hope for. The resulting action should be you have proper disaster recovery, failover, etc. Not sure I would trust these folks if this is the conclusion they are coming to from this experience. Any cloud provider can/will do this to you.
- MrDarcy 5mo agoGoogle restored access but did not resolve the problem. VM’s were still shut down.
- 0cf8612b2e1e 5mo agoGoogle resolved the problem in seven minutes for a billion dollar company. Good luck if you are a nobody. The best you could hope for is that if there is something fishy with your account, you are contacted by Google to address it.
- tantalor 5mo agoSomebody else's computer
- sneak 5mo agoNo US large cloud provider can. They all spy for their national military, as well.