23 ms·
Incident Report: May 19, 2026 – GCP Account Suspension
Previous thread: Incident Report: Railway Blocked by Google Cloud [resolved] - https://news.ycombinator.com/item?id=48201484 https://news.ycombinator.com/item?id=48201484
- stefan_ 5mo agoIt's reassuring to know they will ban a million dollar enterprise customer just like they will ban your GMail of 20 years.
- SoftTalker 5mo agoIt really is amazing that there is not some level at which "human review" becomes mandatory. Customers of that size already have dedicated account rep contacts.
- tedd4u 5mo agoI can't believe Kurian has not put his foot down about this. Adverse action against accounts over $X ARR absolutely must have review by revenue-carrying people before the action is taken.
- Jgrubb 5mo agoRailway has not had the best month in the tech press have they? And in both cases it was an automated process belonging to some other party that put them there, damaging their reputation. I was going to talk to our google rep about their killing the Gemini cli but this is way more concerning.
- QuercusMax 5mo agoBuilding on someone else's platform is always gonna be a risky move, and building a platform on top of someone else's platform is even riskier. My company used to use a hosting provider that was basically AWS plus some extra guarantees. We just finished migrating onto regular AWS because they now offer what we need directly.
- gandreani 5mo agoBut...AWS is a platform too, no? Seems like you're in the same category of risk you just moved to a more well-known name. Granted, Amazon is the most reliable even if they have their own quirks.
- QuercusMax 5mo agoEach critical dependency you stack multiplies your risk. Now you have to worry about Railway AND Google causing business-damaging outages.
- stingraycharles 5mo agoI was looking at this from Railway’s perspective. I really wonder what caused their account to be flagged, and they hint at more accounts being erroneously flagged as well.
- ImPostingOnHN 5mo agoIn the case of them giving AI admin credentials to delete their production database, and it deleted their production database: that's on them. They were the only ones who put the admin account credentials into their AI. Then they took no personal responsibility. That definitely damaged their reputation. Here, they are taking at least some responsibility. Props to them on improving. Also, GCP does indeed have serious reliability issues, and Google does indeed have serious customer support issues. EDIT: It has been brought to my attention below that the first 2 paragraphs are misattributed, and were not Railway, but rather a customer of theirs. Sorry, Railway!
- nightpool 5mo agoDid Railway give admin credentials to delete their production database? My memory of the incident is that a customer of Railways used an AI tool to delete their production database, and then blamed Railway for it. The customer was the one who put their own account credentials into their own AI, not Railway
- ImPostingOnHN 5mo agoYou are totally right, my mistake.
- AlfieJones 5mo agoThis isn’t the first time Google Cloud has seriously messed with a customer’s account: https://cloud.google.com/blog/products/infrastructure/details-of-google-cloud-gcve-incident https://cloud.google.com/blog/products/infrastructure/detail...
- rurban 5mo agoGoogle, the new Microsoft!
- redwood 5mo agoHonestly they really are starting to look that way. Total opinionated Walled Garden that's against an open and thriving ecosystem. Unlike Microsoft the technology is not yet garbage but I hope this isn't where they're going to end up
- pesus 5mo agoIt sure is heading towards being garbage, though. Search is actively being degraded in favor of a barely functioning AI, and I'm sure it's not going to stop there. Seems like it was inevitable once ad/finance people got ahold of the company.
- cryo32 5mo agoI think this is just the default endgame of large corporates which suck up large quantities of customers. They are a race to the bottom and you end up with service by footgun. My own company is responsible for doing this in our sector. Literally every technology decision favours automation over verification because it's cheaper to say sorry than do it right.
- raverbashing 5mo agoAmazon played AWS from day 1 as if they were the runner-up (and in a sense they were), and while it does look like it's day 2 there, they are not letting the momentum down Microsoft might have technical warts but commercially they are strong and Azure is a lot of times bundled with other services and you know you can get someone on the phone if needed Google has... ?
- guluarte 5mo agotldr: AI suspended an almost a billion dollar startup account.
- ChrisArchitect 5mo agoRelated discussion during the incident: https://news.ycombinator.com/item?id=48201484 https://news.ycombinator.com/item?id=48201484
- tptacek 5mo agoDuplicate of: https://news.ycombinator.com/item?id=48201484 https://news.ycombinator.com/item?id=48201484
- r721 5mo agoLooks like it was endorsed by dang: https://news.ycombinator.com/item?id=48210941 https://news.ycombinator.com/item?id=48210941
- ryanSrich 5mo agoQuestion: for a smaller SaaS tool, or even internal product. If a team doesn't want to manage AWS or another IaaS provider, what are the best alternatives for the following 1.) Vercel - having a bad month 2.) Supabase - having a bad month 3.) Railway - now having a bad month
- mattmatters 5mo agoHaven't used railway but my understanding is they are something similar to Heroku. Fly.io has been pretty great for tiny projects in that niche. For Vercel if your nextjs site can be compiled statically you could probably throw it up on almost anything. We've self hosted before which is pretty straightforward but you lose a lot of the image optimization stuff unless you go deep into setting up open next.
- levkk 5mo agoDigitalOcean. Seriously. They have been around a long long time and built a lot of the core infrastructure you rely on every day (e.g. Ceph).
- wouldbecouldbe 5mo agoI;ve had my share of VPS & Managed DB outages at DO, so they are also not faultless.
- jasonlotito 5mo agoNot if, but when. No one is faultless. Chasing after 100% is a fool's errand.
- efdee 5mo agoI've been with DO since checks mailbox 2014. Honestly never experienced an unannounced outage.
- wouldbecouldbe 5mo agoYeah overall they are ok. I think 3 times managed db and one or twice a vps just dead. No issues in a year or so. They were always hardware failures, took about 45-120min. Not the end of the world, but also not fun getting lot of client complaints.
- ibejoeb 5mo agoI've been getting serious, recently, about moving all my workloads to equipment that I control in datacenters with which I have professional relationships. It's less expensive, easier, and this kind of nonsense doesn't happen. These cloud providers need to step back and observe how terrible they've made these products. Footguns everywhere, pricing that is impossible to forecast or reason about, broken APIs, and automated self destruction. Then you have third-party providers sitting on top of them, adding another layer of each antifeature. Crazy.
- lacewing 5mo ago> These cloud providers need to step back and observe how terrible they've made these products. They don't, because the allure of effortless scaling is hard to resist: everyone thinks of themselves as the next tech unicorn. And if you actually become an unicorn, you're already too dependent on AWS / Azure / GCP to easily move somewhere else. At best, your strategy is to become "multi-cloud".
- ibejoeb 5mo agoThat effortlessness is a fantasy. That's illustrated right here in this write-up by how complicated their system is. >Railway’s network is a mesh ring, built up of high availability fiber interconnects between Metal <> GCP <> AWS. However, in this ring, there was still a hard dependency on workload discoverability being tied to the network control plane API that was hosted on the machines running in Google Cloud What the hell is even that?
- foobar1726 5mo ago"We had all of our workers set up in an open office layout to to make sure everyone could talk to each other without a single point of failure. But last night the boss got too drunk and didn't come in, so everyone spent the day scrolling tiktok."
- bombcar 5mo agoThe thing that's nice about physical datacenters with people is that they often have to physically walk over to disconnect you - it's not as easy as some automated system doing an AI.
- Animats 5mo ago"Finally, we are in planning to remove Google Cloud services from our data plane’s hot path, and keeping them only for secondary/failover." That's pretty clear. Google can no longer be trusted as a B2B service provider.
- nikanj 5mo agoNever could. Google might block your entire company because one of your workers did something nasty on their personal account, and their ban hammer is mighty and blocks all related accounts to the Nth degree
- dpkirchner 5mo agoI've been wondering if I should be interrogating my friends before allowing them access to my wifi. "Have you or any of your family members ever been banned by Google?"
- Zamicol 5mo agoMore businesses need to hear this message. Google has proven time and time again they cannot be trusted as a service provider, exactly because of this problem.
- shimman 5mo agoThey trust them enough to still give them money, just goes to show how entrenched big tech is and why they need to be broken up into dozens of pieces.
- jimbokun 5mo agoIn the meantime start by migrating away from them for anything serious.
- tjwebbnorfolk 5mo agoThey have not explained WHY their account was suspended. That's the most important part, imo. Cloud Providers don't suspend entire accounts for no reason.
- teraflop 5mo ago> May 19, 22:10 UTC - Our automated monitoring detected API health check failures and paged our on-calls, who started investigating the issue. > At 22:20 UTC on May 19, Google Cloud placed Railway’s production account into a suspended status incorrectly, as part of an automated action. If the timestamps are accurate, what was causing the errors 10 minutes before the account was suspended? The simplest explanation is just that one or the other of these timestamps is wrong, which wouldn't be a big deal. But if the timestamps aren't known with certainty, it seems very odd to include them in the writeup as though they are certain, even though they are very obviously inconsistent with each other.
- Shank 5mo ago> If the timestamps are accurate, what was causing the errors 10 minutes before the account was suspended? Assuming the timestamps are accurate, Google probably started terminating resources while the account was not "suspended" and only completed that after all resources were disabled.
- sroussey 5mo agoOr the account started doing something nefarious (assuming one of their customers as root cause, not railway itself) that started causing real problems and Google shut it down. The problem with not having the data is that it’s easy to make assumptions.
- bink 5mo agoThe absence of any explanation for the suspension does seem intentional. If it were me that's one of the first things I would've asked so that I could make sure it doesn't happen again.
- sroussey 5mo agoMy ebay account is partially suspended -- i can buy but i can not sell. Of course i asked why! But they refuse to tell me.
- indrex 5mo agoHad similar experience with GCP. Terminated VMs six times, and responded zero times.
- theredleft 5mo agoback to on-prem
- mxuribe 5mo agoHonestly, i have been wanting to suggest to my leaders that we should go to on-prem for primary, and use cloud only as extra for peak traffic and/or failover, etc...but, the culture where i'm at is so bought into cloud as if it solves all problems...and then, in the next breath they all ask me to drastically reduce cloud costs and ensure 100% uptime at all times 24/7/365 (1005 uptime without complexity and without any added costs!).
- in_a_society 5mo agoGoogle has a culture problem. This is not something that can change easily nor will it change when it’s not recognized as being an issue within their organization. Between my peer c-suites, the conversation is that GCP cannot even be in the consideration set until such a time as a several-year period has elapsed without this kind of incident.
- shwetanshu21 5mo agoTK TK TK TK. Welcome to the jungle.
- tcdent 5mo agoThe interesting and yet-to-be-explained part is why google flagged the account? Put all the timestamps you want in the post mortem about what you observed, but you haven't addressed the root cause. The "this doesn't make sense" part of the story likely has a real explanation that nobody wants to reveal yet.
- Aperocky 5mo agoShouldn't Google answer this if they are unhappy with this incident report? Are we even sure that Railway knows?
- tomComb 5mo agoThey can't - that would violate the privacy rights of their customer. They need to tell Railway and Railway needs to tell us, or Railway can tell us that Google is refusing to tell them. Either way, we need to hear about this from Railway.
- e40 5mo agoI seriously doubt Railway knows. That's the MO for Google and others, suspend account without explanation.
- SoftTalker 5mo agoThe report at this point is pretty much just a timeline of what happened. No explanation of why, no accusations, no blame. A PR piece, to Railway's customers, reassuring them that "we're not ignoring this." Now the lawyers are huddling. IMO there won't be a lot more said publicly by either side, at least until any threat of lawsuits for damages is settled.
- kyrra 5mo agothe Railway PM doesn't say they weren't told. It just sort of glosses over this. I would be interested to know if they were told (or not).
- croes 5mo agoThat‘s the point where Google tells you they won’t tell you the exact reason because of security reasons
- majdalsado 5mo agoUnfortunately we had to make emergency migration off to Azure yesterday due to this. Thankfully our DB was not hosted on Railway and we were back up in a couple hours. As much as we loved the simplicity they provided us, there's just been too many mishaps and shortcomings for us to continue running a B2B enterprise app on their infrastructure. Sad day :(
- gandreani 5mo agoAzure suspended your account as well?
- jmaw 5mo agoI think they meant that they migrated off of railway TO azure as opposed to FROM azure
- jmaw 5mo agoWhat were your reasons for going with railway in the first place? I'm not super familiar with them, but did you choose them for unique offerings, or essentially just VMs? If unique offerings, how rough was the migration out?
- pocksuppet 4mo agoRailway can probably sue Google for their loss of income from you. Should be fun.
- binarycleric 4mo agoUnlikely to be successful. GCP and AWS have pretty tight terms of service and you're legally SOL unless there was true negligence. Accounts rarely get suspended for no reason. There is likely more to this story that Railway isn't sharing.
- Bender 5mo agoI've read all the threads and their main page and I still don't really understand what this service is. Is this like a commercial alternative to Gerrit? What do people use this for? I'm not a developer, just curious what this is.
- natbennett 5mo agoThe category is “Platform as a Service” Alternative to Fly or Heroku Here is my source code Run it on the cloud for me I do not care how In this case it looks like they also bundle together a bunch of the other services you would need to get code onto the platform, monitor it once it’s there and so on
- Bender 5mo agoOh I see, so they manage the server hosting and application server configuration, optimization and all that jazz. Almost like one step away from managed hosting. Makes sense now, thankyou!
- mococa 5mo agoFlagged by some AI automation.
- shwetanshu21 5mo agoThis should be a warning to anyone running GCP. They suspend accounts left right and centre without even thinking about what they're doing. It seems like they use Gemini 3.1 Pro to run their production decisions. TK has a history of absolutely destroying the culture of the place like in OCI and has done something similar in GCP from what I've heard. GCP and Google are completely different entities with how they work. Don't expect Google quality from the name. It's just like those old brands which now have cheap licensed products like Nokia (An exaggeration I know but not far from truth). Not only that they are known to shut off their services randomly giving you like 6 months to migrate. They have lots of engineers not doing anything, so they put them on migrating internal users off those services, most of their clients don't. There was a brilliant article on this by an ex-GCP employee that I can't find right now. Avoid GCP like plague if you are serious about your business. Edit: Gemini (unironically) found the article on this, a very good read: https://steve-yegge.medium.com/dear-google-cloud-your-deprecation-policy-is-killing-you-ee7525dc05dc https://steve-yegge.medium.com/dear-google-cloud-your-deprec...
- JohnMakin 5mo agoAll google products work like this. Should never be used for anything critical.
- shwetanshu21 5mo agoYeah, sadly know that from being burned from one of their depreciations. In fact, 2-3. But you live and you learn. And it is better to learn from other's mistakes always.
- cyco130 5mo agoAnd this is Railway, a big enough name to top the HN main page and presumably find someone from Google to intervene at some point. I would have zero recourse if it was some little product that I built.
- shwetanshu21 5mo ago100% agree, I've seen on Twitter and HN small players facing similar issues with no recourse and response from Google. I don't know what kind of place they are trying to build there. They got TK to woo the enterprise customers who were forced to be hostage to OCI. But it seems they are still doing opposite of hostage here.
- AtNightWeCode 5mo agoSo, what was the reason for the account suspension. Why did it happen? I know Google can be a bit stupid with their automatons but I am bit skeptical here. There are sites more critical than Railway hosted on GCP.
- corndoge 5mo ago> Your customers don't care whether the failure was Google or Railway; they see your product. Refreshing. So tired of businesses blaming their vendors. Oh it wasn't us spamming you text messages and emails, it was Shopify. Oh, our delivery guarantee said 2 days and it's been a week? That's not us, it's UPS. I don't care. I didn't pay UPS or Shopify. I paid you.
- mellosouls 5mo agoEven if it ultimately turns out to be "Google's fault" (as this report seems to be saying), Railway say they own the incident but make no apology here.
- dantillberg 5mo agoWhat drives Google to apply these actions so completely and immediately, versus a more deliberate approach, with notification and delay before action, manual review for paying customers, or a warning to resolve within X hours/days? Once or twice could be errors or bad implementation, but these can't explain away the pattern. It would seem that Google's counsel has deemed that whenever _____ is detected, the company must immediately and completely sever the business relationship. What is that driving concern? Is it sanctions enforcement? CSAM? Something else?
- e40 5mo agoThe problem is scale. Google uses automation and doesn't have the people to review the actions of that automation. I never worked at Google but this is the most obvious explanation from watching these things happen for years and years. Please, someone that worked at Google, please comment.
- ihsw 5mo ago[dead]
- BitWiseVibe 5mo agoIt could be automated action based on abuse reports. TONS of spam comes from Railway associated networks.
- spogbiper 5mo agoI work in a security adjacent role and I know we have had a few incidents that involved Railway networks lately. Could be something to that, I don't know
- ur-whale 5mo agoPerfect reminder that it's time to use Google Takeout while I still can.
- loxodrome 5mo agoI will definitely not be signing up on GCP because of this.
- tamimio 5mo ago> Railway’s production account into a suspended status incorrectly, as part of an automated action. Be it individuals or companies, this time is the best time to ditch all dependence on anything clouds or SaaS since all are using automated AI, more and more of these incidents will occur.
- tristanb 5mo ago"Your customers don't care whether the failure was Google or Railway; they see your product. Your uptime is our responsibility, and we'll keep delivering on it." - Thanks Claude!
- 1970-01-01 5mo agoThey forgot to get reimbursement for downtime. A free month of GCP is better than nothing.
- deleted 5mo ago[deleted]
- FajitaNachos 5mo ago19 minutes from detection to getting the google account restored is pretty awesome honestly.
- koliber 5mo agoNow given the logic that you can't be dependent on any one service to run your SaaS, how does Railway convince its customers to run their SaaS on a single service?
- dan_sbl 5mo ago> As a side effect, Terms-of-service acceptance records were also reset, prompting users to re-accept on their next visit to the dashboard. Don't get me wrong- the rest of this mess falls pretty clearly on Google Cloud, but this one feels like something Railway did to themselves.
- llmslave 5mo agoMajor infra provider -> has no backups/game plan if GCP goes down
- beauregardener 5mo agoSadly, My Railway project is still having issues 24 hours later. Already started emergency migration away from Railway backend :(
- shwetanshu21 5mo ago"Railway owns our vendor choices, and we ultimately own this one. Your customers don't care whether the failure was Google or Railway; they see your product. Your uptime is our responsibility, and we'll keep delivering on it." Kudos to them for acknowledging it and not doing PR speak. It shows it was an architectural failure from their part of trusting GCP, and they are working to fix it. Should they have seen it coming? Yes. But better late than never.
- 8note 5mo agoit sounds to me like templated text from the UVic ESS office somewhere:P
- whirlwin 5mo agoThe RCA and preventive measures was a pleasant read. I got a lot of respect for companies putting a lot of effort into incident reports like these. Makes them appear very professional rather than just blaming the cloud provider outright.
- alansaber 5mo agoUnfortunately we've also had a litany of problems with our GCP deployment and chose to remove them completely as a service provider.
- charcircuit 5mo ago>Google Cloud placed Railway’s production account into a suspended status incorrectly, as part of an automated action. There is no justification given on why this action was incorrect. It's possible they actually did something wrong.
- nosefrog 5mo agoYup...
- dpkirchner 5mo agoBut not so wrong that they could get their account back in 10 minutes.
- siliconc0w 5mo agoWhy would you use an infrastructure provider on top of another infrastructure provider? It adds cost and risk, it's always going to be a leaky abstraction, and it's not hard to learn how to use GCP or AWS correctly - especially with agents.
- bagels 5mo agoWhat intermediate is involved?
- nosefrog 5mo agoIt's highly unlikely that GCP banning their account without telling them is true, but GCP is probably not going to go public with the real reason.
- christophilus 5mo agoI’m 100% willing to believe it based on Google’s track record.
- pm90 5mo agoI don’t understand why Google still has TK helming GCP when its obviously not achieved the kind of success it should. Google infra is some of the best in the world yet GCP is meh. It continues to underperform and seems content to be a distant 3rd behind AWS and Azure.
- _justme 5mo agoDoes this qualify for a list entry on killedbygoogle.com ?
- myself248 5mo agoHow many trains were delayed or incorrectly routed as a result?
- kittikitti 5mo agoIt's not my proudest moment, but at least being banned and suspended all the time brings some wisdom.
- phendrenad2 5mo agoNov 28, 2012, user fbuilesv posted: Google Compute Engine is on Limited Preview right now. If you're planning to offer a service that you care about you should consider this. Do we know if GCP has ever left limited preview..??!
- yugoru 5mo agoAll my services have lost their databases. All their AI agent is offering me is to delete all the data (thousands of data) and create a new database. It insists there's no other way. But I have clients, I have hundreds of courses on the platform!
- deleted 4mo ago[deleted]
- dimitrismrtzs 4mo agostories like this are why i self-host most things on proxmox instead of depending on a single cloud provider. Ok I have to do the maintenance but at least this way no one can suspend my entire stack with not even an explanation.