4 ms·

>We have resolved this incident and a post mortem is available here. >https://blog.railway.com/p/incident-report-may-19-2026-gcp-account-outage https://blog.ra

by r721 5mo ago

>We have resolved this incident and a post mortem is available here. >https://blog.railway.com/p/incident-report-may-19-2026-gcp-account-outage https://blog.railway.com/p/incident-report-may-19-2026-gcp-a... >May 20, 07:57 UTC https://status.railway.com/incident/I23M92U0 https://status.railway.com/incident/I23M92U0

sschueller 5mo ago
It should be possible to sue Google for damages in such cases. This isnt a network outage or service failure which I would consider part of ToS.
- VladVladikoff 5mo ago
  What if the reason for their stuff being shut down was a payment issue like an expired credit card or maxed credit account? Unless I missed it skim reading their post I don’t see any information anywhere about their communications with Google.
  - bastawhiz 5mo ago
    If you have an account manager and a contract, there's zero excuse for automated suspension. That's literally the whole point of having a dedicated person. From the report: > May 19, 22:22 UTC - P0 ticket filed with Google Cloud. Railway's GCP account manager engaged directly.
- Cthulhu_ 5mo ago
  It's always possible to sue, but Google has good terms of service and lawyers - I'm 99% confident that a lawsuit would end up nowhere.
  - bastawhiz 5mo ago
    They have every right to sue, and if they did sue they almost certainly would win. This is clear breach of contract. The only argument Google could make is "they did something to violate our agreement" but they'd have to prove that, and then have a damn good explanation for why they were in the right to suspend the account without any outreach. Unless Railway did something egregious, Google clearly made an error. But that's not what will happen. Google will offer an apology (perhaps even a public one), a giant pile of account credit, and a pinky promise not to do it again. Railway will accept it and hmmm and haw internally about whether to decrease their reliance on GCP, and then when they calculate the cost of going in on other clouds more heavily (or their own metal), they'll just think harder about weird failure modes.
    - danudey 5mo ago
      One company I worked at is highly reliant on Google Cloud, but at one point we moved some services to Azure. Azure noticed, and immediately hit us with a discount offer in the hopes of getting more of our business. Google noticed, and immediately hit us with a discount offer in the hopes of keeping more of our business. This is just a reminder that your multi-cloud strategy doesn't have to be 'deploy everything across multiple clouds'; it can even just be 'make it obvious that you have leverage'.
    - array_key_first 5mo ago
      I'm sure their contract explicitly states that their account can be suspended or terminated immediately without prior notice upon violating some TOS. And, most TOS are incredibly wide and vague, it wouldn't necessarily be hard to find something they violated. This is sort of the problem with these new-age internet companies. The contracts are incredibly hostile. Most TOS you see amount to "you have no rights and we can fuck you up the ass" Google is a B2C company so I'm sure some of that culture transfers over to B2B relations, but I'm speculating. Maybe the contracts are more normal for B2B.
      - cindyllm 5mo ago
        [dead]
      - m11a 5mo ago
        If Railway's account was suspended due to an error, not a TOS violation, I doubt they'll pull such a card. If Google were sued, such a blatant lie would be found in discovery pretty quickly and I doubt a court would look upon it favourably. They'd also be starting a public PR battle that they'd almost certainly lose (assuming there was indeed no real TOS violation) which would make them look completely untrustworthy. So I doubt Google will do that. bastawhiz is probably right in that Google will offer some credits and an apology, and Railway will reduce their dependence (probably), rather than a lawsuit. And I doubt Railway wants to be on the bad side of one of the few big cloud providers. But I'd be surprised if Railway didn't have a good argument to make for compensation or a lawsuit.
      - 5mo ago
- redwood 5mo ago
  I can assure you that Google will be giving them significant commercial incentives as an apology for this behind the scene
quentindanjou 5mo ago
Railway say the incident is resolved but many are still down (returning 502): on our side, we had to manually trigger a redeploy to fix it but I believe it should have been triggered automatically by Railway and I can't understand how they can mark this as resolved while many are still down. In total, down for >11 hours on our side.
gcr 5mo ago
> Railway owns our vendor choices, and we ultimately own this one. Your customers don't care whether the failure was Google or Railway; they see your product. Your uptime is our responsibility, and we'll keep delivering on it. This is an excellent closing statement.
dang 5mo ago
Thanks! The post-mortem is currently on the frontpage here: Incident Report: May 19, 2026 – GCP Account Suspension - https://news.ycombinator.com/item?id=48204770 https://news.ycombinator.com/item?id=48204770

by r721 5mo ago

sschueller 5mo ago

It should be possible to sue Google for damages in such cases. This isnt a network outage or service failure which I would consider part of ToS.

VladVladikoff 5mo ago
What if the reason for their stuff being shut down was a payment issue like an expired credit card or maxed credit account? Unless I missed it skim reading their post I don’t see any information anywhere about their communications with Google.
- bastawhiz 5mo ago
  If you have an account manager and a contract, there's zero excuse for automated suspension. That's literally the whole point of having a dedicated person. From the report: > May 19, 22:22 UTC - P0 ticket filed with Google Cloud. Railway's GCP account manager engaged directly.
Cthulhu_ 5mo ago
It's always possible to sue, but Google has good terms of service and lawyers - I'm 99% confident that a lawsuit would end up nowhere.
- bastawhiz 5mo ago
  They have every right to sue, and if they did sue they almost certainly would win. This is clear breach of contract. The only argument Google could make is "they did something to violate our agreement" but they'd have to prove that, and then have a damn good explanation for why they were in the right to suspend the account without any outreach. Unless Railway did something egregious, Google clearly made an error. But that's not what will happen. Google will offer an apology (perhaps even a public one), a giant pile of account credit, and a pinky promise not to do it again. Railway will accept it and hmmm and haw internally about whether to decrease their reliance on GCP, and then when they calculate the cost of going in on other clouds more heavily (or their own metal), they'll just think harder about weird failure modes.
  - danudey 5mo ago
    One company I worked at is highly reliant on Google Cloud, but at one point we moved some services to Azure. Azure noticed, and immediately hit us with a discount offer in the hopes of getting more of our business. Google noticed, and immediately hit us with a discount offer in the hopes of keeping more of our business. This is just a reminder that your multi-cloud strategy doesn't have to be 'deploy everything across multiple clouds'; it can even just be 'make it obvious that you have leverage'.
  - array_key_first 5mo ago
    I'm sure their contract explicitly states that their account can be suspended or terminated immediately without prior notice upon violating some TOS. And, most TOS are incredibly wide and vague, it wouldn't necessarily be hard to find something they violated. This is sort of the problem with these new-age internet companies. The contracts are incredibly hostile. Most TOS you see amount to "you have no rights and we can fuck you up the ass" Google is a B2C company so I'm sure some of that culture transfers over to B2B relations, but I'm speculating. Maybe the contracts are more normal for B2B.
    - cindyllm 5mo ago
      [dead]
    - m11a 5mo ago
      If Railway's account was suspended due to an error, not a TOS violation, I doubt they'll pull such a card. If Google were sued, such a blatant lie would be found in discovery pretty quickly and I doubt a court would look upon it favourably. They'd also be starting a public PR battle that they'd almost certainly lose (assuming there was indeed no real TOS violation) which would make them look completely untrustworthy. So I doubt Google will do that. bastawhiz is probably right in that Google will offer some credits and an apology, and Railway will reduce their dependence (probably), rather than a lawsuit. And I doubt Railway wants to be on the bad side of one of the few big cloud providers. But I'd be surprised if Railway didn't have a good argument to make for compensation or a lawsuit.
    - 5mo ago
redwood 5mo ago
I can assure you that Google will be giving them significant commercial incentives as an apology for this behind the scene

quentindanjou 5mo ago

Railway say the incident is resolved but many are still down (returning 502): on our side, we had to manually trigger a redeploy to fix it but I believe it should have been triggered automatically by Railway and I can't understand how they can mark this as resolved while many are still down. In total, down for >11 hours on our side.

gcr 5mo ago

> Railway owns our vendor choices, and we ultimately own this one. Your customers don't care whether the failure was Google or Railway; they see your product. Your uptime is our responsibility, and we'll keep delivering on it. This is an excellent closing statement.

dang 5mo ago

Thanks! The post-mortem is currently on the frontpage here: Incident Report: May 19, 2026 – GCP Account Suspension - https://news.ycombinator.com/item?id=48204770 https://news.ycombinator.com/item?id=48204770