4 ms·

> Look at the Railway GCP account ban situation. A literally billion dollar startup running on Google Cloud and Google just randomly snaps their fingers and del

by raggi 5mo ago

> Look at the Railway GCP account ban situation. A literally billion dollar startup running on Google Cloud and Google just randomly snaps their fingers and deletes their account. Zero warning. No phone number to call. No account rep. Poof. Gone. It is actual insanity to me. A billion dollar customer gets the exact same automated middle finger as a low effort spam bot. Your B2B business is completely cooked if that is how you treat people. The enterprise cloud gravy train is right there and Google is standing on the tracks begging to get hit by the train. If you've been around a while you know that at any business critical scale at all you establish a relationship with your cloud provider and get an account manager. When you do this, you have a number to call. A billion dollar startup not doing this is a keen lesson for the CTO. Yes, Google likely screwed up here, but being unprepared for account problems, having no established relationship with your provider is a critical mistake. The article goes on to talk about Hetzner as an example: their pricing is great for individuals but they literally don't even offer account management relationships - even at scale they actively refuse them. There are equivalent stories of account terminations with Hetzner, which is also a key point: this isn't just a big business problem, at all.

aurareturn 5mo ago
I would think that Railway certainly had an account manager, right? Did they say they didn’t have one?
- MrDarcy 5mo ago
  The article is incorrect and misleading. Railway did have an account manager and they did call them and they did pick up the phone and work with them to restore service.
  - Forgeties79 5mo ago
    An account manager overseeing such a major client should’ve never let this happen. If they don’t, why the hell are they the account manager? What are they even doing to earn their keep? This was such a preventable situation. I once worked at a company that had their domain lapse because of an internal error at the company that was paid for the domain. There was no alert, there was no attempt to rectify the situation, one day we woke up and we simply did not have control of our website for a full week. There was nothing wrong with our payment method, there was no reason the payment shouldn’t have happened, it was completely their fault. They found out because we called them in a panic. This was a major company. We left them a week later and our CEO talks about it constantly as a horror story to other companies and there is no way our situation was unique. It’s not just about the value of the contract. This whole situation has been in the news for days now. It’s terrible PR and I guarantee you it is costing them business in the long run. All I have seen for days is people talking about how poor Google’s support is, and I’m not even somebody who makes those decisions. I get it, “Google is too big to fail.” But eventually, that stops being true
    - esrauch 5mo ago
      I'm not sure what you think account managers do that they can prevent accidents/bugs like that?
      - cco 5mo ago
        Typically a strong account team builds processes with other teams (compliance, engineering, etc) that enshrines and insulates important accounts from accidents like this. In this case, I'd expect major accounts (and maybe Railway isn't above this level?) to be in a protected tier that is immune from automated suspensions like this. If suspicious traffic occurs that _would_ trigger a suspension like that, the account team would be paged. Because this may mean your important account was compromised, shipped a bug, has been hit by something and you should immediately start working _with_ them to figure it out. Fairly basic for a company with any customer management motion at all.
        thibaut_barrere 5mo ago
        See https://news.ycombinator.com/item?id=48231538 https://news.ycombinator.com/item?id=48231538, they appeared to have everything you can have, and it wasn’t enough to prevent this.
        cco 5mo ago
        Oh I was just describing to GP what a strong customer motion would do to protect against these things, not commenting on what Railway had or didn't. But clearly GCP _doesn't_ have a strong customer motion or this story wouldn't have happened.
      - deleted 5mo ago
        [deleted]
      - deleted 5mo ago
        [deleted]
    - ralph84 5mo ago
      The account manager was almost surely just as blindsided as the customer.
  - sreekanth850 5mo ago
    What is the use of Account manager if a billion dollar company got screwedup for straight 5 + hours?
    - aurareturn 5mo ago
      Without an account manager, you submit a ticket and no one looks at it for a few days. With an account manager, you call this person up immediately, and they hound the devs in person or call up the devs personally.
      - sreekanth850 5mo ago
        If iam an enterprise customer, the least i expect will be a prior communication and a warning email before account termination. Is this too much?
        aurareturn 5mo ago
        This problem likely had nothing to do with the account manager though. I doubt it was the account manager making the call to suspend the account.
sylens 5mo ago
Our team at work uses all three major public clouds and our Google Account Team is by far the worst of the three. Nothing like having to explain the same problem I’m having from the start again on every monthly cadence call because they don’t note anything or try to help resolve it
- verst 5mo ago
  Who is the best? Just curious.
  - jvuygbbkuurx 5mo ago
    Microsoft of course
    - isaisabella 5mo ago
      Why not Oracle
      - smallmancontrov 5mo ago
        Because Oracle = Larry Ellison = The Lawnmower > You need to think of Larry Ellison the way you think of a lawnmower. You don’t anthropomorphize your lawnmower, the lawnmower just mows the lawn - you stick your hand in there and it’ll chop it off, the end. You don’t think "oh, the lawnmower hates me" – lawnmower doesn’t give a shit about you, lawnmower can’t hate you. Don’t anthropomorphize the lawnmower. Don’t fall into that trap about Oracle. -Bryan Cantrill A few years back someone at work stuck their hand in the lawnmower. I've seen this happen a few times, but that time it ended with Oracle fining us for every VirtualBox install and the company sent The IT Spanish Inquisition around to make sure we all deleted it off our computers. Fun times.
        senkora 5mo ago
        You honestly just have to treat any Oracle product as malware, and proactively scan for it / block it from being installed on employee laptops in the first place.
sergiotapia 5mo ago
You are completely misinformed here. Railway had dedicated reps and everything. Even the number of the head. One of their railway company leads said so on X or their Discord during the outage. They paid for everything you can pay for to get Google on the horn and ready because they are not a tiny company. Yet it was still terrible.
- lern_too_spel 5mo ago
  Spot on. This is the wrong takeaway. The account manager can't do anything to prevent the problem. They can only escalate, and that's what they did. The right takeaway is you should not use Google Cloud because Google itself doesn't use Google Cloud for anything critical. I don't know how many times I've said this, but it clearly hasn't sunk in. A Googler once responded, "Well ackshually, Google Domains is built on Google Cloud." Google Domains isn't mission critical and has since been sold to Squarespace.
xbar 5mo ago
Victim-blaming is not appropriate here. Google did not "likely" screw up. To be sure, however, this is a keen lesson for every CTO.
- raggi 5mo ago
  i was responding to the article, i hadn't bothered to read railways pm as it's not super interesting, anyway, i've read it now, here: https://blog.railway.com/p/incident-report-may-19-2026-gcp-account-outage https://blog.railway.com/p/incident-report-may-19-2026-gcp-a... and indeed they weren't at fault
RamblingCTO 5mo ago
> account management relationships in my experience hetzner does have that. my VP of infra had regular contact with them, was quite important actually because we had scaling needs they couldn't deliver on and we had timelines. hetzner still has issues getting enough intel servers on it seems
dzonga 5mo ago
& people wonder why Microsoft quickly gained ground in Cloud. they know how to support businesses - small or large. better support if you're enterprise. hell M$ can even have people at your place in an instant - not the "forward-deployed engineer" propaganda being spread around by SV these days.

by raggi 5mo ago

aurareturn 5mo ago

I would think that Railway certainly had an account manager, right? Did they say they didn’t have one?

MrDarcy 5mo ago
The article is incorrect and misleading. Railway did have an account manager and they did call them and they did pick up the phone and work with them to restore service.
- Forgeties79 5mo ago
  An account manager overseeing such a major client should’ve never let this happen. If they don’t, why the hell are they the account manager? What are they even doing to earn their keep? This was such a preventable situation. I once worked at a company that had their domain lapse because of an internal error at the company that was paid for the domain. There was no alert, there was no attempt to rectify the situation, one day we woke up and we simply did not have control of our website for a full week. There was nothing wrong with our payment method, there was no reason the payment shouldn’t have happened, it was completely their fault. They found out because we called them in a panic. This was a major company. We left them a week later and our CEO talks about it constantly as a horror story to other companies and there is no way our situation was unique. It’s not just about the value of the contract. This whole situation has been in the news for days now. It’s terrible PR and I guarantee you it is costing them business in the long run. All I have seen for days is people talking about how poor Google’s support is, and I’m not even somebody who makes those decisions. I get it, “Google is too big to fail.” But eventually, that stops being true
  - esrauch 5mo ago
    I'm not sure what you think account managers do that they can prevent accidents/bugs like that?
    - cco 5mo ago
      Typically a strong account team builds processes with other teams (compliance, engineering, etc) that enshrines and insulates important accounts from accidents like this. In this case, I'd expect major accounts (and maybe Railway isn't above this level?) to be in a protected tier that is immune from automated suspensions like this. If suspicious traffic occurs that _would_ trigger a suspension like that, the account team would be paged. Because this may mean your important account was compromised, shipped a bug, has been hit by something and you should immediately start working _with_ them to figure it out. Fairly basic for a company with any customer management motion at all.
      - thibaut_barrere 5mo ago
        See https://news.ycombinator.com/item?id=48231538 https://news.ycombinator.com/item?id=48231538, they appeared to have everything you can have, and it wasn’t enough to prevent this.
        cco 5mo ago
        Oh I was just describing to GP what a strong customer motion would do to protect against these things, not commenting on what Railway had or didn't. But clearly GCP _doesn't_ have a strong customer motion or this story wouldn't have happened.
    - deleted 5mo ago
      [deleted]
    - deleted 5mo ago
      [deleted]
  - ralph84 5mo ago
    The account manager was almost surely just as blindsided as the customer.
- sreekanth850 5mo ago
  What is the use of Account manager if a billion dollar company got screwedup for straight 5 + hours?
  - aurareturn 5mo ago
    Without an account manager, you submit a ticket and no one looks at it for a few days. With an account manager, you call this person up immediately, and they hound the devs in person or call up the devs personally.
    - sreekanth850 5mo ago
      If iam an enterprise customer, the least i expect will be a prior communication and a warning email before account termination. Is this too much?
      - aurareturn 5mo ago
        This problem likely had nothing to do with the account manager though. I doubt it was the account manager making the call to suspend the account.

sylens 5mo ago

Our team at work uses all three major public clouds and our Google Account Team is by far the worst of the three. Nothing like having to explain the same problem I’m having from the start again on every monthly cadence call because they don’t note anything or try to help resolve it

verst 5mo ago
Who is the best? Just curious.
- jvuygbbkuurx 5mo ago
  Microsoft of course
  - isaisabella 5mo ago
    Why not Oracle
    - smallmancontrov 5mo ago
      Because Oracle = Larry Ellison = The Lawnmower > You need to think of Larry Ellison the way you think of a lawnmower. You don’t anthropomorphize your lawnmower, the lawnmower just mows the lawn - you stick your hand in there and it’ll chop it off, the end. You don’t think "oh, the lawnmower hates me" – lawnmower doesn’t give a shit about you, lawnmower can’t hate you. Don’t anthropomorphize the lawnmower. Don’t fall into that trap about Oracle. -Bryan Cantrill A few years back someone at work stuck their hand in the lawnmower. I've seen this happen a few times, but that time it ended with Oracle fining us for every VirtualBox install and the company sent The IT Spanish Inquisition around to make sure we all deleted it off our computers. Fun times.
      - senkora 5mo ago
        You honestly just have to treat any Oracle product as malware, and proactively scan for it / block it from being installed on employee laptops in the first place.

sergiotapia 5mo ago

You are completely misinformed here. Railway had dedicated reps and everything. Even the number of the head. One of their railway company leads said so on X or their Discord during the outage. They paid for everything you can pay for to get Google on the horn and ready because they are not a tiny company. Yet it was still terrible.

lern_too_spel 5mo ago
Spot on. This is the wrong takeaway. The account manager can't do anything to prevent the problem. They can only escalate, and that's what they did. The right takeaway is you should not use Google Cloud because Google itself doesn't use Google Cloud for anything critical. I don't know how many times I've said this, but it clearly hasn't sunk in. A Googler once responded, "Well ackshually, Google Domains is built on Google Cloud." Google Domains isn't mission critical and has since been sold to Squarespace.

xbar 5mo ago

Victim-blaming is not appropriate here. Google did not "likely" screw up. To be sure, however, this is a keen lesson for every CTO.

raggi 5mo ago
i was responding to the article, i hadn't bothered to read railways pm as it's not super interesting, anyway, i've read it now, here: https://blog.railway.com/p/incident-report-may-19-2026-gcp-account-outage https://blog.railway.com/p/incident-report-may-19-2026-gcp-a... and indeed they weren't at fault

RamblingCTO 5mo ago

> account management relationships in my experience hetzner does have that. my VP of infra had regular contact with them, was quite important actually because we had scaling needs they couldn't deliver on and we had timelines. hetzner still has issues getting enough intel servers on it seems

dzonga 5mo ago

& people wonder why Microsoft quickly gained ground in Cloud. they know how to support businesses - small or large. better support if you're enterprise. hell M$ can even have people at your place in an instant - not the "forward-deployed engineer" propaganda being spread around by SV these days.