4 ms·

Would you want your vendors publicly disclosing potentially private reasons for an outage?

by ceejayoz 6mo ago

hun3 6mo ago
With consent, yes.
- ceejayoz 6mo ago
  Is there any indication Railway has consented to such disclosure?
  - mlmonkey 6mo ago
    But Railway has been blaming GCP for the outage. Shouldn't GCP be given an opportunity to defend itself?
    - ceejayoz 6mo ago
      Is that what you'd want your vendors to do?
    - SoftTalker 6mo ago
      Railway hasn't placed any blame that I've seen. They've posted a timeline of what happened, without any speculation on causes.
    - quietsegfault 6mo ago
      No, that’s what the money is for.
deleted 6mo ago
[deleted]
r_lee 6mo ago
yes, especially since this didn't seem to be exactly "private" where it was anything specific, it was just some kind of automated system without a human in the loop
- ceejayoz 6mo ago
  But shouldn't that be disclosed to Railway, and not the public? If they had someone running a botnet on compromised accounts there, for example. If Railway isn't satisfied with the explanation, they're able to say so publicly, yes?
  - r_lee 6mo ago
    if it wasn't something specific to their setup, it should be disclosed publicly, because this is a catastrophic incident that makes you think it could happen to you as well, and there's no way to know what could trigger it
    - ceejayoz 6mo ago
      > if it wasn't something specific to their setup They're a web host; it could be any number of plausible mundane things that triggered automated action. This is a big recurring problem for any shared hosting provider.
      - r_lee 6mo ago
        a huge account like theirs should not be subject to automated actions like that. an entire gcp project deleted along with its persistent disks. how does that make any sense? nobody thought to call them or anything
        ceejayoz 6mo ago
        > a huge account like theirs should not be subject to automated actions like that. No matter how damaging the behavior? > an entire gcp project deleted along with its persistent disks. Railway doesn't say that - "persistent disks inaccessible", followed by "persistent disks restored to ready state". It was a suspension, not a wipe.
        r_lee 6mo ago
x0x0 6mo ago
If it were google's bug as Railway has certainly at least insinuated, then yes. I would also be fine with them saying "blah blah blah abuse was detected; we're working through it with our customer and we apologize to those impacted." I'd also expect a story around how it is this happened w/o a human spending at least an hour working his/her way through a call list to reach someone at Railway. Starting with ops and escalating to the ceo if necessary.

Would you want your vendors publicly disclosing potentially private reasons for an outage?

hun3 6mo ago

With consent, yes.

ceejayoz 6mo ago
Is there any indication Railway has consented to such disclosure?
- mlmonkey 6mo ago
  But Railway has been blaming GCP for the outage. Shouldn't GCP be given an opportunity to defend itself?
  - ceejayoz 6mo ago
    Is that what you'd want your vendors to do?
  - SoftTalker 6mo ago
    Railway hasn't placed any blame that I've seen. They've posted a timeline of what happened, without any speculation on causes.
  - quietsegfault 6mo ago
    No, that’s what the money is for.

deleted 6mo ago

[deleted]

r_lee 6mo ago

yes, especially since this didn't seem to be exactly "private" where it was anything specific, it was just some kind of automated system without a human in the loop

ceejayoz 6mo ago
But shouldn't that be disclosed to Railway, and not the public? If they had someone running a botnet on compromised accounts there, for example. If Railway isn't satisfied with the explanation, they're able to say so publicly, yes?
- r_lee 6mo ago
  if it wasn't something specific to their setup, it should be disclosed publicly, because this is a catastrophic incident that makes you think it could happen to you as well, and there's no way to know what could trigger it
  - ceejayoz 6mo ago
    > if it wasn't something specific to their setup They're a web host; it could be any number of plausible mundane things that triggered automated action. This is a big recurring problem for any shared hosting provider.
    - r_lee 6mo ago
      a huge account like theirs should not be subject to automated actions like that. an entire gcp project deleted along with its persistent disks. how does that make any sense? nobody thought to call them or anything
      - ceejayoz 6mo ago
        > a huge account like theirs should not be subject to automated actions like that. No matter how damaging the behavior? > an entire gcp project deleted along with its persistent disks. Railway doesn't say that - "persistent disks inaccessible", followed by "persistent disks restored to ready state". It was a suspension, not a wipe.
        r_lee 6mo ago

x0x0 6mo ago

If it were google's bug as Railway has certainly at least insinuated, then yes. I would also be fine with them saying "blah blah blah abuse was detected; we're working through it with our customer and we apologize to those impacted." I'd also expect a story around how it is this happened w/o a human spending at least an hour working his/her way through a call list to reach someone at Railway. Starting with ops and escalating to the ceo if necessary.