5 ms·

I simply don't understand one thing: why such company didn't have/used staged rollout? it's normal that from time to time someone will fuck up, but if they wou

I simply don't understand one thing: why such company didn't have/used staged rollout? it's normal that from time to time someone will fuck up, but if they would rollout this update to only 1% and see that nothing that updated is coming back and bailed out this would be literally 100x less disruptive that what happened

dataflow 2y ago
I've been wondering the same, then wondering: is it possible that they did do a staged rollout, but the cadence was insufficient? The only way for them to receive feedback would be out of band, so they might not have budgeted for the reaction time they should expect.
- another-dave 2y ago
  They'd also have feedback in the sense of "an update was pushed out and none of the updated machines have given us an ACK of 'health: OK' after rebooting".
  - noduerme 2y ago
    Given the fact that the software itself allows basically unfettered remote access to the "endpoint" machines it's installed on, one would think they'd get more than just an ACK when one boots up. But yeah, that would be a reasonably low bar to check on a few dozen devices before bricking everything in the world.
  - notepad0x90 2y ago
    it was a data/content update, not code, so more than likely it may not crash immediately. how long would the test vm/container wait before it gives the green light on uptime? The buggy code would have to process it after some time/trigger and other factors for the bsod to show. They may have not done any testing at all because they were adding a named pipe to be monitored, kind of like adding a domain or ip to be monitored.
dannyw 2y ago
I’ve heard quite a few, unfortunately hearsay, reports that CrowdStrike’s engineering practices are not good. Stories I’ve heard: • Developers building binaries on their own local systems (identified via metadata) that are then pushed out globally. Inference: 1 xz-insider = global ring0. • Customer success teams sending 50 page reports on why they can’t do staged rollout (“respond to 0days”), but reality is they’re unwilling to invest in the eng infra to do so. • When they broke Linux endpoints a few months ago, “post mortem” was ignored and nothing remedied.
- dist-epoch 2y ago
  > “post mortem” was ignored and nothing remedied. Losing billions of dollars in stock market cap, recovery costs, and possible law-suits tends to focus the mind on these things.
  - thunky 2y ago
    > Losing billions of dollars in stock market cap Still up 20% ytd from an already high valuation, so they have a long way to fall. If they don't fall all the way to zero (spoiler alert: they won't) they got off easy.
notepad0x90 2y ago
my impression is, they don't test content updates the same way they do code updates or at least not in the same scale and test cases.

dataflow 2y ago

I've been wondering the same, then wondering: is it possible that they did do a staged rollout, but the cadence was insufficient? The only way for them to receive feedback would be out of band, so they might not have budgeted for the reaction time they should expect.

another-dave 2y ago
They'd also have feedback in the sense of "an update was pushed out and none of the updated machines have given us an ACK of 'health: OK' after rebooting".
- noduerme 2y ago
  Given the fact that the software itself allows basically unfettered remote access to the "endpoint" machines it's installed on, one would think they'd get more than just an ACK when one boots up. But yeah, that would be a reasonably low bar to check on a few dozen devices before bricking everything in the world.
- notepad0x90 2y ago
  it was a data/content update, not code, so more than likely it may not crash immediately. how long would the test vm/container wait before it gives the green light on uptime? The buggy code would have to process it after some time/trigger and other factors for the bsod to show. They may have not done any testing at all because they were adding a named pipe to be monitored, kind of like adding a domain or ip to be monitored.

dannyw 2y ago

I’ve heard quite a few, unfortunately hearsay, reports that CrowdStrike’s engineering practices are not good. Stories I’ve heard: • Developers building binaries on their own local systems (identified via metadata) that are then pushed out globally. Inference: 1 xz-insider = global ring0. • Customer success teams sending 50 page reports on why they can’t do staged rollout (“respond to 0days”), but reality is they’re unwilling to invest in the eng infra to do so. • When they broke Linux endpoints a few months ago, “post mortem” was ignored and nothing remedied.

dist-epoch 2y ago
> “post mortem” was ignored and nothing remedied. Losing billions of dollars in stock market cap, recovery costs, and possible law-suits tends to focus the mind on these things.
- thunky 2y ago
  > Losing billions of dollars in stock market cap Still up 20% ytd from an already high valuation, so they have a long way to fall. If they don't fall all the way to zero (spoiler alert: they won't) they got off easy.

notepad0x90 2y ago

my impression is, they don't test content updates the same way they do code updates or at least not in the same scale and test cases.