4 ms·

To be sure. But the fact is the release broke. I'm not sure: is having test servers that it passed any better than none at all?

martinky24 2y ago
Yes, yes it is. Because there's tons more breakages that have likely been caught. One uncaught downstream failure doesn't invalidate the effort into all the previously caught failures.
- JKCalhoun 2y ago
  At the same time (and I am looking directly at Unit Tests when I say this) I have seen what is perhaps dangerous confidence that, because such and such tests are in place, we can be a more lax when pushing out changes.
strken 2y ago
It is absolutely better to catch some errors than none. In this case it gives me vibes of something going wrong after the CI pipeline, during the rollout. Maybe they needed advice a bit more specific than "just use a staging environment bro", like "use checksums to verify a release was correctly applied before cutting over to the new definitions" and "do staged rollouts, and ideally release to some internal canary servers first".
- exe34 2y ago
  I don't understand why you wouldn't do staged roll outs at this scale. even a few hours delay might have been enough to stop the release from going global.
- martinky24 2y ago
  "Have these idiots even heard of CI/CD???" strangely seems to be a common condescending comment in this thread. I honestly though HN was slightly higher quality than most of the comments here. I am proven wrong.
  - kristjansson 2y ago
    Big threads draw a lot of people; we regress toward the mean
  - StressedDev 2y ago
    Agreed - The worst part is most of the people making these unhelpful comments are probably doing the same sorts of things which caused this outage.
  - chuckadams 2y ago
    > I honestly though HN was slightly higher quality HN reminds me of nothing so much as Slashdot in the early 2000's, for both good and ill. Fewer stupid memes about Beowulf Clusters and Natalie Portman tho.
- chuckadams 2y ago
  They almost certainly have such a process, but it got bypassed by accident, probably got put into a "minor updates" channel (you don't run your model checker every time you release a new signature file after all). Surprise, business processes have bugs too. But naw, must be every random commentator on HN knows how to run the company better.
  - TeMPOraL 2y ago
    > (you don't run your model checker every time you release a new signature file after all) Wonder if the higher-ups who mandated this software to be installed in their hospitals were informed about that fact.
chatmasta 2y ago
The release didn’t break. A data file containing nulls was downloaded by a buggy kernel module that crashed when reading the file. For all we know there is a test case that failed and they decided to push the module anyway (“it’s not like anyone is gonna upload a file of all nulls”). Btw: where are these files sourced from? Could a malicious Crowdstrike customer trick the system into generating this data file, by e.g. reporting it saw malware with these (null) signatures?
leptons 2y ago
A lot of the software industry focuses on strong types, testing of all kinds, linting, and plenty of other sideshows that make programmers feel like they're in control, but these things only account for the problems you can test for and the systems you control. So what if a function gets a null instead of a float? It shouldn't crash half the tech-connected world. Software resilience is kind of lacking in favor of trusting that strong types and tests will catch most bugs, and that's good enough?

martinky24 2y ago

Yes, yes it is. Because there's tons more breakages that have likely been caught. One uncaught downstream failure doesn't invalidate the effort into all the previously caught failures.

JKCalhoun 2y ago
At the same time (and I am looking directly at Unit Tests when I say this) I have seen what is perhaps dangerous confidence that, because such and such tests are in place, we can be a more lax when pushing out changes.

strken 2y ago

It is absolutely better to catch some errors than none. In this case it gives me vibes of something going wrong after the CI pipeline, during the rollout. Maybe they needed advice a bit more specific than "just use a staging environment bro", like "use checksums to verify a release was correctly applied before cutting over to the new definitions" and "do staged rollouts, and ideally release to some internal canary servers first".

exe34 2y ago
I don't understand why you wouldn't do staged roll outs at this scale. even a few hours delay might have been enough to stop the release from going global.
martinky24 2y ago
"Have these idiots even heard of CI/CD???" strangely seems to be a common condescending comment in this thread. I honestly though HN was slightly higher quality than most of the comments here. I am proven wrong.
- kristjansson 2y ago
  Big threads draw a lot of people; we regress toward the mean
- StressedDev 2y ago
  Agreed - The worst part is most of the people making these unhelpful comments are probably doing the same sorts of things which caused this outage.
- chuckadams 2y ago
  > I honestly though HN was slightly higher quality HN reminds me of nothing so much as Slashdot in the early 2000's, for both good and ill. Fewer stupid memes about Beowulf Clusters and Natalie Portman tho.
chuckadams 2y ago
They almost certainly have such a process, but it got bypassed by accident, probably got put into a "minor updates" channel (you don't run your model checker every time you release a new signature file after all). Surprise, business processes have bugs too. But naw, must be every random commentator on HN knows how to run the company better.
- TeMPOraL 2y ago
  > (you don't run your model checker every time you release a new signature file after all) Wonder if the higher-ups who mandated this software to be installed in their hospitals were informed about that fact.

chatmasta 2y ago

The release didn’t break. A data file containing nulls was downloaded by a buggy kernel module that crashed when reading the file. For all we know there is a test case that failed and they decided to push the module anyway (“it’s not like anyone is gonna upload a file of all nulls”). Btw: where are these files sourced from? Could a malicious Crowdstrike customer trick the system into generating this data file, by e.g. reporting it saw malware with these (null) signatures?

leptons 2y ago

A lot of the software industry focuses on strong types, testing of all kinds, linting, and plenty of other sideshows that make programmers feel like they're in control, but these things only account for the problems you can test for and the systems you control. So what if a function gets a null instead of a float? It shouldn't crash half the tech-connected world. Software resilience is kind of lacking in favor of trusting that strong types and tests will catch most bugs, and that's good enough?