5 ms·

How does a company this big not have automated tests for their config files, and not have gradual/staggered rollouts for their deployments? Is there some good

How does a company this big not have automated tests for their config files, and not have gradual/staggered rollouts for their deployments? Is there some good reason for this approach (need to get config updates into the wild as quickly as possible to combat zero-days or zero-hours?) or was this just a massive oversight? Side rant... their postmortem took forever to get to the point, first explaining all their jargon and product names. Makes me really appreciate the Cloudflare ones.

blackeyeblitzar 2y ago
Often this is not because the team doesn’t know about these things but because they have low staffing or other priorities or deadlines. This event looks to me like company rot that can be laid at the CEO’s feet
lupusreal 2y ago
My bet is they have some normal process for updates that has testing but that process is only enforced by policy, not code, and somebody simply decided it was a waste of their time.
hakdbha 2y ago
[dead]
geodel 2y ago
Yeah, management should sternly tell All code / config must be tested before deployment to prod. Millions of companies have issued this order and after that they are running free of problem for decades. It is so straightforward and it always works.
- toomuchtodo 2y ago
  Without regulation, best practices are simply opinions and suggestions. “Please do” is insufficient for critical infrastructure. See: financial infra regulatory apparatus. Incentives, outcomes, the usual.
  - specialist 2y ago
    Regulations bad. The invisible foot of Freedom Markets™ will fix exactly these kinds of market failures. Rationally speaking.
    - paulddraper 2y ago
      Hand, but yes.
devoutsalsa 2y ago
How many conversations have you had with people at work about how something was a bad idea, only to unsuccessfully avoid it?
- jvanderbot 2y ago
  And conversely, how many good ideas have you proposed to have them put on "next quarter"'s schedule?
  - red-iron-pine 2y ago
    features get pushed to next quarter all the time. tag some JIRA L2s for [Future Sprint Owner] to worry about. But you don't release until QA is done, esp. if you're touching safety-critical systems. Turns out CS didn't realize they ran safety critical systems in airlines and hospitals.
    - jvanderbot 2y ago
      That's... not quite what I meant. What I meant was, How many times do you think someone proposed: We should harden our code / delivery in case the system is important Only to hear We can add robustness later, we will patch it in Terms of Service "for now"
- Affric 2y ago
  And because you knew it was coming you have the fix ready that you didn’t get paid any extra for and then the project gets declared a success by C-suite and the idiots who devised and it and almost destroyed the business get rewarded…
- EasyMark 2y ago
  If you suggested we let through potentially bricking level changes without 100% testing coverage by basic testing including reboot testing on many different hardware/VM combinations then I would fire you in that same meeting if you weren't joking with me (and I was in a position to do it)
rfoo 2y ago
> not have automated tests for their config files They very likely have automated tests. However, what if bug only triggers 90% of the time and you hit the lucky 10% during automated tests? Of course you can run tests 100 times but... is this a common practice? Moreover, we have both code and anecdotal evidences that the bug may indeed happen randomly. Tavis Ormandy posted a rough analysis of the crash context: https://x.com/taviso/status/1814762302337654829 https://x.com/taviso/status/1814762302337654829. It looks like the crash is caused by first checking if an uninitialized pointer is NULL, and if not, dereferencing it. If the uninitialized leftover data just happened to be zero, no crash happens. And anecdotally, we saw people reporting that repeatedly rebooting their machines for 15+ times fixed the problem for them - because eventually you got lucky and in a boot it didn't happen and CrowdStrike managed to update itself to not crash. > not have gradual/staggered rollouts for their deployments No idea. Maybe their poor reliability guy got overrided by another team, like "how dare you delaying our important definition update? we're racing with threat actors!". I hope they learned their lesson.
- pantalaimon 2y ago
  The crash was triggered by a config file that just contained null bytes as payload
  - rfoo 2y ago
    This is a false rumor. Please stop spreading misinformation.
- zelphirkalt 2y ago
  Running tests 100 times only helps, if you have some sufficiently randomized input data, so that the issue can happen.
chasd00 2y ago
hey they're moving fast and breaking things..
51Cards 2y ago
I am wondering if the update passed the test farm just fine, but when the file was moved to the update distribution system that's when the issue happened. The file copied as all nulls and there was no validation check that the file posted ok. Compounded by no validation check on the file after downloading by the end system. Compounded by not having a staged roll-out process for updates.
kongolongo 2y ago
>Is there some good reason for this approach (need to get config updates into the wild as quickly as possible to combat zero-days or zero-hours?) or was this just a massive oversight? I'm curious of this too. Has there ever been a scenario where zero-days could have caused so much damage that it'd warrant this speediness in patching? In a cost-benefit analysis would it justify x% of patches like this happening in preventing whatever security issues could occur without this type of infrastructure?

blackeyeblitzar 2y ago

Often this is not because the team doesn’t know about these things but because they have low staffing or other priorities or deadlines. This event looks to me like company rot that can be laid at the CEO’s feet

lupusreal 2y ago

My bet is they have some normal process for updates that has testing but that process is only enforced by policy, not code, and somebody simply decided it was a waste of their time.

hakdbha 2y ago

[dead]

geodel 2y ago

Yeah, management should sternly tell All code / config must be tested before deployment to prod. Millions of companies have issued this order and after that they are running free of problem for decades. It is so straightforward and it always works.

toomuchtodo 2y ago
Without regulation, best practices are simply opinions and suggestions. “Please do” is insufficient for critical infrastructure. See: financial infra regulatory apparatus. Incentives, outcomes, the usual.
- specialist 2y ago
  Regulations bad. The invisible foot of Freedom Markets™ will fix exactly these kinds of market failures. Rationally speaking.
  - paulddraper 2y ago
    Hand, but yes.

devoutsalsa 2y ago

How many conversations have you had with people at work about how something was a bad idea, only to unsuccessfully avoid it?

jvanderbot 2y ago
And conversely, how many good ideas have you proposed to have them put on "next quarter"'s schedule?
- red-iron-pine 2y ago
  features get pushed to next quarter all the time. tag some JIRA L2s for [Future Sprint Owner] to worry about. But you don't release until QA is done, esp. if you're touching safety-critical systems. Turns out CS didn't realize they ran safety critical systems in airlines and hospitals.
  - jvanderbot 2y ago
    That's... not quite what I meant. What I meant was, How many times do you think someone proposed: We should harden our code / delivery in case the system is important Only to hear We can add robustness later, we will patch it in Terms of Service "for now"
Affric 2y ago
And because you knew it was coming you have the fix ready that you didn’t get paid any extra for and then the project gets declared a success by C-suite and the idiots who devised and it and almost destroyed the business get rewarded…
EasyMark 2y ago
If you suggested we let through potentially bricking level changes without 100% testing coverage by basic testing including reboot testing on many different hardware/VM combinations then I would fire you in that same meeting if you weren't joking with me (and I was in a position to do it)

rfoo 2y ago

> not have automated tests for their config files They very likely have automated tests. However, what if bug only triggers 90% of the time and you hit the lucky 10% during automated tests? Of course you can run tests 100 times but... is this a common practice? Moreover, we have both code and anecdotal evidences that the bug may indeed happen randomly. Tavis Ormandy posted a rough analysis of the crash context: https://x.com/taviso/status/1814762302337654829 https://x.com/taviso/status/1814762302337654829. It looks like the crash is caused by first checking if an uninitialized pointer is NULL, and if not, dereferencing it. If the uninitialized leftover data just happened to be zero, no crash happens. And anecdotally, we saw people reporting that repeatedly rebooting their machines for 15+ times fixed the problem for them - because eventually you got lucky and in a boot it didn't happen and CrowdStrike managed to update itself to not crash. > not have gradual/staggered rollouts for their deployments No idea. Maybe their poor reliability guy got overrided by another team, like "how dare you delaying our important definition update? we're racing with threat actors!". I hope they learned their lesson.

pantalaimon 2y ago
The crash was triggered by a config file that just contained null bytes as payload
- rfoo 2y ago
  This is a false rumor. Please stop spreading misinformation.
zelphirkalt 2y ago
Running tests 100 times only helps, if you have some sufficiently randomized input data, so that the issue can happen.

chasd00 2y ago

hey they're moving fast and breaking things..

51Cards 2y ago

I am wondering if the update passed the test farm just fine, but when the file was moved to the update distribution system that's when the issue happened. The file copied as all nulls and there was no validation check that the file posted ok. Compounded by no validation check on the file after downloading by the end system. Compounded by not having a staged roll-out process for updates.

kongolongo 2y ago

>Is there some good reason for this approach (need to get config updates into the wild as quickly as possible to combat zero-days or zero-hours?) or was this just a massive oversight? I'm curious of this too. Has there ever been a scenario where zero-days could have caused so much damage that it'd warrant this speediness in patching? In a cost-benefit analysis would it justify x% of patches like this happening in preventing whatever security issues could occur without this type of infrastructure?