6 ms·

It wasn't software update. It was signature database update. It's supposed to roll out as fast as possible. When you learn about new virus, it's already in the

It wasn't software update. It was signature database update. It's supposed to roll out as fast as possible. When you learn about new virus, it's already in the wild, so every minute counts. You don't want to delay update for a day just to find out that your servers were breached 20 hours ago.

TeMPOraL 2y ago
We can see clearly now that this is a stupid approach. Viruses don't move that fast. This situation is akin to the immune system overreacting and melting the patient in response to a papercut. This sometimes happens, but it's considered a serious medical condition, and I believe the treatment is to nuke someone's immune system entirely with hard radiation, and reinstall a less aggressive copy. Take from that analogy what you want.
- orf 2y ago
  > Viruses don't move that fast Yes they do? And it’s more akin to a shared immune system than a single organism. In this case, it’s not like viruses move fast relative to the total population of machines, but within the population of machines being targeted they do move fast.
  - TeMPOraL 2y ago
    Still, better to let them spread a bit and deal with the localized damage than risk nuking everything. There is such a thing as treatment that's very effective, but not used because of a low probability risk of terminal damage.
  - proveitbh 2y ago
    Cite one virus thay crashed the supposed 10 or 100 million machines in 70 minutes. Just one.
    - orf 2y ago
      Microsoft puts the count at 8.5 million computers. So, percentage wise, the MyDoom virus in 2004 infected a far greater % of computers in a month: which in the context of internet penetration, availability and speeds (40kb/s average, 450kb/s fastest) in 2004 was about as fast as it could have. So it might as well have been 70 minutes, given downloading a 50mb file on dial up would take way longer than 70 mins. To the smart people below: It’s clear to everyone that 70 minutes is not 1 month. The point is that it’s not a fair comparison: it would simply not have been possible to infect that many computers in 70 minutes: the internet infrastructure just wasn’t there. It’s like saying “the Spanish flu didn’t do that much damage because there where less people on the planet” - it’s a meaningless absolute comparison, whereas the relative comparison is what matters.
      - lolilokol 2y ago
        [flagged]
      - klijibbn 2y ago
        [flagged]
      - smartpeoplebelw 2y ago
        There's also orders of magnitudes more machines today than 20 years ago -- so it should be easier to infect more machines now than before, and yet no one can sight a virus that was as quickly moving and damaging as what crowdstrike did through gross negligence. Be better.
        orf 2y ago
        This entire thread is stupid. Computer security as a whole has improved, whilst the complexity of interconnected systems has exponentially increased. This has made the barrier to entry for malware higher, and so means we no longer have the same historic examples of large scale worms targeting consumer machines that we used to. At the same time the financial rewards for finding and exploiting a vulnerability within an organisations complex stack have greatly increased. The rewards are coupled to the time it takes to execute on the vulnerability. This leads to what we have today: localised, and often specialised attacks against valuable targets that are executed as fast as possible in order to minimise the chance a target has to respond or the vulnerability they are exploiting to be burned. Of course the “smart people belw” must know this, so it’s unclear why they are pretending to be dumb.
  - aldousd666 2y ago
    No they're in the wrong. They didn't test adequately, regardless of their motive for not doing so. Obviously reality is not backing up your theory there
    - orf 2y ago
      FYI, both the following statements can be true: 1. Crowdstrike didn’t test adequately 2. Viruses can move pretty fast once a foothold is gained
- UncleMeat 2y ago
  https://en.wikipedia.org/wiki/SQL_Slammer https://en.wikipedia.org/wiki/SQL_Slammer There is no real "speed limit" on malware spread.
  - TeMPOraL 2y ago
    No, but there are impenetrable barriers. 0days in paricular are usually very specific and affect few systems directly but even the broader ones aren't usually followed by a blanket attack that pwns everything and steals all the data or monies. Just about the only way to achieve this kind of blast radius is to have a kernel-level backdoor installed in every other computer on the planet - which is exactly what those endpoint "security" systems are.
pyeri 2y ago
But why does a signature database update have to mess with the kernel in any kind of way? Shouldn't such a database stay in the user land?
- vbezhenar 2y ago
  Because kernel needs to parse the data in some way and that parser apparently was broken enough. Whether it could be done in a more resilient manner, I don't know, you need to remember that antivirus works in hostile environment and can't necessarily trust userspace, so probably they need to verify signatures and parse payload in the kernel space.
- theshrike79 2y ago
  The scanner is a Ring 0[0] program. Windows only has 2 options 0 and 3. 3 won't work for any kind of security scanners, so they're forced to use 0. The proper place would be Ring 1, which doesn't exist on Windows. And being a kernel-level operation, it has the capability to crash the whole system before the actual OS has any chance to intervene. [0] https://en.wikipedia.org/wiki/Protection_ring https://en.wikipedia.org/wiki/Protection_ring
  - leosarev 2y ago
    Why is so?
    - hello_moto 2y ago
      That's a question for Microsoft OS architects
    - benchloftbrunch 2y ago
      Historical reasons. Windows NT was designed to support architectures with only two privilege rings.
    - layer8 2y ago
      All modern OSes only use ring 0 and 3. Intel is considering removing rings 1 and 2 in a future revision for that reason: https://www.intel.com/content/www/us/en/developer/articles/technical/envisioning-future-simplified-architecture.html https://www.intel.com/content/www/us/en/developer/articles/t...
siscia 2y ago
Thanks for the clarification, this makes more sense.
LeonB 2y ago
It’s quite impressive really — crowdstrike were deploying a content update to all of their servers to warn them of the “nothing but nulls, anti-crowdstrike virus” Their precognitive intelligence suggested that a world wide attack was only moments away. The same precognitive system showed that the virus was so totally incapacitating that the only safe response was to incapacitate the server. Knowing that the virus was capable of taking down every crowdstrike server, they didn’t waste time trying it on a subset of servers. When you know you know.
Ensorceled 2y ago
Surely there is a happy medium between zero (nil,none,nada,zilch) staging and 24 hours of rolling updates? A single 30 second or so VM test would have revealed this issues.
- layer8 2y ago
  There should have been a test catching the error before rollout, however this doesn’t require a staged rollout as suggested by the GP comment, testing the update at some customers (which would still be hosed in that case), it only requires executing the test before the rollout.
jrochkind1 2y ago
Yup. If they were delaying update to half of their customers for 24 hours, and in that 24 hours some of their customers got hacked by a zero day, say leading to ransomeware, the comment threads would be demanding their head for that!
- sateesh 2y ago
  Even if it is a staged rollout why would one do it in 24 hour phases ? It can be a hourly (say) staggered rollout too.
  - jrochkind1 2y ago
    Sure. And if someone showed up here with a story about how they got attacked and ransomwared enterprise-wide in the however many several hours that they were waiting for their turn to rollout, what do you think HN response would be? Hmm, maybe you could have companies pay more to be in the first rollout group? That'd go over well too.
    - sateesh 2y ago
      True, there will be comments blaming CS for not doing faster rollout. But there would be some comments empathizing with CS viewpoint and pointing out the conflicting compromise between velocity, and correctness. Even now I think the comments wouldn't have been unequivocally critical, of CS, if the hosts affected were a variant of windows (say issue was seen on version of windows 10 which was two update behind),there would have been some emphasizing the thorniness of the problem and sympathetic of CS.
sateesh 2y ago
It doesn't matter what kind of update it was: signature, content,etc. Only thing that matters is does the update has a potential to disrupt the user's normal activity (leave alone bricking the host), if yes ensure it either works or have a staged rollout with a remediation plan.
aldousd666 2y ago
You do want to fuzz test it like crazy. Can be automated. Takes minutes, saves billions

TeMPOraL 2y ago

We can see clearly now that this is a stupid approach. Viruses don't move that fast. This situation is akin to the immune system overreacting and melting the patient in response to a papercut. This sometimes happens, but it's considered a serious medical condition, and I believe the treatment is to nuke someone's immune system entirely with hard radiation, and reinstall a less aggressive copy. Take from that analogy what you want.

orf 2y ago
> Viruses don't move that fast Yes they do? And it’s more akin to a shared immune system than a single organism. In this case, it’s not like viruses move fast relative to the total population of machines, but within the population of machines being targeted they do move fast.
- TeMPOraL 2y ago
  Still, better to let them spread a bit and deal with the localized damage than risk nuking everything. There is such a thing as treatment that's very effective, but not used because of a low probability risk of terminal damage.
- proveitbh 2y ago
  Cite one virus thay crashed the supposed 10 or 100 million machines in 70 minutes. Just one.
  - orf 2y ago
    Microsoft puts the count at 8.5 million computers. So, percentage wise, the MyDoom virus in 2004 infected a far greater % of computers in a month: which in the context of internet penetration, availability and speeds (40kb/s average, 450kb/s fastest) in 2004 was about as fast as it could have. So it might as well have been 70 minutes, given downloading a 50mb file on dial up would take way longer than 70 mins. To the smart people below: It’s clear to everyone that 70 minutes is not 1 month. The point is that it’s not a fair comparison: it would simply not have been possible to infect that many computers in 70 minutes: the internet infrastructure just wasn’t there. It’s like saying “the Spanish flu didn’t do that much damage because there where less people on the planet” - it’s a meaningless absolute comparison, whereas the relative comparison is what matters.
    - lolilokol 2y ago
      [flagged]
    - klijibbn 2y ago
      [flagged]
    - smartpeoplebelw 2y ago
      There's also orders of magnitudes more machines today than 20 years ago -- so it should be easier to infect more machines now than before, and yet no one can sight a virus that was as quickly moving and damaging as what crowdstrike did through gross negligence. Be better.
      - orf 2y ago
        This entire thread is stupid. Computer security as a whole has improved, whilst the complexity of interconnected systems has exponentially increased. This has made the barrier to entry for malware higher, and so means we no longer have the same historic examples of large scale worms targeting consumer machines that we used to. At the same time the financial rewards for finding and exploiting a vulnerability within an organisations complex stack have greatly increased. The rewards are coupled to the time it takes to execute on the vulnerability. This leads to what we have today: localised, and often specialised attacks against valuable targets that are executed as fast as possible in order to minimise the chance a target has to respond or the vulnerability they are exploiting to be burned. Of course the “smart people belw” must know this, so it’s unclear why they are pretending to be dumb.
- aldousd666 2y ago
  No they're in the wrong. They didn't test adequately, regardless of their motive for not doing so. Obviously reality is not backing up your theory there
  - orf 2y ago
    FYI, both the following statements can be true: 1. Crowdstrike didn’t test adequately 2. Viruses can move pretty fast once a foothold is gained
UncleMeat 2y ago
https://en.wikipedia.org/wiki/SQL_Slammer https://en.wikipedia.org/wiki/SQL_Slammer There is no real "speed limit" on malware spread.
- TeMPOraL 2y ago
  No, but there are impenetrable barriers. 0days in paricular are usually very specific and affect few systems directly but even the broader ones aren't usually followed by a blanket attack that pwns everything and steals all the data or monies. Just about the only way to achieve this kind of blast radius is to have a kernel-level backdoor installed in every other computer on the planet - which is exactly what those endpoint "security" systems are.

pyeri 2y ago

But why does a signature database update have to mess with the kernel in any kind of way? Shouldn't such a database stay in the user land?

vbezhenar 2y ago
Because kernel needs to parse the data in some way and that parser apparently was broken enough. Whether it could be done in a more resilient manner, I don't know, you need to remember that antivirus works in hostile environment and can't necessarily trust userspace, so probably they need to verify signatures and parse payload in the kernel space.
theshrike79 2y ago
The scanner is a Ring 0[0] program. Windows only has 2 options 0 and 3. 3 won't work for any kind of security scanners, so they're forced to use 0. The proper place would be Ring 1, which doesn't exist on Windows. And being a kernel-level operation, it has the capability to crash the whole system before the actual OS has any chance to intervene. [0] https://en.wikipedia.org/wiki/Protection_ring https://en.wikipedia.org/wiki/Protection_ring
- leosarev 2y ago
  Why is so?
  - hello_moto 2y ago
    That's a question for Microsoft OS architects
  - benchloftbrunch 2y ago
    Historical reasons. Windows NT was designed to support architectures with only two privilege rings.
  - layer8 2y ago
    All modern OSes only use ring 0 and 3. Intel is considering removing rings 1 and 2 in a future revision for that reason: https://www.intel.com/content/www/us/en/developer/articles/technical/envisioning-future-simplified-architecture.html https://www.intel.com/content/www/us/en/developer/articles/t...

siscia 2y ago

Thanks for the clarification, this makes more sense.

LeonB 2y ago

It’s quite impressive really — crowdstrike were deploying a content update to all of their servers to warn them of the “nothing but nulls, anti-crowdstrike virus” Their precognitive intelligence suggested that a world wide attack was only moments away. The same precognitive system showed that the virus was so totally incapacitating that the only safe response was to incapacitate the server. Knowing that the virus was capable of taking down every crowdstrike server, they didn’t waste time trying it on a subset of servers. When you know you know.

Ensorceled 2y ago

Surely there is a happy medium between zero (nil,none,nada,zilch) staging and 24 hours of rolling updates? A single 30 second or so VM test would have revealed this issues.

layer8 2y ago
There should have been a test catching the error before rollout, however this doesn’t require a staged rollout as suggested by the GP comment, testing the update at some customers (which would still be hosed in that case), it only requires executing the test before the rollout.

jrochkind1 2y ago

Yup. If they were delaying update to half of their customers for 24 hours, and in that 24 hours some of their customers got hacked by a zero day, say leading to ransomeware, the comment threads would be demanding their head for that!

sateesh 2y ago
Even if it is a staged rollout why would one do it in 24 hour phases ? It can be a hourly (say) staggered rollout too.
- jrochkind1 2y ago
  Sure. And if someone showed up here with a story about how they got attacked and ransomwared enterprise-wide in the however many several hours that they were waiting for their turn to rollout, what do you think HN response would be? Hmm, maybe you could have companies pay more to be in the first rollout group? That'd go over well too.
  - sateesh 2y ago
    True, there will be comments blaming CS for not doing faster rollout. But there would be some comments empathizing with CS viewpoint and pointing out the conflicting compromise between velocity, and correctness. Even now I think the comments wouldn't have been unequivocally critical, of CS, if the hosts affected were a variant of windows (say issue was seen on version of windows 10 which was two update behind),there would have been some emphasizing the thorniness of the problem and sympathetic of CS.

sateesh 2y ago

It doesn't matter what kind of update it was: signature, content,etc. Only thing that matters is does the update has a potential to disrupt the user's normal activity (leave alone bricking the host), if yes ensure it either works or have a staged rollout with a remediation plan.

aldousd666 2y ago

You do want to fuzz test it like crazy. Can be automated. Takes minutes, saves billions