7 ms·

The thing I don't understand about all of this is another, much less technical and much more important. Why the blas radius was so huge? I have deployed much

by siscia 2y ago

The thing I don't understand about all of this is another, much less technical and much more important. Why the blas radius was so huge? I have deployed much less important services much more slowly with automatic monitoring and rollback in place. You first deploy to beta, where you don't get customers traffic, if everything goes right to a small part of your fleet, and slowly increase the percentage of hosts that receives the updates. This would have stopped the issue immediately, and I somehow I thought it was common practices...

moogly 2y ago
They don't seem to dogfood their own software. They don't seem to think it's very useful software in their own org, I guess.
INTPenis 2y ago
Considering the impact this incident had they definitely should have a large staging environment of windows clients to deploy first. There are so many ways to avoid this issue, or at least minimize the risk of it happening, but as always profits come before people.
andy81 2y ago
Even if there was a canary release process for code updates, the config updates seem to have been on a separate channel. The expectation being that people want up-to-date virus detection rules constantly even if they don't want potentially breaking changes. The missed edge case being an untested config that breaks existing code. Source: Pure speculation, don't quote this in news articles.
vbezhenar 2y ago
It wasn't software update. It was signature database update. It's supposed to roll out as fast as possible. When you learn about new virus, it's already in the wild, so every minute counts. You don't want to delay update for a day just to find out that your servers were breached 20 hours ago.
- TeMPOraL 2y ago
  We can see clearly now that this is a stupid approach. Viruses don't move that fast. This situation is akin to the immune system overreacting and melting the patient in response to a papercut. This sometimes happens, but it's considered a serious medical condition, and I believe the treatment is to nuke someone's immune system entirely with hard radiation, and reinstall a less aggressive copy. Take from that analogy what you want.
  - orf 2y ago
    > Viruses don't move that fast Yes they do? And it’s more akin to a shared immune system than a single organism. In this case, it’s not like viruses move fast relative to the total population of machines, but within the population of machines being targeted they do move fast.
    - TeMPOraL 2y ago
      Still, better to let them spread a bit and deal with the localized damage than risk nuking everything. There is such a thing as treatment that's very effective, but not used because of a low probability risk of terminal damage.
    - proveitbh 2y ago
      Cite one virus thay crashed the supposed 10 or 100 million machines in 70 minutes. Just one.
      - orf 2y ago
        Microsoft puts the count at 8.5 million computers. So, percentage wise, the MyDoom virus in 2004 infected a far greater % of computers in a month: which in the context of internet penetration, availability and speeds (40kb/s average, 450kb/s fastest) in 2004 was about as fast as it could have. So it might as well have been 70 minutes, given downloading a 50mb file on dial up would take way longer than 70 mins. To the smart people below: It’s clear to everyone that 70 minutes is not 1 month. The point is that it’s not a fair comparison: it would simply not have been possible to infect that many computers in 70 minutes: the internet infrastructure just wasn’t there. It’s like saying “the Spanish flu didn’t do that much damage because there where less people on the planet” - it’s a meaningless absolute comparison, whereas the relative comparison is what matters.
robxorb 2y ago
"Blast radius" seems... apt. It would be rather easier to understand and explain if it were intentional. Likely not able to be discussed though. Anyone able to do that here?
rplnt 2y ago
It's answered in the post (in the thread) as well. But for comparison, when I worked for an AV vendor we pushed maybe 4 updates a day to a much bigger customer base (if the numbers reported by MS are true).
- kchr 2y ago
  I'm curious, what did your deployment plan look like? Phased/staggered, if so how?
  - rplnt 2y ago
    It was a long time ago and I wasn't as involved with this, so I don't know with certainty what was used and how. We had multiple channels for major product versions + beta customers on the most recent one. On top of these we could stage different CDN regions. There were different types of data you could update and those might have been treated differently (e.g. simple file signatures vs definitions for heuristics).

by siscia 2y ago

moogly 2y ago

They don't seem to dogfood their own software. They don't seem to think it's very useful software in their own org, I guess.

INTPenis 2y ago

Considering the impact this incident had they definitely should have a large staging environment of windows clients to deploy first. There are so many ways to avoid this issue, or at least minimize the risk of it happening, but as always profits come before people.

andy81 2y ago

Even if there was a canary release process for code updates, the config updates seem to have been on a separate channel. The expectation being that people want up-to-date virus detection rules constantly even if they don't want potentially breaking changes. The missed edge case being an untested config that breaks existing code. Source: Pure speculation, don't quote this in news articles.

vbezhenar 2y ago

It wasn't software update. It was signature database update. It's supposed to roll out as fast as possible. When you learn about new virus, it's already in the wild, so every minute counts. You don't want to delay update for a day just to find out that your servers were breached 20 hours ago.

TeMPOraL 2y ago
We can see clearly now that this is a stupid approach. Viruses don't move that fast. This situation is akin to the immune system overreacting and melting the patient in response to a papercut. This sometimes happens, but it's considered a serious medical condition, and I believe the treatment is to nuke someone's immune system entirely with hard radiation, and reinstall a less aggressive copy. Take from that analogy what you want.
- orf 2y ago
  > Viruses don't move that fast Yes they do? And it’s more akin to a shared immune system than a single organism. In this case, it’s not like viruses move fast relative to the total population of machines, but within the population of machines being targeted they do move fast.
  - TeMPOraL 2y ago
    Still, better to let them spread a bit and deal with the localized damage than risk nuking everything. There is such a thing as treatment that's very effective, but not used because of a low probability risk of terminal damage.
  - proveitbh 2y ago
    Cite one virus thay crashed the supposed 10 or 100 million machines in 70 minutes. Just one.
    - orf 2y ago
      Microsoft puts the count at 8.5 million computers. So, percentage wise, the MyDoom virus in 2004 infected a far greater % of computers in a month: which in the context of internet penetration, availability and speeds (40kb/s average, 450kb/s fastest) in 2004 was about as fast as it could have. So it might as well have been 70 minutes, given downloading a 50mb file on dial up would take way longer than 70 mins. To the smart people below: It’s clear to everyone that 70 minutes is not 1 month. The point is that it’s not a fair comparison: it would simply not have been possible to infect that many computers in 70 minutes: the internet infrastructure just wasn’t there. It’s like saying “the Spanish flu didn’t do that much damage because there where less people on the planet” - it’s a meaningless absolute comparison, whereas the relative comparison is what matters.

robxorb 2y ago

"Blast radius" seems... apt. It would be rather easier to understand and explain if it were intentional. Likely not able to be discussed though. Anyone able to do that here?

rplnt 2y ago

It's answered in the post (in the thread) as well. But for comparison, when I worked for an AV vendor we pushed maybe 4 updates a day to a much bigger customer base (if the numbers reported by MS are true).

kchr 2y ago
I'm curious, what did your deployment plan look like? Phased/staggered, if so how?
- rplnt 2y ago
  It was a long time ago and I wasn't as involved with this, so I don't know with certainty what was used and how. We had multiple channels for major product versions + beta customers on the most recent one. On top of these we could stage different CDN regions. There were different types of data you could update and those might have been treated differently (e.g. simple file signatures vs definitions for heuristics).