4 ms·

Security: You must install Microsoft Defender on all Linux VM's Devs: Ugh...why? Security: For safety! Devs: Fine, we won't argue. Deploy it if you may. A f

Security: You must install Microsoft Defender on all Linux VM's Devs: Ugh...why? Security: For safety! Devs: Fine, we won't argue. Deploy it if you may. A few moments later... Devs: All of our VM's are slow as crap! Defender is using 100% of the CPU! Security: Add another core to your VM's. ticket closed Management: Why are our developers up 30% on their cloud spend!?

lanstin 2y ago
Even worse, at least aesthetically, than the AV is the pacct stuff to log 1 line for every single syscall. Talk about sublinear scaling. And for what, no one can say. Actually, even worse than that was we had to install AV on all the images that the ephemeral map-reduce/Hadoop clusters we spun up, but the way the AV stuff worked, the computes were gone by the time the registration for the new compute had gone thru whole process. And, in AWS accounts where there were maybe 63 IPs and say 600 EC2s/day they used IP as the primary key for the "list of compute in the VPC." So they stitched together totally unrelated stuff as if it was the same continuous compute. I guess it would be eventually fixed, but the bad security data was not a real concern of the devops team that was building out stuff as rapidly as possible, nor were the EC2 made in a sealed off VPC and only lived for a few tens of minutes or hours at best a serious security concern to the actual security people. Just a check list solution hitting a novel environment.
blibble 2y ago
and all microsoft have to do to increase cloud revenue is make defender chew up 5% more CPU every few months directly incentivised to make shit software
- 01HNNWZ0MV43FF 2y ago
  To a point. After a while they're wasting hardware and another vendor will undercut them. After some decades people will start leaving
  - sebazzz 2y ago
    That is very long term. Not interesting to shareholders.
- whycome 2y ago
  Isn't this kinda the hardware model for Apple devices too? Eg batterygate
  - Hardwired8976 2y ago
    Batterygate was just making sure the phone doesn't shut down suddenly as the battery deteriotes and is less capable.
    - mrguyorama 2y ago
      People say this like it lets Apple off the hook. Let me explain why it doesn't. Apple had full control over the whole phone's software stack, in a very good way, meaning they built a good mobile OS that had good systems for power management and an app lifecycle that could actually kill apps at will to maintain efficiency, without disrupting the user. With this, they decided to ship smaller batteries so they could make slimmer phones. Except, they used garbage batteries. They were so small (1600mAh on the iPhone 6) that normal wear and tear of a few years degraded them to the point that the battery chemistry could not keep up with normal processor frequency and power ramping. Apple started getting a lot of complaints because people were understandably upset that their 2-3 year old phone couldn't run for more than an hour off the charger. Apple didn't like increasing support load, even though they weren't covering anyone's battery replacement. Instead of putting out a press release that they had shipped sub-standard batteries in their phones, and offering free battery replacements with a new battery that wouldn't have the same problem in another 2-3 years, they included code in the new version of iOS to SIGNIFICANTLY slow down your 3 year old or less phone. Apple made a product that deteriorated way too quickly, and then tried to hide it. That's batterygate. If LG sold a fridge that would die after five years because of compressor fatigue and then silently updated their fridges to not operate colder than 45 degrees F to extend the life of the compressor, I would hope you would be pissed at that, right? A reminder that the iPhone 6 was also "Bendgate", which internal apple memos showed they knew was a serious problem before they sold it, and then claimed two years after release they only had 9 complaints of phone bending and that it wouldn't bend in normal use.
      - 2y ago
thdxr 2y ago
there was a production incident at a customer that was this exact scenario
jordanb 2y ago
We had cylance take out all of our kubernetes clusters a few years ago. The whole cybersecurity concept of installing third party mystery meat in the kernel controllable over the internet by a different company seems contrary both to good security practices and software quality assurance, immutable production architecture and repeatable builds.
- pennomi 2y ago
  Third party mystery meat is mostly intended for scapegoating if a problem does occur.
  - greesil 2y ago
    But sometimes the goat escapes and then breaks everything. Then somebody's got your goat.
    - kchr 2y ago
      Escape Goat!
pixl97 2y ago
I work supporting software that processes millions of small files a day, with a lot of these scripting languages. The speed difference in total iops where AV is installed vs not installed is huge. 30-50% loss is no joke.
- hulitu 2y ago
  The problem is that the AV does not increase security.
bb88 2y ago
That's a management problem. IT security didn't communicate that to the finance folks. Microsoft didn't communicate that to IT security. And if they're on Azure, it's more money for Microsoft.
- sebazzz 2y ago
  Oh finance doesn't care. Security is paramount and not something you can save money on, is the mindset. Besides, the real costs of this is allocated to many individual cost centers so in the bigger picture you won't see it.
Nemo_bis 2y ago
Speaking of which, does anyone use offline Microsoft Defender updates (maybe to have direct control of canary/staggered updates)? https://learn.microsoft.com/en-us/defender-endpoint/linux-support-offline-security-intelligence-update https://learn.microsoft.com/en-us/defender-endpoint/linux-su...

lanstin 2y ago

Even worse, at least aesthetically, than the AV is the pacct stuff to log 1 line for every single syscall. Talk about sublinear scaling. And for what, no one can say. Actually, even worse than that was we had to install AV on all the images that the ephemeral map-reduce/Hadoop clusters we spun up, but the way the AV stuff worked, the computes were gone by the time the registration for the new compute had gone thru whole process. And, in AWS accounts where there were maybe 63 IPs and say 600 EC2s/day they used IP as the primary key for the "list of compute in the VPC." So they stitched together totally unrelated stuff as if it was the same continuous compute. I guess it would be eventually fixed, but the bad security data was not a real concern of the devops team that was building out stuff as rapidly as possible, nor were the EC2 made in a sealed off VPC and only lived for a few tens of minutes or hours at best a serious security concern to the actual security people. Just a check list solution hitting a novel environment.

blibble 2y ago

and all microsoft have to do to increase cloud revenue is make defender chew up 5% more CPU every few months directly incentivised to make shit software

01HNNWZ0MV43FF 2y ago
To a point. After a while they're wasting hardware and another vendor will undercut them. After some decades people will start leaving
- sebazzz 2y ago
  That is very long term. Not interesting to shareholders.
whycome 2y ago
Isn't this kinda the hardware model for Apple devices too? Eg batterygate
- Hardwired8976 2y ago
  Batterygate was just making sure the phone doesn't shut down suddenly as the battery deteriotes and is less capable.
  - mrguyorama 2y ago
    People say this like it lets Apple off the hook. Let me explain why it doesn't. Apple had full control over the whole phone's software stack, in a very good way, meaning they built a good mobile OS that had good systems for power management and an app lifecycle that could actually kill apps at will to maintain efficiency, without disrupting the user. With this, they decided to ship smaller batteries so they could make slimmer phones. Except, they used garbage batteries. They were so small (1600mAh on the iPhone 6) that normal wear and tear of a few years degraded them to the point that the battery chemistry could not keep up with normal processor frequency and power ramping. Apple started getting a lot of complaints because people were understandably upset that their 2-3 year old phone couldn't run for more than an hour off the charger. Apple didn't like increasing support load, even though they weren't covering anyone's battery replacement. Instead of putting out a press release that they had shipped sub-standard batteries in their phones, and offering free battery replacements with a new battery that wouldn't have the same problem in another 2-3 years, they included code in the new version of iOS to SIGNIFICANTLY slow down your 3 year old or less phone. Apple made a product that deteriorated way too quickly, and then tried to hide it. That's batterygate. If LG sold a fridge that would die after five years because of compressor fatigue and then silently updated their fridges to not operate colder than 45 degrees F to extend the life of the compressor, I would hope you would be pissed at that, right? A reminder that the iPhone 6 was also "Bendgate", which internal apple memos showed they knew was a serious problem before they sold it, and then claimed two years after release they only had 9 complaints of phone bending and that it wouldn't bend in normal use.
    - 2y ago

thdxr 2y ago

there was a production incident at a customer that was this exact scenario

jordanb 2y ago

We had cylance take out all of our kubernetes clusters a few years ago. The whole cybersecurity concept of installing third party mystery meat in the kernel controllable over the internet by a different company seems contrary both to good security practices and software quality assurance, immutable production architecture and repeatable builds.

pennomi 2y ago
Third party mystery meat is mostly intended for scapegoating if a problem does occur.
- greesil 2y ago
  But sometimes the goat escapes and then breaks everything. Then somebody's got your goat.
  - kchr 2y ago
    Escape Goat!

pixl97 2y ago

I work supporting software that processes millions of small files a day, with a lot of these scripting languages. The speed difference in total iops where AV is installed vs not installed is huge. 30-50% loss is no joke.

hulitu 2y ago
The problem is that the AV does not increase security.

bb88 2y ago

That's a management problem. IT security didn't communicate that to the finance folks. Microsoft didn't communicate that to IT security. And if they're on Azure, it's more money for Microsoft.

sebazzz 2y ago
Oh finance doesn't care. Security is paramount and not something you can save money on, is the mindset. Besides, the real costs of this is allocated to many individual cost centers so in the bigger picture you won't see it.

Nemo_bis 2y ago

Speaking of which, does anyone use offline Microsoft Defender updates (maybe to have direct control of canary/staggered updates)? https://learn.microsoft.com/en-us/defender-endpoint/linux-support-offline-security-intelligence-update https://learn.microsoft.com/en-us/defender-endpoint/linux-su...