4 ms·

> How Do We Prevent This From Happening Again? > Software Resiliency and Testing > * Improve Rapid Response Content testing by using testing types such

> How Do We Prevent This From Happening Again? > Software Resiliency and Testing > * Improve Rapid Response Content testing by using testing types such as: > * Local developer testing So no one actually tested the changes before deploying?!

Narretz 2y ago
And why is it "local developer testing" and not CI/CD. This makes them look like absolute amateurs.
- belter 2y ago
  > This makes them look like absolute amateurs. This applies also to all Architects and CTO's at all these Fortune 500 companies, who allowed these self updating systems into their critical systems. I would offer a copy of Antifragile to each of these teams: https://en.wikipedia.org/wiki/Antifragile_(book) https://en.wikipedia.org/wiki/Antifragile_(book) "Every captain goes down with every ship"
  - acdha 2y ago
    Architects likely do not have a choice. These things are driven by auditors and requirements for things like insurance or PCI and it’s expensive to protest those. I know people who’ve gone full serverless just to lop off the branches of the audit tree about general purpose server operating systems, and now I’m wondering whether anyone is thinking about iOS/ChromeOS for the same reason. The more successful path here is probably demanding proof of a decent SDLC, use of memory-safe languages, etc. in contract language.
    - belter 2y ago
      > Architects likely do not have a choice. Architects don't have a choice, CTO are well paid to golf with the CEO and delegate to their teams, Auditors just audit but are not involved with the technical implementations, Developers just develop according to the Spec, and Security team just are a pain in the ass. Nobody owns it... Everybody get's well paid, and at the end we have to get lessons learned...It's a s*&^&t show...
      - mardifoufs 2y ago
        Some industries are forced by regulation or liability to have something like crowdstrike deployed on their systems. And crowdstrike doesn't have a lot of alternatives that tick as many checkboxes and are as widely recognized.
        belter 2y ago
        Please give me an example of that specific regulation.
        toomuchtodo 2y ago
        PCI DSS v4.0 Requirements 5 and 6 speaks very broadly for anti-malware controls, which Crowdstrike provides as EDR, and cybersecurity (liability, ransomware, etc) insurance absolutely requires it from the questionnaires I’ve completed and am required to attest to. > In its first version, PCI DSS included controls for detecting, removing, blocking, and containing malicious code (malware). Until version 3.2.1, these controls were generically referred to as "anti-virus software", which was incorrect technically because they protect not just against viruses, but also against other known malware variants (worms, trojans, ransomware, spyware, rootkits, adware, backdoors, etc.). As a result, the term "antimalware" is now used not only to refer to viruses, but also to all other types of malicious code, more in line with the requirement's objectives. > To avoid the ambiguities seen in previous versions of the standard about which operating systems should have an anti-malware solution installed and which should not, a more operational approach has been chosen: the entity should perform a periodic assessment to determine which system components should require an anti-malware solution. All other assets that are determined not to be affected by malware should be included in a list (req. 5.2.3). > Updates of the anti-malware solution must be performed automatically (req. 5.3.1). > Finally, the term "real-time scanning" is explicitly included for the anti-malware solution (this is a type of persistent, continuous scanning where a scan for security risks is performed every time a file is received, opened, downloaded, copied or modified). Previously, there was a reference to the fact that anti-malware mechanisms should be actively running, which gave rise to different interpretations. > Continuous behavioral analysis of systems or processes is incorporated as an accepted anti-malware solution scanning method, as an alternative to traditional periodic (scheduled and on-demand) and real-time (on-access) scans (req. 5.3.2). https://www.advantio.com/blog/analysis-of-pci-dss-v4.0-part-4-requirements-5-6 https://www.advantio.com/blog/analysis-of-pci-dss-v4.0-part-...
- RaftPeople 2y ago
  The fact that they even listed "local developer testing" is pretty weird. That is just part of the basic process and is hardly the thing that ensures a problem like this doesn't happen.
- radicaldreamer 2y ago
  They don't care, CI/CD, like QA, is considered a cost center for some of these companies. The cheapest thing for them is to offload the burden of testing every configuration onto the developer, who is also going to be tasked with shipping as quickly as possible or getting canned. Claw back executive pay, stock, and bonuses imo and you'll see funded QA and CI teams.
- hyperpape 2y ago
  It sure sounds like the "Content Validator" they mention is a form of CI/CD. The problem is that it passed that validation, but was capable of failing in reality.
  - yuliyp 2y ago
    The content validator is a form of validation done in CI. Their CD pipeline is the bigger problem here: it was extremely reckless given the system it was used in (configuring millions of customer machines in unknown environments). A CD pipeline for a tiny startup's email service can just deploy straight away. Crowdstrike (as they finally realized) need a CD pipeline with much more rigorous validation.
spacebanana7 2y ago
This also becomes a security issue at some point. If these updates can go in untested, what's to stop a rogue employee from deliberately pushing a malicious update? I know insider threats are very hard to protect against in general but these companies must be the most juicy target for state actors. Imagine what you could do with kernel space code in emergency services, transport infrastructure and banks.
deleted 2y ago
[deleted]

Narretz 2y ago

And why is it "local developer testing" and not CI/CD. This makes them look like absolute amateurs.

belter 2y ago
> This makes them look like absolute amateurs. This applies also to all Architects and CTO's at all these Fortune 500 companies, who allowed these self updating systems into their critical systems. I would offer a copy of Antifragile to each of these teams: https://en.wikipedia.org/wiki/Antifragile_(book) https://en.wikipedia.org/wiki/Antifragile_(book) "Every captain goes down with every ship"
- acdha 2y ago
  Architects likely do not have a choice. These things are driven by auditors and requirements for things like insurance or PCI and it’s expensive to protest those. I know people who’ve gone full serverless just to lop off the branches of the audit tree about general purpose server operating systems, and now I’m wondering whether anyone is thinking about iOS/ChromeOS for the same reason. The more successful path here is probably demanding proof of a decent SDLC, use of memory-safe languages, etc. in contract language.
  - belter 2y ago
    > Architects likely do not have a choice. Architects don't have a choice, CTO are well paid to golf with the CEO and delegate to their teams, Auditors just audit but are not involved with the technical implementations, Developers just develop according to the Spec, and Security team just are a pain in the ass. Nobody owns it... Everybody get's well paid, and at the end we have to get lessons learned...It's a s*&^&t show...
    - mardifoufs 2y ago
      Some industries are forced by regulation or liability to have something like crowdstrike deployed on their systems. And crowdstrike doesn't have a lot of alternatives that tick as many checkboxes and are as widely recognized.
      - belter 2y ago
        Please give me an example of that specific regulation.
        toomuchtodo 2y ago
        PCI DSS v4.0 Requirements 5 and 6 speaks very broadly for anti-malware controls, which Crowdstrike provides as EDR, and cybersecurity (liability, ransomware, etc) insurance absolutely requires it from the questionnaires I’ve completed and am required to attest to. > In its first version, PCI DSS included controls for detecting, removing, blocking, and containing malicious code (malware). Until version 3.2.1, these controls were generically referred to as "anti-virus software", which was incorrect technically because they protect not just against viruses, but also against other known malware variants (worms, trojans, ransomware, spyware, rootkits, adware, backdoors, etc.). As a result, the term "antimalware" is now used not only to refer to viruses, but also to all other types of malicious code, more in line with the requirement's objectives. > To avoid the ambiguities seen in previous versions of the standard about which operating systems should have an anti-malware solution installed and which should not, a more operational approach has been chosen: the entity should perform a periodic assessment to determine which system components should require an anti-malware solution. All other assets that are determined not to be affected by malware should be included in a list (req. 5.2.3). > Updates of the anti-malware solution must be performed automatically (req. 5.3.1). > Finally, the term "real-time scanning" is explicitly included for the anti-malware solution (this is a type of persistent, continuous scanning where a scan for security risks is performed every time a file is received, opened, downloaded, copied or modified). Previously, there was a reference to the fact that anti-malware mechanisms should be actively running, which gave rise to different interpretations. > Continuous behavioral analysis of systems or processes is incorporated as an accepted anti-malware solution scanning method, as an alternative to traditional periodic (scheduled and on-demand) and real-time (on-access) scans (req. 5.3.2). https://www.advantio.com/blog/analysis-of-pci-dss-v4.0-part-4-requirements-5-6 https://www.advantio.com/blog/analysis-of-pci-dss-v4.0-part-...
RaftPeople 2y ago
The fact that they even listed "local developer testing" is pretty weird. That is just part of the basic process and is hardly the thing that ensures a problem like this doesn't happen.
radicaldreamer 2y ago
They don't care, CI/CD, like QA, is considered a cost center for some of these companies. The cheapest thing for them is to offload the burden of testing every configuration onto the developer, who is also going to be tasked with shipping as quickly as possible or getting canned. Claw back executive pay, stock, and bonuses imo and you'll see funded QA and CI teams.
hyperpape 2y ago
It sure sounds like the "Content Validator" they mention is a form of CI/CD. The problem is that it passed that validation, but was capable of failing in reality.
- yuliyp 2y ago
  The content validator is a form of validation done in CI. Their CD pipeline is the bigger problem here: it was extremely reckless given the system it was used in (configuring millions of customer machines in unknown environments). A CD pipeline for a tiny startup's email service can just deploy straight away. Crowdstrike (as they finally realized) need a CD pipeline with much more rigorous validation.

spacebanana7 2y ago

This also becomes a security issue at some point. If these updates can go in untested, what's to stop a rogue employee from deliberately pushing a malicious update? I know insider threats are very hard to protect against in general but these companies must be the most juicy target for state actors. Imagine what you could do with kernel space code in emergency services, transport infrastructure and banks.

deleted 2y ago

[deleted]