5 ms·

I'm very curious what organisations would have such a policy. I can't imagine it being viable for any size of org without significant self-deception (or banning

I'm very curious what organisations would have such a policy. I can't imagine it being viable for any size of org without significant self-deception (or banning the use of all open source at which point CVEs are moot anyway).

SirFatty 2mo ago
ITAR
- lucideer 2mo ago
  ITAR has no such hard requirements. Might be some orgs that tell themselves they're attempting this under ITAR but they're not doing it in any comprehensive way. The only thing within ITAR that I'm aware of concerning itself with software supply chain is SP 800-218 requirements & that's just a load of open-to-interpretation weasel words about having CVE detection & automations in place & some defined plans for reducing the number of vulns. Pretty sure that component of it is even eligible for self-assessment.
  - bluGill 2mo ago
    Yes and no. ITAR (and other laws like it) are self assessment and don't specifically say thing thing. However your interpretation / self-assessment is subject to various reviews/audits. These days the reviewers are not going to be kind to someone who just says "not an issue", they will demand strong justification. Most organizations take the view that is is easier to fix all CVEs than try to pass audits. Thus by the letter of the law you are correct. However to meet the letter of the law without fixing CVEs is generally seen as harder than thus fixing CVEs. So the effect is ITAR (and similar laws) force you to fix CVEs.
  - SirFatty 2mo ago
    Whatever you say, chief. I worked in that environment for quite a while, and maybe on a technicality you're right, the effect outcome is that you will do it if not for all your customers that will require it.
clbrmbr 2mo ago
Many orgs (esp w ISO27000) have a vulnerability management policy that involves patching at least critical CVEs within a short timeline. Tools like trivvy make it possible to do the scans…
- lucideer 2mo ago
  I've been in such an org, & I've led initiatives to set up automated detection at very large scale. We started by issuing tickets to teams to resolve CVEs within varying timelines - ranging from a 24hr fix to 6 months - connected to the CVSS score. It wasn't viable. - Firstly, you quickly realise how irrelevant CVSS scores are - initiatives like First's EPSS are designed to fix this but they aren't there yet - Secondly, you need to begin implementing localised heuristics to determine exploitable code paths. This has generally been incredibly difficult to do reliably - LLMs have started to make it easier, but it's expensive. - Lastly, you need to factor in consideration of actionable remediation pathways. A dependency upgrade for critical infrastructure might contain breaking changes that take months to fix, or two competing CVEs might be present in interdependent versions of transitive dependencies in your sbom tree. Most orgs aren't applying any of the above three filters to reduce their CVE remediation burden, & even if they are, it's still too high to make zero a viable target. In reality, most orgs aren't doing comprehensive detection to begin with - if you haven't discovered all of your CVEs, your remediation burden is going to be a lot more manageable.
  - mr_mitm 2mo ago
    > - Firstly, you quickly realise how irrelevant CVSS scores are Even if you factor in the environmental score? I realize it's a lot more work, but it basically allows you to tune the score to get any value you want.
    - michaelt 2mo ago
      Imagine a YAML parsing library that can cause an out-of-memory exception if you give it a YAML file greater than 3 megabytes. If you're an online service where untrusted users can submit arbitrary YAML, and an out-of-memory exception is a severe problem, then it's severity 10. If you're an online service that doesn't use yaml in any way, but your web framework bundled the library as a transitive dependency because yaml is one of their five supported configuration options, then it's severity 2. The problem is figuring out which of those situations you're in takes a load of time - and the flow of CVEs is endless, as CVE numbers are given out like candy at halloween. Often it's quicker to just update to the latest version of the YAML library.
anygivnthursday 2mo ago
If I remember correctly, we had to patch or provide justification for CVEs flagged by tools like AWS Inspector for SOC2 as well.
- YeahThisIsMe 2mo ago
  So you didn't have to patch all of them.
  - bluGill 2mo ago
    No, but if you don't patch them you need to convince an auditor that they are not a problem. Often patching is easier. I'm working on such a problem now - we are using an old web browser (no longer supported) to show help on one system. That is web pages were generate internally, with no links elsewhere, and no provision for the user to enter a URL. It is still easier port to a newer supported browser than to convince the auditors that that we are not exploitable. Sure it is obvious that everything is internal and we won't write html that exploits bugs, but nobody wants to convince an auditor of that.
jeltz 2mo ago
Many large organizations like banks have requirements like this and they solve it through a mix of automatic scanners, e.g. Trivvy, and self-deception as not all systems are actually scanned in any sufficiently large org.
traceroute66 2mo ago
> I'm very curious what organisations would have such a policy. I would humbly suggest any org of any size that has insurance cover that covers anything tech related (e.g. data loss/recovery, cyber etc.) has a very good look at the small print. Over the last few years insurers have aggressively been adding "no vulnerability patch, no claim" exclusion clauses.
- saghm 2mo ago
  Yeah, policies like this are often not coming from engineering directly but often through other parts of the company like legal, or even sales from contract negotiations. Not that it's entirely comparable, but I was at AWS when the big log4j vulnerability happened, and the handling for it was not left up to individual engineering teams, which I don't think would surprise anyone. At a large enough company, processes for handling things like security vulnerabilities will have a lot of stakeholders with incentives that are not necessarily perfectly aligned.
- dns_snek 2mo ago
  > Over the last few years insurers have aggressively been adding "no vulnerability patch, no claim" exclusion clauses. If you take an even closer look- what exactly does that mean? It should stipulate patching real vulnerabilities in your system and not "all CVEs in all dependencies irrespective of their usage or applicability to your system", that's madness. It's like voiding your health insurance policy because a smoker moved in across the street. Most vulnerabilities in your dependencies don't become vulnerabilities in your system, and most vulnerabilities in your system (probably) don't originate from your dependencies.
  - traceroute66 2mo ago
    > If you take an even closer look- what exactly does that mean? Here is one example: "Critical Vulnerability Exclusion We will not pay you under the cyber and data risks section of cover where your legal liability or any loss that you suffer arises from a cyber attack that exploits a critical vulnerability within your computer equipment. However this exclusion will only apply where a patch or fix for any critical vulnerability exploited has been available for 21 days prior to the date of the incident and has not been applied to your computer equipment. Critical Vulnerability means a common vulnerability and exposure (CVE) in the National Vulnerability Database operated by the National Institute of Standards and Technology which has a score of 8.0 or higher on the Common Vulnerability Scoring System (CVSS)"
ignore_prev 2mo ago
[flagged]
vrighter 2mo ago
I have been given a list by security. "We had an automated tool scan that machine. It reported these. Fix anything medium severity and above. Never mind that some of them involved vulnerabilities in some part of the bluetooth stack (servers in our datacenter don't even have bluetooth). But they just didn't care
- ptx 2mo ago
  This does make some sense if it's considered a valid fix to document that you have verified that Bluetooth is disabled on the servers and therefore not vulnerable. But that assumes that the scanning tool can be told about this kind of fix, so that it stops warning about it, which I guess it might not.
- SoftTalker 2mo ago
  So run apt full-upgrade and get the new bluetooth driver. Why bother with a fight over something that isn't even used? Just do the quickest thing to get it off your plate.
  - icedchai 2mo ago
    This might cause other problems, problems of the "if it's not broken, don't fix it" variety. Upgrading everything only to break something else, in a previously stable configuration, isn't worth it.
    - pixl97 2mo ago
      Really the days of "Lets run this stable configuration forever" are gone. Getting rid of as much stuff in your OS and software stack as possible should be the security teams ultimate goal, so you have less to upgrade in the end. But actual security updates just come out at a tremendous rate, and you need a QA system that checks as much as it can before prod is upgraded.
      - icedchai 2mo ago
        I do agree with more frequent upgrades, but it has to be part of the culture. The longer you wait, the harder it gets. Unfortunately, I have worked in some heavily tech-debt-laden environments where upgrading anything required an act of god. I've logged on to production servers with 1500+ days of uptime at multiple companies. Nothing had been updated since well before that time. At one place, I recall encountering a 10+ year old dependency. On top of that, they were still using Python 2.7. This wasn't that long ago.
michaelt 2mo ago
SOC2 CC7.1 [1] requires a vulnerability scanner, findings tracked with tickets, assigned severities according to a documented risk-based system, severity-based SLAs for remediation, and that the SLAs mostly be complied with or have tracked exceptions. However it doesn't mandate any particular SLA, or the details of how risks are to be evaluated. Organisations get to write their own policy, and they don't need to commit to patching every CVE within 24 hours or anything like that. [1] https://www.compliancebase.org/controls/soc-2/cc7-1 https://www.compliancebase.org/controls/soc-2/cc7-1
- deleted 2mo ago
  [deleted]
regularfry 2mo ago
Any org large enough to have separated the people responsible for the security exposure of the organisation from the developers with familiarity of what's deployed is likely to have done exactly this. The thing you have to remember is that CVEs can be a) scanned for without exerting mental effort, and b) counted.
agilob 2mo ago
It's more common than many think https://old.reddit.com/r/kubernetes/comments/1vb3x2c/where_are_people_getting_the_best_cvefree_images/ https://old.reddit.com/r/kubernetes/comments/1vb3x2c/where_a...
swiftcoder 2mo ago
> I'm very curious what organisations would have such a policy Do you provide SOC2, HIPAA, GDPR, or similar certifications to your b2b customers? Then your tech stack undergoes an annual audit, and in your audit you will need to provide a paper trail for every single vulnerability in your stack. In practice, this means that your audit compliance software (something like Vanta.com) is going to be setup to mandate every CVE in the whole stack is patched within SLA.
jmull 2mo ago
It's quite common in enterprisey environments. For one thing, bigcorps in regulated areas like it a lot. They push hard to get it required by the regulations (in practice if not directly). Although it's quite inefficient, it becomes a regulatory moat. A cost they can bear that potential upstart competitors cannot.

SirFatty 2mo ago

ITAR

lucideer 2mo ago
ITAR has no such hard requirements. Might be some orgs that tell themselves they're attempting this under ITAR but they're not doing it in any comprehensive way. The only thing within ITAR that I'm aware of concerning itself with software supply chain is SP 800-218 requirements & that's just a load of open-to-interpretation weasel words about having CVE detection & automations in place & some defined plans for reducing the number of vulns. Pretty sure that component of it is even eligible for self-assessment.
- bluGill 2mo ago
  Yes and no. ITAR (and other laws like it) are self assessment and don't specifically say thing thing. However your interpretation / self-assessment is subject to various reviews/audits. These days the reviewers are not going to be kind to someone who just says "not an issue", they will demand strong justification. Most organizations take the view that is is easier to fix all CVEs than try to pass audits. Thus by the letter of the law you are correct. However to meet the letter of the law without fixing CVEs is generally seen as harder than thus fixing CVEs. So the effect is ITAR (and similar laws) force you to fix CVEs.
- SirFatty 2mo ago
  Whatever you say, chief. I worked in that environment for quite a while, and maybe on a technicality you're right, the effect outcome is that you will do it if not for all your customers that will require it.

clbrmbr 2mo ago

Many orgs (esp w ISO27000) have a vulnerability management policy that involves patching at least critical CVEs within a short timeline. Tools like trivvy make it possible to do the scans…

lucideer 2mo ago
I've been in such an org, & I've led initiatives to set up automated detection at very large scale. We started by issuing tickets to teams to resolve CVEs within varying timelines - ranging from a 24hr fix to 6 months - connected to the CVSS score. It wasn't viable. - Firstly, you quickly realise how irrelevant CVSS scores are - initiatives like First's EPSS are designed to fix this but they aren't there yet - Secondly, you need to begin implementing localised heuristics to determine exploitable code paths. This has generally been incredibly difficult to do reliably - LLMs have started to make it easier, but it's expensive. - Lastly, you need to factor in consideration of actionable remediation pathways. A dependency upgrade for critical infrastructure might contain breaking changes that take months to fix, or two competing CVEs might be present in interdependent versions of transitive dependencies in your sbom tree. Most orgs aren't applying any of the above three filters to reduce their CVE remediation burden, & even if they are, it's still too high to make zero a viable target. In reality, most orgs aren't doing comprehensive detection to begin with - if you haven't discovered all of your CVEs, your remediation burden is going to be a lot more manageable.
- mr_mitm 2mo ago
  > - Firstly, you quickly realise how irrelevant CVSS scores are Even if you factor in the environmental score? I realize it's a lot more work, but it basically allows you to tune the score to get any value you want.
  - michaelt 2mo ago
    Imagine a YAML parsing library that can cause an out-of-memory exception if you give it a YAML file greater than 3 megabytes. If you're an online service where untrusted users can submit arbitrary YAML, and an out-of-memory exception is a severe problem, then it's severity 10. If you're an online service that doesn't use yaml in any way, but your web framework bundled the library as a transitive dependency because yaml is one of their five supported configuration options, then it's severity 2. The problem is figuring out which of those situations you're in takes a load of time - and the flow of CVEs is endless, as CVE numbers are given out like candy at halloween. Often it's quicker to just update to the latest version of the YAML library.

anygivnthursday 2mo ago

If I remember correctly, we had to patch or provide justification for CVEs flagged by tools like AWS Inspector for SOC2 as well.

YeahThisIsMe 2mo ago
So you didn't have to patch all of them.
- bluGill 2mo ago
  No, but if you don't patch them you need to convince an auditor that they are not a problem. Often patching is easier. I'm working on such a problem now - we are using an old web browser (no longer supported) to show help on one system. That is web pages were generate internally, with no links elsewhere, and no provision for the user to enter a URL. It is still easier port to a newer supported browser than to convince the auditors that that we are not exploitable. Sure it is obvious that everything is internal and we won't write html that exploits bugs, but nobody wants to convince an auditor of that.

jeltz 2mo ago

Many large organizations like banks have requirements like this and they solve it through a mix of automatic scanners, e.g. Trivvy, and self-deception as not all systems are actually scanned in any sufficiently large org.

traceroute66 2mo ago

> I'm very curious what organisations would have such a policy. I would humbly suggest any org of any size that has insurance cover that covers anything tech related (e.g. data loss/recovery, cyber etc.) has a very good look at the small print. Over the last few years insurers have aggressively been adding "no vulnerability patch, no claim" exclusion clauses.

saghm 2mo ago
Yeah, policies like this are often not coming from engineering directly but often through other parts of the company like legal, or even sales from contract negotiations. Not that it's entirely comparable, but I was at AWS when the big log4j vulnerability happened, and the handling for it was not left up to individual engineering teams, which I don't think would surprise anyone. At a large enough company, processes for handling things like security vulnerabilities will have a lot of stakeholders with incentives that are not necessarily perfectly aligned.
dns_snek 2mo ago
> Over the last few years insurers have aggressively been adding "no vulnerability patch, no claim" exclusion clauses. If you take an even closer look- what exactly does that mean? It should stipulate patching real vulnerabilities in your system and not "all CVEs in all dependencies irrespective of their usage or applicability to your system", that's madness. It's like voiding your health insurance policy because a smoker moved in across the street. Most vulnerabilities in your dependencies don't become vulnerabilities in your system, and most vulnerabilities in your system (probably) don't originate from your dependencies.
- traceroute66 2mo ago
  > If you take an even closer look- what exactly does that mean? Here is one example: "Critical Vulnerability Exclusion We will not pay you under the cyber and data risks section of cover where your legal liability or any loss that you suffer arises from a cyber attack that exploits a critical vulnerability within your computer equipment. However this exclusion will only apply where a patch or fix for any critical vulnerability exploited has been available for 21 days prior to the date of the incident and has not been applied to your computer equipment. Critical Vulnerability means a common vulnerability and exposure (CVE) in the National Vulnerability Database operated by the National Institute of Standards and Technology which has a score of 8.0 or higher on the Common Vulnerability Scoring System (CVSS)"

ignore_prev 2mo ago

[flagged]

vrighter 2mo ago

I have been given a list by security. "We had an automated tool scan that machine. It reported these. Fix anything medium severity and above. Never mind that some of them involved vulnerabilities in some part of the bluetooth stack (servers in our datacenter don't even have bluetooth). But they just didn't care

ptx 2mo ago
This does make some sense if it's considered a valid fix to document that you have verified that Bluetooth is disabled on the servers and therefore not vulnerable. But that assumes that the scanning tool can be told about this kind of fix, so that it stops warning about it, which I guess it might not.
SoftTalker 2mo ago
So run apt full-upgrade and get the new bluetooth driver. Why bother with a fight over something that isn't even used? Just do the quickest thing to get it off your plate.
- icedchai 2mo ago
  This might cause other problems, problems of the "if it's not broken, don't fix it" variety. Upgrading everything only to break something else, in a previously stable configuration, isn't worth it.
  - pixl97 2mo ago
    Really the days of "Lets run this stable configuration forever" are gone. Getting rid of as much stuff in your OS and software stack as possible should be the security teams ultimate goal, so you have less to upgrade in the end. But actual security updates just come out at a tremendous rate, and you need a QA system that checks as much as it can before prod is upgraded.
    - icedchai 2mo ago
      I do agree with more frequent upgrades, but it has to be part of the culture. The longer you wait, the harder it gets. Unfortunately, I have worked in some heavily tech-debt-laden environments where upgrading anything required an act of god. I've logged on to production servers with 1500+ days of uptime at multiple companies. Nothing had been updated since well before that time. At one place, I recall encountering a 10+ year old dependency. On top of that, they were still using Python 2.7. This wasn't that long ago.

michaelt 2mo ago

SOC2 CC7.1 [1] requires a vulnerability scanner, findings tracked with tickets, assigned severities according to a documented risk-based system, severity-based SLAs for remediation, and that the SLAs mostly be complied with or have tracked exceptions. However it doesn't mandate any particular SLA, or the details of how risks are to be evaluated. Organisations get to write their own policy, and they don't need to commit to patching every CVE within 24 hours or anything like that. [1] https://www.compliancebase.org/controls/soc-2/cc7-1 https://www.compliancebase.org/controls/soc-2/cc7-1

deleted 2mo ago
[deleted]

regularfry 2mo ago

Any org large enough to have separated the people responsible for the security exposure of the organisation from the developers with familiarity of what's deployed is likely to have done exactly this. The thing you have to remember is that CVEs can be a) scanned for without exerting mental effort, and b) counted.

agilob 2mo ago

It's more common than many think https://old.reddit.com/r/kubernetes/comments/1vb3x2c/where_are_people_getting_the_best_cvefree_images/ https://old.reddit.com/r/kubernetes/comments/1vb3x2c/where_a...

swiftcoder 2mo ago

> I'm very curious what organisations would have such a policy Do you provide SOC2, HIPAA, GDPR, or similar certifications to your b2b customers? Then your tech stack undergoes an annual audit, and in your audit you will need to provide a paper trail for every single vulnerability in your stack. In practice, this means that your audit compliance software (something like Vanta.com) is going to be setup to mandate every CVE in the whole stack is patched within SLA.

jmull 2mo ago

It's quite common in enterprisey environments. For one thing, bigcorps in regulated areas like it a lot. They push hard to get it required by the regulations (in practice if not directly). Although it's quite inefficient, it becomes a regulatory moat. A cost they can bear that potential upstart competitors cannot.