6 ms·

This resonates. Unfortunately, the CVE database(s) are too noisy to be useful. It could benefit from higher standards and more thorough vetting. (Maybe take som

by mholt 3y ago

This resonates. Unfortunately, the CVE database(s) are too noisy to be useful. It could benefit from higher standards and more thorough vetting. (Maybe take some lessons from academia.) A "security researcher" once filed a CVE for a regular bug in Caddy [0], making claims that were totally provably false. It was assigned 7.5... the same as Heartbleed [1] -- yes, the one that leaked almost all the private encryption keys on the Internet back in 2014. When I appealed to NVD for retraction (or whatever they do in this case), I never heard back, despite several emails / form submissions. It is not well-maintained. It is poorly managed. There is virtually no oversight. NVD is a trainwreck. More recently I inadvertently discovered a 0-day RCE in acme.sh [2]. (ACME clients are security-sensitive contexts since they typically deal with private keys and download signed credentials.) Anyway, it was assigned a CVSS 3.x score of * 9.8 * [3] -- I imagine that should be like "cyber-nuclear meltdown" territory, but no, this was actually benign as far as we can tell. Probably deserves more like a 4 or 5 or something. So, again: There is virtually no oversight. NVD is a trainwreck. Anyway, the whole system is broken, and I'm effectively ignoring CVEs now. But if someone just tells me to patch my <whatever>, I'll probably do it, and that's good enough for me. [0]: https://github.com/caddyserver/caddy/issues/4775 https://github.com/caddyserver/caddy/issues/4775 [1]: https://nvd.nist.gov/vuln/detail/cve-2014-0160 https://nvd.nist.gov/vuln/detail/cve-2014-0160 [2]: https://matt.life/writing/the-acme-protocol-in-practice-and-reality https://matt.life/writing/the-acme-protocol-in-practice-and-... [3]: https://nvd.nist.gov/vuln/detail/CVE-2023-38198 https://nvd.nist.gov/vuln/detail/CVE-2023-38198

jabroni_salad 3y ago
Honestly if you just stick with CISA's known exploited list then you will be ahead of most operators and spend less time ghost hunting stupid shit. I really have come to despise CVSS (and especially how tenable will put predictions in the CVSS column for officially unrated vulns) and how assessors seemingly so not have to justify their 'findings'. Not exploited, no poc, needs root to use, why is this a High? Come back to me with something good.
- 1MachineElf 3y ago
  CISA-KEV is a good list, but updates to it can be delayed by days to months.
resonious 3y ago
It seems a lot of infosec folks have these shallow "X = bad" mappings in their brains. Like in that Caddy issue, "out of bounds read = bad" even though realistically you can't do anything bad with it. I see similar thinking all the time with bug bounties at work. We had an XSS report once, but it was under an odd domain that doesn't host any authenticated resources. Yet "XSS = bad" so the report had a way higher urgency score than it should've. Sure we want to fix it - and we did - but it wasn't a credential-stealing nightmare scenario XSS.
- MattPalmer1086 3y ago
  I certainly find there are many people who take that naive view. The right thing to do is to triage them in your context and make your own risk assessment. The CVSS numbers are a guide to which ones you should prioritise looking at, but not a risk assessment. However, in the security and risk world, you have to take a worst case view in the absence of further reliable information. This can lead to fixing things that maybe don't really need fixing, even when a triage has been done. It's often hard to discover enough reliable information.
- mike_d 3y ago
  > "out of bounds read = bad" even though realistically you can't do anything bad with it Schneier's law [1] is at play here. Especially in situations where the person who wrote the bad code in the first place is later tasked to fix it. If they couldn't see the problem the first time it is going to be tough for them the second time around. I've gazed longingly at a promising bug for weeks with no idea how to weaponize it, only to finally give up and ask someone smarter who tends to point out a trick I had never seen before. Even with a career in security old enough to buy beer, I am still amazed by the clever shit I miss. 1. "Any person can invent a security system so clever that she or he can't think of how to break it."
  - jeremyjh 3y ago
    > If they couldn't see the problem the first time it is going to be tough for them the second time around. No. Its easy to overlook a bug like overflow or use after free, but once you see it, you understand it just fine.
ylk 3y ago
CVSS is not a measure for risk. I feel like it’s pretty hard to define an objective measure for “how much should you care about this” that applies to everyone, since you’d need to know how common the affected software is among all computers and how many of them are using the specific vulnerable configuration, etc. What you can do instead is have a measure for “if this affects you, it’s pretty bad/not that bad/…”. Heartbleed wasn’t an RCE, it just leaked sensitive data. RCE often results in a total loss of confidentiality, integrity and availability, so it receiving a higher severity rating makes sense.
1MachineElf 3y ago
Do you have any thoughts on CVSSv4[0]? It appears to incorporate finer-grained and organization-specific scoring to address issues many have with the one size fits all approach currently used for CVEs. [0] https://www.first.org/cvss/v4-0/ https://www.first.org/cvss/v4-0/
tetha 3y ago
I recently had similar thoughts about a security vulnerability in a storage component, which is sometimes even intended to be exposed to the internet as a static file host. That thing had a security vulnerability such that you can do one curl-call and get the entire environment of the system, including the global, special cased admin user credentials for that system. So, practically speaking, an unpatched system has no authorization and is open to anonymous access. Now I'm not sure if I'm weird, but I have a hard time imagining a worse bug in the authorization system in a data store than this kind of trivially exploitable fail open. Easily in the "Stop and drop everything and start patching". But nah, it's just a 7.5. All is good. On the other hand, the fact that the root user can bypass some restrictions and upload files with some permissions is rated as an 8.8 because it's a possible RCE jump. Naturally.
flanked-evergl 3y ago
> Maybe take some lessons from academia. In my experience, the vetting process in academia is an abject failure and mostly just political theatre.
Aachen 3y ago
You're conflating two systems here: CVSS' ratings and CVE's vulnerability IDing Something receiving 9.8 when you feel like it should be a 4–5 is a common enough complaint, though typically not this extreme. If the parameters are filled in correctly (you didn't mention even checking why the result was 9.8) then that's not a flaw in the CVE system. Conversely, CVE authorities not responding to requests for deletion aren't the same as using an inaccurate impact/exploitability calculation system called CVSS
irdc 3y ago
> (Maybe take some lessons from academia.) One thing that academia takes seriously and the CVE database doesn't is the reputation of the researcher(s). The CVE database allows one to anonymously make a serious claim against other people's work, and thus avoid consequences if the CVE turns out to be incorrect. In academia this would be unacceptable: reputation is everything. Now I'm not saying academia does everything right, and it does have a strong tendency for credentialism which is at odds with how security research has always operated. But it makes judging people by their actions a whole lot easier...
- Kalium 3y ago
  There are still corners of infosec where doing the wrong kind of research will get you the business end of a massive lawsuit... or a pair of shiny new bracelets. A lot of us started in security in an era where that was more of the rule than the exception. That said, reputation is quite real. For example, taviso is well known for finding weird and scary things.
jjav 3y ago
> Unfortunately, the CVE database(s) are too noisy to be useful. I used to hate on CVSS scores a lot more than I do these days. Sure, the scores are rarely correct and sometimes nonsensical. To be fair, it's impossible to have a scoring system that precisely ranks every potential risk in the optimal order for everyone. Technically, the best answer is to take every report, evaluate it internally based on exactly how this code is used, where and for what and create our own risk assessment and prioritize changes based on that. That would get us risk scores as good as we're going to get for our product. Nobody has time for that. Even in the best funded organizations I've seen, it's not going to happen. Turns out it's like an order of magnitude cheaper to just say we must address Critical, High and Mediums and call it a day. As much as it bothers me at a technical level, because I know we're fixing stuff that wasn't worth fixing and leaving out some important fixes that got mis-scored too low, the pragmatic reality is this is easier to do and thus can get done. On average, that will capture many of the things that really did need fixing so our security did get better. Yes, we did waste time fixing some that didn't need it, but it was less wasted time than it would've taken to do a deep evaluation so we still saved some time. So overall, not bad.
citizenpaul 3y ago
I personally think this is all by design to create more jobs in the booming snake oil infosec industry. Hear me out before you get the pitchforks. I've started seeing some of the most incompetent people I used to work with are suddenly now "director of security", "senior infosec" and things like that. These were people that struggled to remember what an IP address was when I worked with them. I highly doubt they all suddenly decided to actually learn anything about how computers work. The icing on my anecdote proof cake is my junkie brother that lives with my mom contacted me to ask about how he could get into the computer security industry about 2 months ago,.
- tru3_power 3y ago
  It’s been like this. There are a lot of people that got into security without really a passion for it. Lots of bs. The way I think of it is all of this revolves around the fire/heat generated from finding vulns (on both sides). If ur not finding vulns/really understand/actively trying to understand then ur just blowing smoke and trying to stay warm. That’s just how it is.

by mholt 3y ago

jabroni_salad 3y ago

Honestly if you just stick with CISA's known exploited list then you will be ahead of most operators and spend less time ghost hunting stupid shit. I really have come to despise CVSS (and especially how tenable will put predictions in the CVSS column for officially unrated vulns) and how assessors seemingly so not have to justify their 'findings'. Not exploited, no poc, needs root to use, why is this a High? Come back to me with something good.

1MachineElf 3y ago
CISA-KEV is a good list, but updates to it can be delayed by days to months.

resonious 3y ago

It seems a lot of infosec folks have these shallow "X = bad" mappings in their brains. Like in that Caddy issue, "out of bounds read = bad" even though realistically you can't do anything bad with it. I see similar thinking all the time with bug bounties at work. We had an XSS report once, but it was under an odd domain that doesn't host any authenticated resources. Yet "XSS = bad" so the report had a way higher urgency score than it should've. Sure we want to fix it - and we did - but it wasn't a credential-stealing nightmare scenario XSS.

MattPalmer1086 3y ago
I certainly find there are many people who take that naive view. The right thing to do is to triage them in your context and make your own risk assessment. The CVSS numbers are a guide to which ones you should prioritise looking at, but not a risk assessment. However, in the security and risk world, you have to take a worst case view in the absence of further reliable information. This can lead to fixing things that maybe don't really need fixing, even when a triage has been done. It's often hard to discover enough reliable information.
mike_d 3y ago
> "out of bounds read = bad" even though realistically you can't do anything bad with it Schneier's law [1] is at play here. Especially in situations where the person who wrote the bad code in the first place is later tasked to fix it. If they couldn't see the problem the first time it is going to be tough for them the second time around. I've gazed longingly at a promising bug for weeks with no idea how to weaponize it, only to finally give up and ask someone smarter who tends to point out a trick I had never seen before. Even with a career in security old enough to buy beer, I am still amazed by the clever shit I miss. 1. "Any person can invent a security system so clever that she or he can't think of how to break it."
- jeremyjh 3y ago
  > If they couldn't see the problem the first time it is going to be tough for them the second time around. No. Its easy to overlook a bug like overflow or use after free, but once you see it, you understand it just fine.

ylk 3y ago

CVSS is not a measure for risk. I feel like it’s pretty hard to define an objective measure for “how much should you care about this” that applies to everyone, since you’d need to know how common the affected software is among all computers and how many of them are using the specific vulnerable configuration, etc. What you can do instead is have a measure for “if this affects you, it’s pretty bad/not that bad/…”. Heartbleed wasn’t an RCE, it just leaked sensitive data. RCE often results in a total loss of confidentiality, integrity and availability, so it receiving a higher severity rating makes sense.

1MachineElf 3y ago

Do you have any thoughts on CVSSv4[0]? It appears to incorporate finer-grained and organization-specific scoring to address issues many have with the one size fits all approach currently used for CVEs. [0] https://www.first.org/cvss/v4-0/ https://www.first.org/cvss/v4-0/

tetha 3y ago

I recently had similar thoughts about a security vulnerability in a storage component, which is sometimes even intended to be exposed to the internet as a static file host. That thing had a security vulnerability such that you can do one curl-call and get the entire environment of the system, including the global, special cased admin user credentials for that system. So, practically speaking, an unpatched system has no authorization and is open to anonymous access. Now I'm not sure if I'm weird, but I have a hard time imagining a worse bug in the authorization system in a data store than this kind of trivially exploitable fail open. Easily in the "Stop and drop everything and start patching". But nah, it's just a 7.5. All is good. On the other hand, the fact that the root user can bypass some restrictions and upload files with some permissions is rated as an 8.8 because it's a possible RCE jump. Naturally.

flanked-evergl 3y ago

> Maybe take some lessons from academia. In my experience, the vetting process in academia is an abject failure and mostly just political theatre.

Aachen 3y ago

You're conflating two systems here: CVSS' ratings and CVE's vulnerability IDing Something receiving 9.8 when you feel like it should be a 4–5 is a common enough complaint, though typically not this extreme. If the parameters are filled in correctly (you didn't mention even checking why the result was 9.8) then that's not a flaw in the CVE system. Conversely, CVE authorities not responding to requests for deletion aren't the same as using an inaccurate impact/exploitability calculation system called CVSS

irdc 3y ago

> (Maybe take some lessons from academia.) One thing that academia takes seriously and the CVE database doesn't is the reputation of the researcher(s). The CVE database allows one to anonymously make a serious claim against other people's work, and thus avoid consequences if the CVE turns out to be incorrect. In academia this would be unacceptable: reputation is everything. Now I'm not saying academia does everything right, and it does have a strong tendency for credentialism which is at odds with how security research has always operated. But it makes judging people by their actions a whole lot easier...

Kalium 3y ago
There are still corners of infosec where doing the wrong kind of research will get you the business end of a massive lawsuit... or a pair of shiny new bracelets. A lot of us started in security in an era where that was more of the rule than the exception. That said, reputation is quite real. For example, taviso is well known for finding weird and scary things.

jjav 3y ago

> Unfortunately, the CVE database(s) are too noisy to be useful. I used to hate on CVSS scores a lot more than I do these days. Sure, the scores are rarely correct and sometimes nonsensical. To be fair, it's impossible to have a scoring system that precisely ranks every potential risk in the optimal order for everyone. Technically, the best answer is to take every report, evaluate it internally based on exactly how this code is used, where and for what and create our own risk assessment and prioritize changes based on that. That would get us risk scores as good as we're going to get for our product. Nobody has time for that. Even in the best funded organizations I've seen, it's not going to happen. Turns out it's like an order of magnitude cheaper to just say we must address Critical, High and Mediums and call it a day. As much as it bothers me at a technical level, because I know we're fixing stuff that wasn't worth fixing and leaving out some important fixes that got mis-scored too low, the pragmatic reality is this is easier to do and thus can get done. On average, that will capture many of the things that really did need fixing so our security did get better. Yes, we did waste time fixing some that didn't need it, but it was less wasted time than it would've taken to do a deep evaluation so we still saved some time. So overall, not bad.

citizenpaul 3y ago

I personally think this is all by design to create more jobs in the booming snake oil infosec industry. Hear me out before you get the pitchforks. I've started seeing some of the most incompetent people I used to work with are suddenly now "director of security", "senior infosec" and things like that. These were people that struggled to remember what an IP address was when I worked with them. I highly doubt they all suddenly decided to actually learn anything about how computers work. The icing on my anecdote proof cake is my junkie brother that lives with my mom contacted me to ask about how he could get into the computer security industry about 2 months ago,.

tru3_power 3y ago
It’s been like this. There are a lot of people that got into security without really a passion for it. Lots of bs. The way I think of it is all of this revolves around the fire/heat generated from finding vulns (on both sides). If ur not finding vulns/really understand/actively trying to understand then ur just blowing smoke and trying to stay warm. That’s just how it is.