4 ms·

Been in similar situations, it is not fun. "You have a vulnerable version of netty" "No, we don't, that's a false positive" "No see, the tool says you need n

by Nursie 4y ago

Been in similar situations, it is not fun. "You have a vulnerable version of netty" "No, we don't, that's a false positive" "No see, the tool says you need netty 4.x and that's version 1.x, you need to update" "OK, but the tool is wrong, it's just picking up anything with 'netty' in the name, and that component is a wrapper around netty for some other thing, it only goes up to 1.8" "You have to update it to 4.x, this is a vulnerability" "Do you understand this isn't part of the thing you're concerned about" "I am only concerned about shipping a product without vulnerabilities" "Well this isn't one, because it's a false positive, here's the CVE, here's what it applies to, here's the link to this package on maven central, it's not part of netty, back off" "But it's insecure, the tool says so" We went round and round for about 3 hours until I told the guy (an infosec 'pro') to leave me alone until he'd figured out how to do his job... he came back to me the next morning for another 3 hour slack argument along the exact same lines at which point I told him to leave me alone permanently and communicate through management if he had anything to say. I'm not massively impressed with infosec as a profession after a few similar encounters.

d4mi3n 4y ago
Security engineer here. This is sadly common. The grim fact seems to be that we have a dearth of information security analysts with engineering experience. If they don't have "Engineer" or similar in their title, odds are they haven't had the pleasure of building or maintaining a non-trivial unit of software over more than a quarter or so. That said, I've seen the inverse problem: engineering staff that either don't understand how their dependencies are managed (not unreasonable for less experienced teams, NPM for example has a tendency for huge dependency trees and this can be a hard problem to manage for a big project) or whom are for a plethora of reasons incentivized to push back on requests for work they don't perceive as important. The middle ground here requires trust between both parties (InfoSec and Engineering org functions). Sadly, not all security programs are well managed and not all engineering teams have mature practices. If you have time or capacity, you can get a lot of leverage here by educating your security analysts on how your technology works (e.g. how the build system functions, how your languages of choice share and bundle code). Trust can be built between engineering and infosec teams by educating where possible. Done well such practices can up-level both sides of these discussions and save you stress in the long run.
- tetha 4y ago
  This is for example why we, as a company, rather took 2 hours to escalate Log4Shell internally. Which was like 2-3 hours before it went globally holy shit. We took the time to analyze the vulnerability, and multiple competent people concluded: Yup. This is serious. Then we figured out plans, mitigations, and communication paths to update the plans and mitigations together with ops, dev and other folks. And then we shoved all of that as a large document into the larger technical organization as "Holy fuck the sky is falling, start patching, here's how to see when you have to, here's how you do. Go". And that's also something I'm currently telling our new full-time security engineer. Be a bit mindful in the communication of criticality and vunlerability. Like, with my team of admins and SREs, you can be like "We're all doomed" and someone will be "but we don't deploy that feature, calm down. Would you like some tea? What evil thing did the CVE say to you?". Or they might escalate accordingly, if it looks bad to them, too. But if you do that to most of our dev-teams? That'll be just an unstructured mess, which will be worse than waiting a moment to structure a proper response to the threat.
- Nursie 4y ago
  > Trust can be built between engineering and infosec teams Yes it can, and I have worked with 'good' infosec people who were all about improving practices together. That attitude goes a long way - developers (well, the good ones) want to ship secure stuff and want to fold in best practices. But the profession is unfortunately rife with people who don't know as much as they think they do, and who wade in heavy-handed, laying down the law to developers who might have been round the block once or twice themselves... I wonder if it wouldn't be better to have a hybrid role that is part of engineering but has specific responsibility in this area, rather than as is often the case, an external party (or a separate team) which is perceived as adversarial.
  - d4mi3n 4y ago
    Some organizations do do this and they call them product security engineers or similar. Sadly, this doesn’t remove the need for an informed organization at large—that function covers a lot more than securing code—but it’s absolutely something that has been done. There is one caveat however: you’ll probably still need dedicated security analysis or security engineers outside the chain of command of your product or engineering org. There is a degree of necessary opposing pressures and incentives for infosec departments vs product departments: product organizations are rewarded for shipping more. Infosec organizations are rewarded for preventing incidents. The most common result of this contention are situations people often lament in the comments here—security programs that are either ineffective, heavy handed, or both. In my mind, a good security program positions itself as an enablement function and should strive to educate and inform over all else. People close to the work will be best positioned to make decisions about effective ways to harden their systems when informed of likely risks, requirements, or other security considerations. With any luck this minimizes the need to mandate security motivated changes and changes they dynamic when your local security professional comes by with specific concerns. It’ll be less of “the sky is falling” and more of “we saw this specific problem and would like to work with you to fix it”.
commandlinefan 4y ago
I sometimes feel that my job description could be "arguing with people who feel passionately about something they don't understand".
- dylan604 4y ago
  Isn't that just a description of life in general?
  - commandlinefan 4y ago
    Not for the people who are arguing with me.
- harveywi 4y ago
  Man: What is the secret to eternal happiness? Guru: To not argue with fools. Man: I disagree. Guru: Yes, you are right.
- d4mi3n 4y ago
  Many advisory jobs fall into this bucket. I'm of the impression that anybody who reaches senior/staff+ roles end up doing this as part of their day to day.
- BrandoElFollito 4y ago
  Like I usually say, everyone is a doctor and a football coach, and recently a security expert. I am coming to a point in my 25+ years career where I stopped to argue. I just tell them that they are wrong but I do not care until they put the company on danger. Unfortunately this leaves me with fat powerpoint presentations which well to be the modern documentation of a company.
Natsu 4y ago
This is where you could probably just have renamed the file.
raesene9 4y ago
Over-reliance on tools is a bad problem in Infosec, often due to the range of areas they have to cover. That said a good infosec person should always have an understanding of where their tools are limited, so they can work around those deficiencies.
mbreese 4y ago
It sounds like some of this should be automated. If all the infosec person is doing is running a tool against a repo and reporting results, that should be automated. When there are false positives, that should be annotated in the repo with multiple people checking off on it. I’ve been impressed by automated bug checking tools in the past and I see this as part of the same issue. I don’t see why this would need an FTE to run code against a tool. CI should be enough.
- dboreham 4y ago
  It's the "intelligent life form determines the alert is a false positive" part that's not automated.
  - mbreese 4y ago
    Of course that’s not automated. From the parent, it sounded like the their issue wasn’t figuring out if something was a false positive. It was that another “intelligent life form” ran a tool and then wouldn’t accept when another “intelligent life form” assessed that a flag was a false positive. If the infosec person is just running a tool and reporting results, why are they part of the loop? Make running the tool mandatory for each git push back to the main repo. Then, if/when there is a false positive, allow them to pass if they’ve already been “approved” by some means (like a .infosec_ignore file).
- Nursie 4y ago
  In this case they weren't even running the tool, just looking over reports generated by the dev team.
BrandoElFollito 4y ago
I run a security team in a large company. What you described is usually the result of some "consulting company" (in our case big ones) that drop stuff with zero actual knowledge. Every year we "review" these with the board and it is annoying as fuck. Recently they got a new manager who understands we are in the same boat. They have to provide a report with some findings and I need to have a secure environment. We talked this over and suddenly the board presentation was cool. As for internal teams, I have a few extraordinary people. They are very young and I had to coach them on how the vulnerability is actually critical or not depending on the context, data, etc. So not all cybersecurity teams suck :)
- d4mi3n 4y ago
  Hi! Question from a fellow practitioner: What tools or processes do you and your team use to identify false positives in such scans and reports? How do you track out-of-band mitigations against real-but-unreachable vulnerabilities that have been identified? e.g. A WAF set to intercept requests that trigger a specific vuln, or network segmentation to limit the scope of a moderate vulnerability. In my experience tuning out false positives and other noise is fairly straightforward when you're only looking at CVEs, but things rapidly break down when you're trying to prioritize work with limited resources (only so many security engineers and engineering teams with spare cycles on their roadmap). I have yet to find a good solution here other than manually whitelisting specific vulns in some kind of tracker and reviewing them whenever the network topology or configuration for an application changes. Curious to hear your experiences, though I suspect this is a common and difficult to solve problem.
  - BrandoElFollito 4y ago
    Well, this is a real circus. I have tested and deployed several scanning tool (and have contributed to nessus some years ago - their scanning capacities were really mediocre compared to the precision of the results. In out 100+k IPs on 10.x networks the time to scan was horrendous. But once an IP was scanned, the results were fine). A small disclaimer: I use several solutions but have no stakes in any of them (neither company nor personal). I have build relationships with them that helps me to provide substantiated feedback that sometimes event get taken into account :) There are several issues, none of them has an automated way of fixing. First there is the criticality of the finding. We found that often (especially on Linux) you will get a "raw" critical issue on a component that was not patched; And why it was not patched? Because the vendor set a low level. It is then that the discussion between the security team and the OS one starts - I have experimentally found that the vendor rating is usually better because they take into account the actual context of the deployment, especially for libraries. OTOH crowdstrike vulnerability rating (I forgot the name of the component, it is on top of the EDR) is actually two ratings: the CVSS-reated one, and their own where they look at actual exploits and deployment. I have decided that critical+critical = actually critical that must be patched no matter what. This is the only way I found to quantitatively auto-decide on important patches. The goal is of course to patch everything but the reality is that there are so many loose ends in products that fighting for anything below the top two levels does not make sense. The very sad thing here is that this is perceived as an issue of the OS/middleware and not the shitty products vendors deploy. They use some undocumented or weak solutions and then cry when everything breaks down. I develop myself (amateur developer) and have never had any issues for 30 years when upgrading the OS because I actually care about following the docs. The second problem is that today products have nested dependencies. For our own products we do an awfully time taking and precise review of the actual impact on the product but this is because people actually care (this is not my team so I ma not taking any credit - I just like their approach). When a customer comes with a question we have a comprehensive response. This works or not, some people when they see "vulnerability" they stop at that and want that word to disappear. When we are on the receiving side of such products (and log4j is a perfect example), we ask the vendor. If this is open source the answer is usually very good, there are some github/gitlab issues that address the point. Commercial vendors are much worse because they either say "no worries" or take weeks to respond. When a "no worry" comment comes from a company that cannot explain why they limit their passwords to 27 characters, it is worrisome - a bit like when my children say "no worries" (this is the moment when I start to worry). Scanners are quite slow to respond to non obvious vulnerabilities (the ones you cannot easily tag though OVAL). For log4j for instance I coded a scanner/reflector on the night of Thursday (the information popped up about noon on Thursday western europe time). The scanners followed up after a week, but one could fine good ones open source quickly as well. So my advice would be to closely monitor GitHub for such tools when there is pressure (we switched to another tool for log4j when my team and the OS ones were successively finding better ones). This also led us to investigate a fantastic tool/framework to write scanners, which name escapes me right now but I can comment here when I am back from vacation and get hold of the gal who is looking at that. The TL;DR version is that - a few tools provide a way to quantify the risk (that's a huge slit but the best I can do) - at least there is a reasonable chance that you will not miss the worst ones - GitHub activity when shit hits the fan is extraordinarily useful - the best tools emerge there, before the vendors catch up - vendors are medium, but is is not as bad as you read - your CISO should make the effort to communicate that there will be availability issues when the IT/security team decides to switch off a key component (say - email) if this presents a risk for the company. I have the chance to have a CEO who not only understands this but actually challenged me once to why I did not cut earlier. Having that arranged upfront talks a huge stress off the security team. - and finally the sad reality is that the vast, vast majority of issues we get are from users who click on stuff I am sure I forgot a lot - let me know if I can help with some details.
dboreham 4y ago
Same experience several times. Tool picks up examples code dependencies from our dependencies is another form of this.

by Nursie 4y ago

d4mi3n 4y ago

Security engineer here. This is sadly common. The grim fact seems to be that we have a dearth of information security analysts with engineering experience. If they don't have "Engineer" or similar in their title, odds are they haven't had the pleasure of building or maintaining a non-trivial unit of software over more than a quarter or so. That said, I've seen the inverse problem: engineering staff that either don't understand how their dependencies are managed (not unreasonable for less experienced teams, NPM for example has a tendency for huge dependency trees and this can be a hard problem to manage for a big project) or whom are for a plethora of reasons incentivized to push back on requests for work they don't perceive as important. The middle ground here requires trust between both parties (InfoSec and Engineering org functions). Sadly, not all security programs are well managed and not all engineering teams have mature practices. If you have time or capacity, you can get a lot of leverage here by educating your security analysts on how your technology works (e.g. how the build system functions, how your languages of choice share and bundle code). Trust can be built between engineering and infosec teams by educating where possible. Done well such practices can up-level both sides of these discussions and save you stress in the long run.

tetha 4y ago
This is for example why we, as a company, rather took 2 hours to escalate Log4Shell internally. Which was like 2-3 hours before it went globally holy shit. We took the time to analyze the vulnerability, and multiple competent people concluded: Yup. This is serious. Then we figured out plans, mitigations, and communication paths to update the plans and mitigations together with ops, dev and other folks. And then we shoved all of that as a large document into the larger technical organization as "Holy fuck the sky is falling, start patching, here's how to see when you have to, here's how you do. Go". And that's also something I'm currently telling our new full-time security engineer. Be a bit mindful in the communication of criticality and vunlerability. Like, with my team of admins and SREs, you can be like "We're all doomed" and someone will be "but we don't deploy that feature, calm down. Would you like some tea? What evil thing did the CVE say to you?". Or they might escalate accordingly, if it looks bad to them, too. But if you do that to most of our dev-teams? That'll be just an unstructured mess, which will be worse than waiting a moment to structure a proper response to the threat.
Nursie 4y ago
> Trust can be built between engineering and infosec teams Yes it can, and I have worked with 'good' infosec people who were all about improving practices together. That attitude goes a long way - developers (well, the good ones) want to ship secure stuff and want to fold in best practices. But the profession is unfortunately rife with people who don't know as much as they think they do, and who wade in heavy-handed, laying down the law to developers who might have been round the block once or twice themselves... I wonder if it wouldn't be better to have a hybrid role that is part of engineering but has specific responsibility in this area, rather than as is often the case, an external party (or a separate team) which is perceived as adversarial.
- d4mi3n 4y ago
  Some organizations do do this and they call them product security engineers or similar. Sadly, this doesn’t remove the need for an informed organization at large—that function covers a lot more than securing code—but it’s absolutely something that has been done. There is one caveat however: you’ll probably still need dedicated security analysis or security engineers outside the chain of command of your product or engineering org. There is a degree of necessary opposing pressures and incentives for infosec departments vs product departments: product organizations are rewarded for shipping more. Infosec organizations are rewarded for preventing incidents. The most common result of this contention are situations people often lament in the comments here—security programs that are either ineffective, heavy handed, or both. In my mind, a good security program positions itself as an enablement function and should strive to educate and inform over all else. People close to the work will be best positioned to make decisions about effective ways to harden their systems when informed of likely risks, requirements, or other security considerations. With any luck this minimizes the need to mandate security motivated changes and changes they dynamic when your local security professional comes by with specific concerns. It’ll be less of “the sky is falling” and more of “we saw this specific problem and would like to work with you to fix it”.

commandlinefan 4y ago

I sometimes feel that my job description could be "arguing with people who feel passionately about something they don't understand".

dylan604 4y ago
Isn't that just a description of life in general?
- commandlinefan 4y ago
  Not for the people who are arguing with me.
harveywi 4y ago
Man: What is the secret to eternal happiness? Guru: To not argue with fools. Man: I disagree. Guru: Yes, you are right.
d4mi3n 4y ago
Many advisory jobs fall into this bucket. I'm of the impression that anybody who reaches senior/staff+ roles end up doing this as part of their day to day.
BrandoElFollito 4y ago
Like I usually say, everyone is a doctor and a football coach, and recently a security expert. I am coming to a point in my 25+ years career where I stopped to argue. I just tell them that they are wrong but I do not care until they put the company on danger. Unfortunately this leaves me with fat powerpoint presentations which well to be the modern documentation of a company.

Natsu 4y ago

This is where you could probably just have renamed the file.

raesene9 4y ago

Over-reliance on tools is a bad problem in Infosec, often due to the range of areas they have to cover. That said a good infosec person should always have an understanding of where their tools are limited, so they can work around those deficiencies.

mbreese 4y ago

It sounds like some of this should be automated. If all the infosec person is doing is running a tool against a repo and reporting results, that should be automated. When there are false positives, that should be annotated in the repo with multiple people checking off on it. I’ve been impressed by automated bug checking tools in the past and I see this as part of the same issue. I don’t see why this would need an FTE to run code against a tool. CI should be enough.

dboreham 4y ago
It's the "intelligent life form determines the alert is a false positive" part that's not automated.
- mbreese 4y ago
  Of course that’s not automated. From the parent, it sounded like the their issue wasn’t figuring out if something was a false positive. It was that another “intelligent life form” ran a tool and then wouldn’t accept when another “intelligent life form” assessed that a flag was a false positive. If the infosec person is just running a tool and reporting results, why are they part of the loop? Make running the tool mandatory for each git push back to the main repo. Then, if/when there is a false positive, allow them to pass if they’ve already been “approved” by some means (like a .infosec_ignore file).
Nursie 4y ago
In this case they weren't even running the tool, just looking over reports generated by the dev team.

BrandoElFollito 4y ago

I run a security team in a large company. What you described is usually the result of some "consulting company" (in our case big ones) that drop stuff with zero actual knowledge. Every year we "review" these with the board and it is annoying as fuck. Recently they got a new manager who understands we are in the same boat. They have to provide a report with some findings and I need to have a secure environment. We talked this over and suddenly the board presentation was cool. As for internal teams, I have a few extraordinary people. They are very young and I had to coach them on how the vulnerability is actually critical or not depending on the context, data, etc. So not all cybersecurity teams suck :)

d4mi3n 4y ago
Hi! Question from a fellow practitioner: What tools or processes do you and your team use to identify false positives in such scans and reports? How do you track out-of-band mitigations against real-but-unreachable vulnerabilities that have been identified? e.g. A WAF set to intercept requests that trigger a specific vuln, or network segmentation to limit the scope of a moderate vulnerability. In my experience tuning out false positives and other noise is fairly straightforward when you're only looking at CVEs, but things rapidly break down when you're trying to prioritize work with limited resources (only so many security engineers and engineering teams with spare cycles on their roadmap). I have yet to find a good solution here other than manually whitelisting specific vulns in some kind of tracker and reviewing them whenever the network topology or configuration for an application changes. Curious to hear your experiences, though I suspect this is a common and difficult to solve problem.
- BrandoElFollito 4y ago
  Well, this is a real circus. I have tested and deployed several scanning tool (and have contributed to nessus some years ago - their scanning capacities were really mediocre compared to the precision of the results. In out 100+k IPs on 10.x networks the time to scan was horrendous. But once an IP was scanned, the results were fine). A small disclaimer: I use several solutions but have no stakes in any of them (neither company nor personal). I have build relationships with them that helps me to provide substantiated feedback that sometimes event get taken into account :) There are several issues, none of them has an automated way of fixing. First there is the criticality of the finding. We found that often (especially on Linux) you will get a "raw" critical issue on a component that was not patched; And why it was not patched? Because the vendor set a low level. It is then that the discussion between the security team and the OS one starts - I have experimentally found that the vendor rating is usually better because they take into account the actual context of the deployment, especially for libraries. OTOH crowdstrike vulnerability rating (I forgot the name of the component, it is on top of the EDR) is actually two ratings: the CVSS-reated one, and their own where they look at actual exploits and deployment. I have decided that critical+critical = actually critical that must be patched no matter what. This is the only way I found to quantitatively auto-decide on important patches. The goal is of course to patch everything but the reality is that there are so many loose ends in products that fighting for anything below the top two levels does not make sense. The very sad thing here is that this is perceived as an issue of the OS/middleware and not the shitty products vendors deploy. They use some undocumented or weak solutions and then cry when everything breaks down. I develop myself (amateur developer) and have never had any issues for 30 years when upgrading the OS because I actually care about following the docs. The second problem is that today products have nested dependencies. For our own products we do an awfully time taking and precise review of the actual impact on the product but this is because people actually care (this is not my team so I ma not taking any credit - I just like their approach). When a customer comes with a question we have a comprehensive response. This works or not, some people when they see "vulnerability" they stop at that and want that word to disappear. When we are on the receiving side of such products (and log4j is a perfect example), we ask the vendor. If this is open source the answer is usually very good, there are some github/gitlab issues that address the point. Commercial vendors are much worse because they either say "no worries" or take weeks to respond. When a "no worry" comment comes from a company that cannot explain why they limit their passwords to 27 characters, it is worrisome - a bit like when my children say "no worries" (this is the moment when I start to worry). Scanners are quite slow to respond to non obvious vulnerabilities (the ones you cannot easily tag though OVAL). For log4j for instance I coded a scanner/reflector on the night of Thursday (the information popped up about noon on Thursday western europe time). The scanners followed up after a week, but one could fine good ones open source quickly as well. So my advice would be to closely monitor GitHub for such tools when there is pressure (we switched to another tool for log4j when my team and the OS ones were successively finding better ones). This also led us to investigate a fantastic tool/framework to write scanners, which name escapes me right now but I can comment here when I am back from vacation and get hold of the gal who is looking at that. The TL;DR version is that - a few tools provide a way to quantify the risk (that's a huge slit but the best I can do) - at least there is a reasonable chance that you will not miss the worst ones - GitHub activity when shit hits the fan is extraordinarily useful - the best tools emerge there, before the vendors catch up - vendors are medium, but is is not as bad as you read - your CISO should make the effort to communicate that there will be availability issues when the IT/security team decides to switch off a key component (say - email) if this presents a risk for the company. I have the chance to have a CEO who not only understands this but actually challenged me once to why I did not cut earlier. Having that arranged upfront talks a huge stress off the security team. - and finally the sad reality is that the vast, vast majority of issues we get are from users who click on stuff I am sure I forgot a lot - let me know if I can help with some details.

dboreham 4y ago

Same experience several times. Tool picks up examples code dependencies from our dependencies is another form of this.