4 ms·

It isn't broken by design. I sympathize and understand it's annoying to have to comb through false positives and mark them as such, but until a level of AI we'r

It isn't broken by design. I sympathize and understand it's annoying to have to comb through false positives and mark them as such, but until a level of AI we're nowhere near being near exists, you can't automate this process, so the only alternative is to ignore all vulnerabilities. Complaining about this is misunderstanding the asymmetrical costs of different types of statistical error. Vulnerability scanners are sensitive by default because the cost of lots of false positives is annoyed developers who have to slow down their delivery cadence. The cost of false negatives is anything from compromising user data to losing your company to bringing down the power grid of a major city depending on what the application is. Which of those is a higher cost? npm can't possibly know the answer to that, so it has to default to assuming security actually matters to you. If it doesn't, your local policies can be more lax, but don't expect the tool to change for you.

bogota 5y ago
That is not the cost of false positives. The cost of false positives is ignoring it completely. I have had jobs where we just block the security scanners because they won’t listen to any feedback about why what they are scanning was intentionally setup for the purpose of testing vulnerabilities on an internal only network. Additionally at other jobs security tickets just start to get ignored because they send too many tickets that do not matter. I feel the security field likes to ignore most feedback and play holier than though. At all companies i worked at i have only had one good security team that worked with people instead of just throwing things over the wall.
- cratermoon 5y ago
  I agree about the false positive problem. Boy who cried wolf and all. I've also worked with security vendors who offer to run "free" vulnerability scans for you, and to absolutely nobody's surprise, they find vulnerabilities that just happen to be the ones that they can fix, if you buy what they are selling. Still, your example is problematic. Beware the "internal-only network". Such a thing has mostly lost meaning today, and it was never much more than a picket fence anyway. "All devices must be capable of maintaining their security policy on an un-trusted network." https://collaboration.opengroup.org/jericho/commandments_v1.2.pdf https://collaboration.opengroup.org/jericho/commandments_v1....
  - watwut 5y ago
    The issue was "intentionally setup for testing purposes".
    - cratermoon 5y ago
      What testing purposes? How long was it up? Without knowing more, that doesn't eliminate it as a problem. There are plenty of cases where something set up "just for testing" ended up being the entry point for attackers.
danabramov 5y ago
>it's annoying to have to comb through false positives and mark them as such There is no way to "mark them as such". That's half of the issue. The other half is that many people reporting these issues have not "opted into" any security tooling and don't understand its tradeoffs. They just ran `npm install`, and npm adopted default behavior of showing these warnings. For a lot of people this is their first programming environment. >you can't automate this process, so the only alternative is to ignore all vulnerabilities. I don't think the answer is necessarily automation. But as a package author, I'd like to be able to mark somewhere that a particular transitive vulnerabilities can't affect my users. Or at least I'd like npm to offer a reliable way to update packages deep in the tree and override the resolved versions. Currently, there isn't such a way.
- eropple 5y ago
  > The other half is that many people reporting these issues have not "opted into" any security tooling and don't understand its tradeoffs. ... For a lot of people this is their first programming environment. This attitude makes me kind of uncomfortable. Like, I have taught software development to a decent number of folks, but I've always done so in a relatively isolated environment. If one is buying into web programming, I have a hard time feeling like it matters that it's their first programming environment--it is a hostile place (the web) and some understanding of that hostility is pretty high on the list, I think, of Things To Get Used To. There's definitely a tension there with "don't overwhelm a novice", but I don't necessarily think optimizing for the novice case is wise, especially when we want those novices to have their heads on a swivel, too. > But as a package author, I'd like to be able to mark somewhere that a particular transitive vulnerabilities can't affect my users. I definitely agree with this, though, and this is a good way to help make something like `npm audit` more intelligible and useful.
  - watwut 5y ago
    Frontend is however typical first place to develop in. And backend should never trust frontend anyway, making frontend actually lower risk place. You will hack yourself basically.
three14 5y ago
The alternative is hiding most of the reports by default. It's a firehose, and almost all of it is guaranteed to be useless. For example, denial of service in a devDependency shouldn't be shown unless the user specifically requests it. Denial of service shouldn't be marked "high" importance, even though occasionally it might be quite important to some user.

bogota 5y ago

That is not the cost of false positives. The cost of false positives is ignoring it completely. I have had jobs where we just block the security scanners because they won’t listen to any feedback about why what they are scanning was intentionally setup for the purpose of testing vulnerabilities on an internal only network. Additionally at other jobs security tickets just start to get ignored because they send too many tickets that do not matter. I feel the security field likes to ignore most feedback and play holier than though. At all companies i worked at i have only had one good security team that worked with people instead of just throwing things over the wall.

cratermoon 5y ago
I agree about the false positive problem. Boy who cried wolf and all. I've also worked with security vendors who offer to run "free" vulnerability scans for you, and to absolutely nobody's surprise, they find vulnerabilities that just happen to be the ones that they can fix, if you buy what they are selling. Still, your example is problematic. Beware the "internal-only network". Such a thing has mostly lost meaning today, and it was never much more than a picket fence anyway. "All devices must be capable of maintaining their security policy on an un-trusted network." https://collaboration.opengroup.org/jericho/commandments_v1.2.pdf https://collaboration.opengroup.org/jericho/commandments_v1....
- watwut 5y ago
  The issue was "intentionally setup for testing purposes".
  - cratermoon 5y ago
    What testing purposes? How long was it up? Without knowing more, that doesn't eliminate it as a problem. There are plenty of cases where something set up "just for testing" ended up being the entry point for attackers.

danabramov 5y ago

>it's annoying to have to comb through false positives and mark them as such There is no way to "mark them as such". That's half of the issue. The other half is that many people reporting these issues have not "opted into" any security tooling and don't understand its tradeoffs. They just ran `npm install`, and npm adopted default behavior of showing these warnings. For a lot of people this is their first programming environment. >you can't automate this process, so the only alternative is to ignore all vulnerabilities. I don't think the answer is necessarily automation. But as a package author, I'd like to be able to mark somewhere that a particular transitive vulnerabilities can't affect my users. Or at least I'd like npm to offer a reliable way to update packages deep in the tree and override the resolved versions. Currently, there isn't such a way.

eropple 5y ago
> The other half is that many people reporting these issues have not "opted into" any security tooling and don't understand its tradeoffs. ... For a lot of people this is their first programming environment. This attitude makes me kind of uncomfortable. Like, I have taught software development to a decent number of folks, but I've always done so in a relatively isolated environment. If one is buying into web programming, I have a hard time feeling like it matters that it's their first programming environment--it is a hostile place (the web) and some understanding of that hostility is pretty high on the list, I think, of Things To Get Used To. There's definitely a tension there with "don't overwhelm a novice", but I don't necessarily think optimizing for the novice case is wise, especially when we want those novices to have their heads on a swivel, too. > But as a package author, I'd like to be able to mark somewhere that a particular transitive vulnerabilities can't affect my users. I definitely agree with this, though, and this is a good way to help make something like `npm audit` more intelligible and useful.
- watwut 5y ago
  Frontend is however typical first place to develop in. And backend should never trust frontend anyway, making frontend actually lower risk place. You will hack yourself basically.

three14 5y ago

The alternative is hiding most of the reports by default. It's a firehose, and almost all of it is guaranteed to be useless. For example, denial of service in a devDependency shouldn't be shown unless the user specifically requests it. Denial of service shouldn't be marked "high" importance, even though occasionally it might be quite important to some user.