6 ms·

Open Source authors [1] [2] (including myself) have complained of automatic security scans. They yield way too many false positives, increasing the burden of ma

Open Source authors [1] [2] (including myself) have complained of automatic security scans. They yield way too many false positives, increasing the burden of maintaining repositories. Specially troublesome are when e.g. the "vulnerability" (if it's even one) is in a devDependency that is not deployed to production. In theory automatic vulnerability scans sounds great, but having every repo ping you with not-actually-an-issue becomes a chore very quickly. So far the vast majority of vulnerabilities I've seen are actually noise/not applicable. If this code checker is actually good, unlike all of the previous ones, that's another thing and might actually be a game changer. Prominent open source authors have often suggested ways that GIthub can help but seem to be ignored, e.g. allowing to add friction to opening random issues would benefit open source greatly. At some point many beginner devs migrated from StackOverflow to Github because their really bad question were being closed there, and now they just overwhelm open source authors. [1] https://twitter.com/sindresorhus/status/1123986529498664961 https://twitter.com/sindresorhus/status/1123986529498664961 [2] https://twitter.com/FPresencia/status/1311551520689713152 https://twitter.com/FPresencia/status/1311551520689713152

_the_inflator 6y ago
It is tricky. I would say the perfect tool still needs to be developed. And I agree with you. Tools need to get better in detecting dependencies. However, it can be helpful to know at least that you rely on vulnerable code even if it does not get deployed to prod. To be automated, security scans are like unit tests. Just because a unit test is green does not mean your app is working and vice versa. So security scans are more like test levels: unit tests, integration tests, and end-to-end tests. Different scans, different results, and it takes us, humans, still to put it in perspective.
- deleted 6y ago
  [deleted]
zdw 6y ago
Github's notification system is incredibly spammy if you have a lot of repos, and there's no obvious way to manage it. There's also a huge need for an "unsubscribe all" in the notification inbox. There's also not a severity indicator - some minor issue not encountered in normal use is just as noisy as an extremely important issue that affects every user. Other tools like Jira and Gerrit are far better at this.
- beardedwizard 6y ago
  The GitHub notifications story is really quite poor for developers, it's extremely difficult to get an alert routed only to the person who triggers it.
hyperrail 6y ago
Microsoft has used products from Semmle (the now-GitHub and thus now-Microsoft division whose tech is in GitHub code scanning) for a few years, and I've personally used it on occasion. From that limited experience, I'd say that false positives are less of a problem with Semmle's checkers than with other security-focused static analysis tools. This is partly due to Semmle checkers being much more customizable; Semmle has developed a declarative query language called CodeQL* which its checkers' built-in and user-provided rules are written in. Microsoft's security development lifecycle has a lot of mandates which are captured by custom CodeQL rules precisely enough to match their intent. You can see some examples of how Microsoft uses Semmle here: https://msrc-blog.microsoft.com/2018/08/16/vulnerability-hunting-with-semmle-ql-part-1/ https://msrc-blog.microsoft.com/2018/08/16/vulnerability-hun... * https://github.com/github/codeql https://github.com/github/codeql
geewee 6y ago
That's actually a very interesting point in regards to switching from StackOverflow to Github - I have noticed the trend that what I could normally find on Stackoverflow, I now often find on Github issues
- Cthulhu_ 6y ago
  Yeah, I wouldn't mind if they added an extra tab for Q&A, with additional features to make it work like a community wiki (like what SO tries to do).
  - marksomnian 6y ago
    Already in beta: http://github.com/vercel/next.js/discussions http://github.com/vercel/next.js/discussions
- franciscop 6y ago
  Seeing many people surprised or like "finally someone said it", I thought this was very common knowledge? I might have been in some specific communities at the critical point 2-3 years ago, where bootcamps and other educators would advice new devs not to go to StackOverflow but instead go to Github. It didn't occur to me back then that the consequences would be so bad.
stevula 6y ago
Just because a dependency isn’t deployed to production doesn’t mean it’s safe. Remember when eslint tried to steal people’s npm credentials? https://news.ycombinator.com/item?id=17513709 https://news.ycombinator.com/item?id=17513709
- deleted 6y ago
  [deleted]
- inbx0 6y ago
  Sure, but the reported issues are usually not nearly that severe. Most of the time what I see is "A dependency of a dependency of a dependency of Webpack is vulnerable to a Regular expression Denial of Service attack" or prototype pollution or something like that.
  - lhnz 6y ago
    Webpack generates code which might be loaded by users and additionally it's very common to run it on a CI server (which can sometimes have network access out to other machines at a firm). In general, it's quite strange to me that vulnerabilities in `devDependencies` are considered less important than those in `dependencies`. These dependencies are generally for tools that are run within your company network, and contrary to what people insist, that seems quite risky to me.
    - falcor84 6y ago
      What would be the attack vector in this case? Assuming that it's a vulnerability rather than an actually malicious package, how would an attacker exploit something running on your CI server? The only way I see is if they already have the ability to modify your source code, at which point it's of course already game over.
      - beardedwizard 6y ago
        "It's already game over", a favorite phrase of many, but a sign you are giving up before the race has even started. Security and insecurity theater at once.
cik 6y ago
There's a genuine security fatigue issue (much like event fatigue) that comes from false positives. Unfortunately that doesn't reduce the value of the scanning - the onus is on the false positives. At the very least, running and pruning scans should happen on projects so that at least we can have the conversation. It's like PCI (as an example, not an ideal); PCI isn't perfect, but at least it encourages a conversation about security. Today we're at the point where almost every single organization at least discusses security; but I remember when PCI first came out. I can't tell you have many times people used to ask why it was problematic to store passwords in plain text. It this the best step forward, probably not. Is it a step forward, absolutely.
- nurettin 6y ago
  Automated proof of vulnerability is valuable. Kafkaesque pattern matching, is not.
  - Closi 6y ago
    These warnings are disorientingly and illogically complex in a surreal or nightmarish way?
    - nurettin 6y ago
      You did find a mild way of putting it.
  - cik 6y ago
    That's only true based on a certain level of care, which sadly is not always the case. Say you have a codebase that wasn't built well. Let's say your code scanner finds you at the very least the text "-----BEGIN R/DSA PRIVATE KEY-----*-----END R/DSA PRIVATE KEY-----". In my codebases that'll find nothing, but for many people, they just found SSH keys committed in plaintext. The same is true for any 1k/2k (++etc) string. It highlights what could be a private key and worth at least looking at. Neither is proof of a vulnerability. Both I will gladly, always 100% validate.
Dolores12 6y ago
For every automated tool we need another automated tool that cancels first one.
iamflimflam1 6y ago
At some point many beginner devs migrated from StackOverflow to Github because their really bad question were being closed there, and now they just overwhelm open source authors I'm glad it's not just me seeing this - My repos aren't even that popular and some of the issues just seem to be "help me build my project..."
- franciscop 6y ago
  I started just quoting them back prices when they send me random emails like that. What'd you think, they disappear quickly!
  - iamflimflam1 6y ago
    Now that's an idea I can get on board with.
Quarrelsome 6y ago
I don't mind help in finding these problems, what I LOATHE though is when people trust the tool more than me and (for example) prevent me from pushing something that disagrees with the tool. So somewhere I imagine some manager will force their devs to make this tool happy and that's wrong.
dwheeler 6y ago
> Open Source authors (including myself) have complained of automatic security scans. They yield way too many false positives, increasing the burden of maintaining repositories. Unfortunately all tools have either false positives or false negatives, and in practice often both. Tools can (and should) take steps to minimize them or their impact. Nothing makes you use any specific tools; if you don't like what a tool does, don't use it. > Specially troublesome are when e.g. the "vulnerability" (if it's even one) is in a devDependency that is not deployed to production. This particular GitHub tool is for analyzing source code, and would not not normally analyze your dependencies. So that doesn't seem relevant in this case. Of course, someone could mindlessly use this tool (or look at its results) and complain. That's easy, just ask for funding to fix the problem, or at least a pull request that fixes it.
ticmasta 6y ago
We have this problem with our non- OS project but found that after the initial review (of which most were false-positives) we could permanently suppress them with a comment and fail the build for net-new work without a huge impact on developers.

_the_inflator 6y ago

It is tricky. I would say the perfect tool still needs to be developed. And I agree with you. Tools need to get better in detecting dependencies. However, it can be helpful to know at least that you rely on vulnerable code even if it does not get deployed to prod. To be automated, security scans are like unit tests. Just because a unit test is green does not mean your app is working and vice versa. So security scans are more like test levels: unit tests, integration tests, and end-to-end tests. Different scans, different results, and it takes us, humans, still to put it in perspective.

deleted 6y ago
[deleted]

zdw 6y ago

Github's notification system is incredibly spammy if you have a lot of repos, and there's no obvious way to manage it. There's also a huge need for an "unsubscribe all" in the notification inbox. There's also not a severity indicator - some minor issue not encountered in normal use is just as noisy as an extremely important issue that affects every user. Other tools like Jira and Gerrit are far better at this.

beardedwizard 6y ago
The GitHub notifications story is really quite poor for developers, it's extremely difficult to get an alert routed only to the person who triggers it.

hyperrail 6y ago

Microsoft has used products from Semmle (the now-GitHub and thus now-Microsoft division whose tech is in GitHub code scanning) for a few years, and I've personally used it on occasion. From that limited experience, I'd say that false positives are less of a problem with Semmle's checkers than with other security-focused static analysis tools. This is partly due to Semmle checkers being much more customizable; Semmle has developed a declarative query language called CodeQL* which its checkers' built-in and user-provided rules are written in. Microsoft's security development lifecycle has a lot of mandates which are captured by custom CodeQL rules precisely enough to match their intent. You can see some examples of how Microsoft uses Semmle here: https://msrc-blog.microsoft.com/2018/08/16/vulnerability-hunting-with-semmle-ql-part-1/ https://msrc-blog.microsoft.com/2018/08/16/vulnerability-hun... * https://github.com/github/codeql https://github.com/github/codeql

geewee 6y ago

That's actually a very interesting point in regards to switching from StackOverflow to Github - I have noticed the trend that what I could normally find on Stackoverflow, I now often find on Github issues

Cthulhu_ 6y ago
Yeah, I wouldn't mind if they added an extra tab for Q&A, with additional features to make it work like a community wiki (like what SO tries to do).
- marksomnian 6y ago
  Already in beta: http://github.com/vercel/next.js/discussions http://github.com/vercel/next.js/discussions
franciscop 6y ago
Seeing many people surprised or like "finally someone said it", I thought this was very common knowledge? I might have been in some specific communities at the critical point 2-3 years ago, where bootcamps and other educators would advice new devs not to go to StackOverflow but instead go to Github. It didn't occur to me back then that the consequences would be so bad.

stevula 6y ago

Just because a dependency isn’t deployed to production doesn’t mean it’s safe. Remember when eslint tried to steal people’s npm credentials? https://news.ycombinator.com/item?id=17513709 https://news.ycombinator.com/item?id=17513709

deleted 6y ago
[deleted]
inbx0 6y ago
Sure, but the reported issues are usually not nearly that severe. Most of the time what I see is "A dependency of a dependency of a dependency of Webpack is vulnerable to a Regular expression Denial of Service attack" or prototype pollution or something like that.
- lhnz 6y ago
  Webpack generates code which might be loaded by users and additionally it's very common to run it on a CI server (which can sometimes have network access out to other machines at a firm). In general, it's quite strange to me that vulnerabilities in `devDependencies` are considered less important than those in `dependencies`. These dependencies are generally for tools that are run within your company network, and contrary to what people insist, that seems quite risky to me.
  - falcor84 6y ago
    What would be the attack vector in this case? Assuming that it's a vulnerability rather than an actually malicious package, how would an attacker exploit something running on your CI server? The only way I see is if they already have the ability to modify your source code, at which point it's of course already game over.
    - beardedwizard 6y ago
      "It's already game over", a favorite phrase of many, but a sign you are giving up before the race has even started. Security and insecurity theater at once.

cik 6y ago

There's a genuine security fatigue issue (much like event fatigue) that comes from false positives. Unfortunately that doesn't reduce the value of the scanning - the onus is on the false positives. At the very least, running and pruning scans should happen on projects so that at least we can have the conversation. It's like PCI (as an example, not an ideal); PCI isn't perfect, but at least it encourages a conversation about security. Today we're at the point where almost every single organization at least discusses security; but I remember when PCI first came out. I can't tell you have many times people used to ask why it was problematic to store passwords in plain text. It this the best step forward, probably not. Is it a step forward, absolutely.

nurettin 6y ago
Automated proof of vulnerability is valuable. Kafkaesque pattern matching, is not.
- Closi 6y ago
  These warnings are disorientingly and illogically complex in a surreal or nightmarish way?
  - nurettin 6y ago
    You did find a mild way of putting it.
- cik 6y ago
  That's only true based on a certain level of care, which sadly is not always the case. Say you have a codebase that wasn't built well. Let's say your code scanner finds you at the very least the text "-----BEGIN R/DSA PRIVATE KEY-----*-----END R/DSA PRIVATE KEY-----". In my codebases that'll find nothing, but for many people, they just found SSH keys committed in plaintext. The same is true for any 1k/2k (++etc) string. It highlights what could be a private key and worth at least looking at. Neither is proof of a vulnerability. Both I will gladly, always 100% validate.

Dolores12 6y ago

For every automated tool we need another automated tool that cancels first one.

iamflimflam1 6y ago

At some point many beginner devs migrated from StackOverflow to Github because their really bad question were being closed there, and now they just overwhelm open source authors I'm glad it's not just me seeing this - My repos aren't even that popular and some of the issues just seem to be "help me build my project..."

franciscop 6y ago
I started just quoting them back prices when they send me random emails like that. What'd you think, they disappear quickly!
- iamflimflam1 6y ago
  Now that's an idea I can get on board with.

Quarrelsome 6y ago

I don't mind help in finding these problems, what I LOATHE though is when people trust the tool more than me and (for example) prevent me from pushing something that disagrees with the tool. So somewhere I imagine some manager will force their devs to make this tool happy and that's wrong.

dwheeler 6y ago

> Open Source authors (including myself) have complained of automatic security scans. They yield way too many false positives, increasing the burden of maintaining repositories. Unfortunately all tools have either false positives or false negatives, and in practice often both. Tools can (and should) take steps to minimize them or their impact. Nothing makes you use any specific tools; if you don't like what a tool does, don't use it. > Specially troublesome are when e.g. the "vulnerability" (if it's even one) is in a devDependency that is not deployed to production. This particular GitHub tool is for analyzing source code, and would not not normally analyze your dependencies. So that doesn't seem relevant in this case. Of course, someone could mindlessly use this tool (or look at its results) and complain. That's easy, just ask for funding to fix the problem, or at least a pull request that fixes it.

ticmasta 6y ago

We have this problem with our non- OS project but found that after the initial review (of which most were false-positives) we could permanently suppress them with a comment and fail the build for net-new work without a huge impact on developers.