5 ms·

Security scorecards for open source projects

37 points by aberoham 6y ago

jart 6y ago
The usage example in the README gives Kubernetes a 10/10 on security. Would it score that highly if the tool took into consideration all of its dependencies? https://github.com/kubernetes/kubernetes/blob/master/vendor/modules.txt https://github.com/kubernetes/kubernetes/blob/master/vendor/...
- spydum 6y ago
  Probably not. I think the intent is for big shops to quantify the risk of adopting an open source project. This is just a way to codify how teams justify whether a project is "mature enough" for adoption.
  - toomuchtodo 6y ago
    Appears to be an open source alternative to Black Duck Software. https://www.blackducksoftware.com/ https://www.blackducksoftware.com/
    - abharya 6y ago
      The goals are different. Our goal is to help both open source developer improve their project so that it is more secure for the community in the long term. It also gives the consumer this information for free, rather than paying / subscribing to a service.
- kimsterv 6y ago
  there's an open issue related to this: https://github.com/ossf/scorecard/issues/27 https://github.com/ossf/scorecard/issues/27
ssddanbrown 6y ago
Received these results for my own open source project: ./scorecard --repo=github.com/bookstackapp/bookstack <removed status text> RESULTS ------- Active: Pass 10 CI-Tests: Pass 8 CII-Best-Practices: Fail 10 Code-Review: Fail 10 Contributors: Pass 10 Frozen-Deps: Pass 10 Fuzzing: Fail 3 Pull-Requests: Pass 7 SAST: Fail 0 Security-Policy: Fail 10 Signed-Releases: Fail 0 Signed-Tags: Pass 10 Results appear fair and accurate. I am confused though in how this project is intended to work at a higher level. The blogpost states: > The goal of the Scorecards project is to auto-generate a “security score” for open source projects to help users as they decide the trust, risk, and security posture for their use case. Will there be a centralised site to gather and display scores for open source projects? Or will it be up to the open source projects themselves to integrate this into their pages and, if so, what does that look like? Some kind of badge or a listing of the results as above? Just trying to understand how end-users will be interpreting the results in a consistent and trusted manner.
- abharya 6y ago
  We are still thinking of ways to capture scores for various open source projects in an automated fashion and/or work with GitHub to display this kind of information as part of their user interface. Right now, project maintainers need to run it themselves.
prepend 6y ago
It seems like an awful waste for me to run these on each of my dependencies in real time. I expected some list that is just maintained and pinged for values. Also, this scorecard doesn’t look for CVEs or problems in particular versions. It seems like it’s much more important that there’s a valid vulnerability in version 1.04 that I’m using than the current version has code reviews for everything. The reason I care is that I wish there was some stamp of approval on pypi packages that would make it easier for me to trust particular packages and releases.
- spydum 6y ago
  I think you miss the point then. The things it's looking for are indicators security practices are even in place. If you wanted vuln scanning, many solutions for that today exist. This is almost like a reputation score. The higher it is, the more likely vulnerabilities will be addressed in the future (or outright prevented). I like the idea, I think it will just be tough to work out the right heuristics.
  - prepend 6y ago
    Having a quantitative, ordinal scale gives the wrong picture I think. Since 10/10 sounds good, but doesn’t actually mean something is safe. Also the name is “scorecard” that again connotes that a high score is good. I don’t get a 10/10 on my assignment because I did my homework, said hello to the teacher each morning, and was polite to my classmates. Those are all positive things and if I wanted to measure students they may be a part of the grade. But if I only graded on these factors that’s not terribly useful. Process measures are good when it’s not possible to measure outcome, but I think in this situation more outcome measures could be factored in.
    - abharya 6y ago
      These are just some initial metrics we picked for v1. Please suggest more by filing by a github issue here - https://github.com/ossf/scorecard/issues https://github.com/ossf/scorecard/issues Over time, we will add more real time metrics such as open vulnerabilities, etc

37 points by aberoham 6y ago

jart 6y ago

The usage example in the README gives Kubernetes a 10/10 on security. Would it score that highly if the tool took into consideration all of its dependencies? https://github.com/kubernetes/kubernetes/blob/master/vendor/modules.txt https://github.com/kubernetes/kubernetes/blob/master/vendor/...

spydum 6y ago
Probably not. I think the intent is for big shops to quantify the risk of adopting an open source project. This is just a way to codify how teams justify whether a project is "mature enough" for adoption.
- toomuchtodo 6y ago
  Appears to be an open source alternative to Black Duck Software. https://www.blackducksoftware.com/ https://www.blackducksoftware.com/
  - abharya 6y ago
    The goals are different. Our goal is to help both open source developer improve their project so that it is more secure for the community in the long term. It also gives the consumer this information for free, rather than paying / subscribing to a service.
kimsterv 6y ago
there's an open issue related to this: https://github.com/ossf/scorecard/issues/27 https://github.com/ossf/scorecard/issues/27

ssddanbrown 6y ago

Received these results for my own open source project: ./scorecard --repo=github.com/bookstackapp/bookstack <removed status text> RESULTS ------- Active: Pass 10 CI-Tests: Pass 8 CII-Best-Practices: Fail 10 Code-Review: Fail 10 Contributors: Pass 10 Frozen-Deps: Pass 10 Fuzzing: Fail 3 Pull-Requests: Pass 7 SAST: Fail 0 Security-Policy: Fail 10 Signed-Releases: Fail 0 Signed-Tags: Pass 10 Results appear fair and accurate. I am confused though in how this project is intended to work at a higher level. The blogpost states: > The goal of the Scorecards project is to auto-generate a “security score” for open source projects to help users as they decide the trust, risk, and security posture for their use case. Will there be a centralised site to gather and display scores for open source projects? Or will it be up to the open source projects themselves to integrate this into their pages and, if so, what does that look like? Some kind of badge or a listing of the results as above? Just trying to understand how end-users will be interpreting the results in a consistent and trusted manner.

abharya 6y ago
We are still thinking of ways to capture scores for various open source projects in an automated fashion and/or work with GitHub to display this kind of information as part of their user interface. Right now, project maintainers need to run it themselves.

prepend 6y ago

It seems like an awful waste for me to run these on each of my dependencies in real time. I expected some list that is just maintained and pinged for values. Also, this scorecard doesn’t look for CVEs or problems in particular versions. It seems like it’s much more important that there’s a valid vulnerability in version 1.04 that I’m using than the current version has code reviews for everything. The reason I care is that I wish there was some stamp of approval on pypi packages that would make it easier for me to trust particular packages and releases.

spydum 6y ago
I think you miss the point then. The things it's looking for are indicators security practices are even in place. If you wanted vuln scanning, many solutions for that today exist. This is almost like a reputation score. The higher it is, the more likely vulnerabilities will be addressed in the future (or outright prevented). I like the idea, I think it will just be tough to work out the right heuristics.
- prepend 6y ago
  Having a quantitative, ordinal scale gives the wrong picture I think. Since 10/10 sounds good, but doesn’t actually mean something is safe. Also the name is “scorecard” that again connotes that a high score is good. I don’t get a 10/10 on my assignment because I did my homework, said hello to the teacher each morning, and was polite to my classmates. Those are all positive things and if I wanted to measure students they may be a part of the grade. But if I only graded on these factors that’s not terribly useful. Process measures are good when it’s not possible to measure outcome, but I think in this situation more outcome measures could be factored in.
  - abharya 6y ago
    These are just some initial metrics we picked for v1. Please suggest more by filing by a github issue here - https://github.com/ossf/scorecard/issues https://github.com/ossf/scorecard/issues Over time, we will add more real time metrics such as open vulnerabilities, etc