5 ms·
As others have mentioned, while this may seem like a good idea, the results are often bizarre, and it's not hard to see why - the metrics and algorithm are here
by hn_throwaway_99 6y ago
As others have mentioned, while this may seem like a good idea, the results are often bizarre, and it's not hard to see why - the metrics and algorithm are here: https://github.com/ossf/criticality_score#criticality-score https://github.com/ossf/criticality_score#criticality-score.
That algorithm seems unnecessarily complicated and includes somewhat dubious metrics when, in my mind, the only thing that really "counts" when it comes to "criticality" are "how many other things use/depend on me", and there are much easier ways to determine this:
1. For languages with a common package repo, how many other packages depend on me? With NPM, for example, it's pretty easy to figure out how many other packages depend on a given package.
2. For "top level" projects, look at downloads.
My guess is just looking at either (or both) of those metrics would give you better results.
- abharya 6y ago- Other metrics such as how many contributors and organizations are involved, how many user feature requests and bugs getting reported, those are all important project importance and not just "dependency count". some projects can be standalone, so as per your algo, those should be very low. - Downloads data is not available for most repos, please find a reliable metric to use. - Package repo dependencies works, but it does not work for C/C++. At some point, we will integrate github's dependency count info as well.
- detaro 6y agoSo basically the algorithm is designed to not find projects that are critical because they are deep in the foundations, depended on by nearly everyone but only worked on by a few people? (which is what "critical" would suggest at least to me) This seems to be mostly a "github marketing index"... I seriously hope no actual decisions about resource allocation etc are made based on this.
- abharya 6y agoWe are working on this problem, it is not simple. Identifying dependency trees reliably across languages is not straightforward [only nice for package manager ones]. Follow https://github.com/ossf/criticality_score/issues/8 https://github.com/ossf/criticality_score/issues/8
- orf 6y agoYou didn’t identify certifi, urllib3, chardet or pytz in your top 10 critical Python dependencies. These are all highly download packages, mostly maintained by one person, which are totally critical to millions of other packages and the Python ecosystem as a whole. A few of your top-10 I can agree with, but when you’re saying a home-automation package (“core”) is more critical than something like pytz then something has gone terribly wrong.
- abharya 6y agoFiled https://github.com/ossf/criticality_score/issues/20 https://github.com/ossf/criticality_score/issues/20, we will fix this, have an idea on the issue.
- outsomnia 6y agoGithub's downloads don't tell you when FAANG added your project to their internal project tree, which is then deployed in hundreds of millions of devices.
- throwaway342422 6y ago...not to mention all the critical software running on power plants, industry, aircrafts, ships, cars, satellites, core Internet routing... Did i forget military, e.g. weapons?
- rhodozelia 6y agoI don’t see any open source software in power generation. Vxworks on plcs and then osi pi and some horrible HMI software. One exception is inductive automation ignition HMI which uses java
- throwaway342422 6y agoThe methodology is pretty silly. It rewards activity and popularity. A lot of critical infrastructure software is not active and not often spoken about! Case in point: The https://www.cip-project.org/faq https://www.cip-project.org/faq project highlight the needs of very-long-term support for OS components that run on critical infrastructure such as power stations. The https://www.cip-project.org/faq https://www.cip-project.org/faq project is based on Debian. Very little of it is on github.
- abharya 6y agoWe have to start somewhere. It is understandable that this is not complete, so welcome your ideas to discover such projects. Please think of any metrics/ways to find such projects.
- rhencke 6y agoI strongly recommend you use the packages maintained in Linux distributions as a means for discovery. They're well-organized and maintained and easily accessible programmatically - you can even parse the package dependencies programmatically, as well as have full access to the original source code.
- lkcl 6y agogood advice. "apt-get install apt-rdepends" and it becomes possible to work out the reverse-dependencies of packages. by counting the numbers it becomes pretty blindingly obvious what the critical dependencies are. as mentioned in another post above, bash and glibc6 are blindingly-obviously high on the list... yet the GNU Project receives an unbelievably low amount of funding despite their critical importance. likewise, this particular bug in binutils ld, which centres around the incredibly short-sighted "4GB should be enough for anyone" removal of Dr Stallman's memory-resident algorithms in the late 90s, is having some very serious consequences: https://sourceware.org/bugzilla/show_bug.cgi?id=22831 https://sourceware.org/bugzilla/show_bug.cgi?id=22831 yet because there's no money not even from redhat nobody's looking at it. likewise: PAM no longer has a proper maintainer, and hasn't had for... a decade? these are projects that people are relying on yet completely forgetting they're a critical part of the infrastructure! why? because, just as rhencke said above: they're not on github, they've not got "unnecessary changes" which are counted as "activity to be glorified and worshipped". abharya: i heard on slashdot the intent to start from github, to exclusively focus on github. this will turn out to be a serious mistake.
- mbakke 6y agoOne cool feature of functional package managers such as Nix and Guix is that the dependency graphs are entirely transparent and can be inspected programmatically. I wrote a script that lists the number of dependents for each package in Guix by traversing the package graphs: https://gist.github.com/mbakke/f354272666fbef09c5229f7b853778f1 https://gist.github.com/mbakke/f354272666fbef09c5229f7b85377... Running it takes about 16 seconds on my laptop, and piping to 'grep -v bootstrap | sort -rn | tail -n 20' gives: 16695 guile 16692 ld-wrapper 14686 pkg-config 14512 perl 14444 ncurses 13135 readline 13133 zlib 13022 libffi 12470 xz 12448 libunistring 12433 openssl 12353 bash 12050 libxml2 12016 gettext-minimal 11950 tar 11942 bzip2 11896 expat 11857 tzdata 11850 net-base 11846 python-minimal (GCC, glibc and binutils are missing for complicated reasons, but should be up there with Guile) By changing (all-packages) on line 26 to: (fold-packages (lambda (package result) (if (string-prefix? "python-" (package-name package)) (cons package result) result)) '())) We get the most popular Python projects: 6672 python-setuptools-scm 6667 python-nose 6662 python-pyparsing 6662 python-more-itertools 6661 python-wcwidth 6660 python-py 6660 python-pluggy 6660 python-atomicwrites 6657 python-six 6593 python-mock 6587 python-appdirs 6586 python-lxml 6584 python-filelock 6584 python-distlib 6583 python-sortedcontainers 6580 python-hypothesis 6577 python-elementpath 6576 python-xmlschema 6575 python-pytest 6303 python-libxml2 This approach misses "end user" packages such as browsers and QEMU, but provides some insight into high value targets. For end user packages, Debians "popularity contest" can be useful: https://popcon.debian.org/ https://popcon.debian.org/
- lkcl 6y agoanalysing a source-only-based distro (one where the compiler is required as a dependency) should make the importance of the compiler(s) clearer. source-only distros include macports, gentoo, and so on.
- mbakke 6y agoGuix and Nix _are_ source-based distros. They just happen to have a facility for transparently downloading "binary substitutes" for local builds. The graphs I'm traversing are the build-time dependency graph of all packages. I could also traverse run-time dependency graphs, but then I'd need to actually build all the things, because those are not known up-front unlike Gentoo and Macports. The reason the toolchain is missing is because it is an "implicit input" as opposed to a "normal" input, so it's invisible/transparent to the query I'm making.
- gfxgirl 6y agoMaybe I mis-understood what's supposed to be being checked here. I thought the point was surfacing critical projects that are under supported. So a highly dependent project that has plenty of support for its needs should not need to be brought to anyone's attention. I'm guessing python, llvm, clang, chromium, webkit, C#, visual studio code, off the top of my head are all well funded and supported open source projects. I don't know what good examples of under supported but critical open source projects are. I guess I've read that OpenSSL was massively under supported but apparently the solution chosen was to throw it under the bus and promote libssl or boringssl or something like that (completely out of my expertise)