10 ms·

Finding Critical Open Source Projects

137 points by QuantumWasp 6y ago

sova 6y ago
Very interesting and I hope this leads to actual recompense/compensation for people "thanklessly maintaining" OSS. Not sure how it possibly fits into Google's newfound profit-at-all-costs motive, but the gesture is an important first step towards some semblance of justice.
- PradeetPatel 6y ago
  Can you elaborate more on the claimed "profit-at-all-costs" statement? It's been established that the purpose of most corporations are for profit, but iirc most large enterprises have a rather robust ethics framework when working with the OSS community.
  - sova 6y ago
    I wish I had reasonable evidence to point to, but I'm not really an insider. Mainly, I think of the numerous applications Google invested lots of engineer time into to create an experience, say Hangouts, only to realize that it hampered sales in another corner of Google, say cell plans, and so one project must be discarded for profits in the other. The functionality is often difficult to replace for the end-user, and weakens the overall allure of the Google network/ecosystem of applications in favor of a bottom-line.
Svetlitski 6y ago
Sounds interesting in theory, but the results are somewhat bizarre. For example, their published list of the top 200 "most critical" open-source C projects includes Urbit and Stellar, and includes micropython but does not include CPython.
- sitzkrieg 6y ago
  seems like completely arbitrary garbage
  - skinkestek 6y ago
    I.e. worse than ordinary search results?
- abharya 6y ago
  As per Github (see languages section on bottom right), Cpython is 63% Python, and 29% C code, so its api returns Python as main language. cpython is in top 10 in python_top_200.csv
harpiaharpyja 6y ago
Seems like a good direction. My main hope is that if metrics like these become more accepted and/or important, that they find a way to avoid running afoul of Goodhart's law.
cycloptic 6y ago
Lots of things to check out here, and it's limited only to projects that are on github, but unsurprisingly it looks like the top 5 are: - Node (0.984) - Tensorflow (0.969) - Git (0.945) - Linux (0.936) - PHP (0.919)
- thundergolfer 6y ago
  Tensorflow ranking higher than Linux is a weird one. Tensorflow debuted to a huge amount of hype, but it could fall over and nearly all our software systems would be totally fine. The rest in that list seem reasonable.
sien 6y ago
List of the top 200 is at : https://commondatastorage.googleapis.com/ossf-criticality-score/index.html https://commondatastorage.googleapis.com/ossf-criticality-sc... (from the repo) So gnucash is #15 . At #75 is gcc . This seems like a great idea, but perhaps some refinements are needed.
- abharya 6y ago
  World is not perfect. In case of gcc, it is only a mirror on github [https://github.com/gcc-mirror/gcc https://github.com/gcc-mirror/gcc], so we dont get all the metrics, we do plan to improve this part [but this will be slow to clone repo, know their custom issue trackers, etc]. Gnucash is still a popular project, has like ~7k downloads a week, see wikipedia page.
  - deleted 6y ago
    [deleted]
  - breakingcups 6y ago
    Strangely enough that's not mentioned anywhere on the announcement. In fact, it seems to announce that very real funding decisions are going to be made based on this score already. How can you make accurate decisions based on inaccurate data? Which reminds me of that Babbage quote: "On two occasions I have been asked, 'Pray, Mr. Babbage, if you put into the machine wrong figures, will the right answers come out?' I am not able rightly to apprehend the kind of confusion of ideas that could provoke such a question."
  - jcelerier 6y ago
    > Gnucash is still a popular project, has like ~7k downloads a week, see wikipedia page. omg they're actually doubling down on this. Libreoffice has something like 50k downloads per day and isn't listed, while among "critical projects" there is... minetest ? tesseract ? battle for wesnoth ? dolphin ? citra ? There's qt creator (and a ton of Qt-using projects) but not Qt ?
    - abharya 6y ago
Buetol 6y ago
Top 10: - Python: salt, core (https://github.com/home-assistant/core https://github.com/home-assistant/core), pandas, scikit-learn, numpy, airflow, erpnext, matplotlib, pytest & pip - Rust: servo, cargo, rust-clippy, tokio, rust-analyzer, tock, tikv, alacritty, libc & substrate - JS: node, react-native, react, gatsby, three.js, bootstrap, material-ui, odoo, next.js & Rocket.Chat - Java: elasticsearch, flink, spring-boot, hadoop, netty, jenkins, beam, bazel, alluxio & pmd - C++: tensorflow, ceph, pytorch, bitcoin, electron, Marlin, Cataclysm-DDA, llvm-project, rocksdb & QGIS - C: git, linux, linux, php-src, openssl, systemd, curl, u-boot, qemu & mbed-os
- EvilEy3 6y ago
  In what universe Bazel is a critical project for Java ecosystem?
  - lokedhs 6y ago
    The Google universe. Don't they use it for all their builds?
  - tziki 6y ago
    In the Android universe?
  - leegraham 6y ago
    It looks like it's in the Java section because it's written _in_ Java, not because it's critical _to_ Java. Similarly, php-src isn't critical to C.
  - abharya 6y ago
    The list captures projects written in the Java language, not for Java ecosystem. See why it is critical here - https://github.com/bazelbuild/bazel/wiki/Bazel-Users https://github.com/bazelbuild/bazel/wiki/Bazel-Users
- megous 6y ago
  Which of these match the image in the article, where some random Nebraskan is maintaining some obscure dependency?
hn_throwaway_99 6y ago
As others have mentioned, while this may seem like a good idea, the results are often bizarre, and it's not hard to see why - the metrics and algorithm are here: https://github.com/ossf/criticality_score#criticality-score https://github.com/ossf/criticality_score#criticality-score. That algorithm seems unnecessarily complicated and includes somewhat dubious metrics when, in my mind, the only thing that really "counts" when it comes to "criticality" are "how many other things use/depend on me", and there are much easier ways to determine this: 1. For languages with a common package repo, how many other packages depend on me? With NPM, for example, it's pretty easy to figure out how many other packages depend on a given package. 2. For "top level" projects, look at downloads. My guess is just looking at either (or both) of those metrics would give you better results.
- abharya 6y ago
  - Other metrics such as how many contributors and organizations are involved, how many user feature requests and bugs getting reported, those are all important project importance and not just "dependency count". some projects can be standalone, so as per your algo, those should be very low. - Downloads data is not available for most repos, please find a reliable metric to use. - Package repo dependencies works, but it does not work for C/C++. At some point, we will integrate github's dependency count info as well.
  - detaro 6y ago
    So basically the algorithm is designed to not find projects that are critical because they are deep in the foundations, depended on by nearly everyone but only worked on by a few people? (which is what "critical" would suggest at least to me) This seems to be mostly a "github marketing index"... I seriously hope no actual decisions about resource allocation etc are made based on this.
    - abharya 6y ago
      We are working on this problem, it is not simple. Identifying dependency trees reliably across languages is not straightforward [only nice for package manager ones]. Follow https://github.com/ossf/criticality_score/issues/8 https://github.com/ossf/criticality_score/issues/8
eh78ssxv2f 6y ago
I don't understand how gnucash is more critical than Arduino, wine or nginx. Is gnucash very widely used?
- abharya 6y ago
  It does seem to be popular w.r.t user downloads and other github metrics. E.g. https://sourceforge.net/projects/gnucash/files/stats/timeline?period=weekly https://sourceforge.net/projects/gnucash/files/stats/timelin... Wikipedia - "As of July 2018, SourceForge shows a count of over 6.3 million downloads of the stable releases starting from November 1999[24] Also, Sourceforge shows that current downloads are running at ~7,000 per week.[25] This does not include other software download sites as well as Linux distributions that provide download from their own repositories."
  - schoen 6y ago
    > This does not include other software download sites as well as Linux distributions that provide download from their own repositories. Isn't that a pretty significant difficulty for the interpretation of this number, though? Couldn't there be some package that is installed by default in, say, Ubuntu (and that Ubuntu can't boot without), so therefore millions of users are using it -- but perhaps the "downloads" seen by GitHub or SourceForge are mostly by OS packagers or by developers contributing to that package. Whereas if there is something where, for some reason, end-users customarily or commonly get it from GitHub or Sourceforge, it might look more widely used even though it is actually less used overall, potentially even by orders of magnitude. (Like GNU coreutils, at an extreme, is probably not that popular at all for end users to download from a source control system, yet it might be running in tens or hundreds of millions of devices, which could not even boot without it.)
    - abharya 6y ago
      coreutils is in the top 200 list, c_top_200.csv:coreutils,https://github.com/coreutils/coreutils,C,90,0,240,5,2.2,50,1,6,0.7,1393,0.59927 https://github.com/coreutils/coreutils,C,90,0,240,5,2.2,50,1... There will always be edge cases and scenarios we are not taking into account, please provide feedback on issue tracker and provide any suggestions so we can account these.
gojomo 6y ago
Neither the blog post nor project README directly mentions whether the 0.0 to 1.0 score goes from most-critical to least, or least-critical to more. It appears 1.0 is most-critical, if you dig into ranking lists.
- abharya 6y ago
  Thanks, fixed in https://github.com/ossf/criticality_score/commit/fddcc093c7e11f097a447b057976391e1860ae85 https://github.com/ossf/criticality_score/commit/fddcc093c7e...
krzyk 6y ago
Strange list, PMD so high, quarkus also, but e.g. jackson is lower then those both. elasticsearch at the top? flink? It is first time I hear about it and I work in Java shops for quite some time.
- watwut 6y ago
  I mean, our company uses elasticsearch. But like, if it went away, it would not be critical or something. Some features would degrade a bit and no one would cared. But yeah, the list is bizarre. It is more of "list of cool things to read about that you dont really need" then "critical list".
cryptica 6y ago
There are some issues with the criticality score (https://github.com/ossf/criticality_score https://github.com/ossf/criticality_score). For example, it counts the number of issues as determining criticality. If there is any kind of money attached to this score (now or in the future), obviously this is going to encourage people to introduce more bugs into their projects.
rwmj 6y ago
It's nice that they're automating this even though the results seem currently a bit random. Fedora has a concept of Critical Path packages (https://fedoraproject.org/wiki/Critical_path_package https://fedoraproject.org/wiki/Critical_path_package). The concept is related to various activities, eg. a package is critical if is is needed when installing a graphical desktop. But as far as I know packages are picked manually and therefore we probably miss packages or over/underestimate criticality.
- abharya 6y ago
  Thanks for feedback, adding it for analysis in https://github.com/ossf/criticality_score/issues/23 https://github.com/ossf/criticality_score/issues/23
commoner 6y ago
The link is to the home page of the Google Open Source Blog, and not this specific blog post. It should be updated to: https://opensource.googleblog.com/2020/12/finding-critical-open-source-projects.html https://opensource.googleblog.com/2020/12/finding-critical-o...
defanor 6y ago
I like this idea, which pops up here and there occasionally, but this particular "criticality score" appears to measure popularity, rather than criticality. For one, it doesn't take into account whether there are common alternatives to a given project/package. Of course data like that is hard to gather automatically and from GitHub, but possibly that's just not a great source, and/or could be combined with others -- such as operating systems' repositories, which would contain information not just about alternatives, but also about packages required to run a minimal system, and to build it.
fsflover 6y ago
See also: https://www.fsf.org/campaigns/priority-projects https://www.fsf.org/campaigns/priority-projects.
vlovich123 6y ago
Have you heard of the WWII bullet holes problem that Wald solved? This project seems to fall into the trap that Wald’s colleagues fell into. The more popular a project the more critical it becomes when that’s not actually the case. You want to know where the bullet holes you don’t see are because that’s the thing that’s going to take you out. All the bullet holes you’re seeing and surviving are not great but survivable. Critical to me means “what’s the thing I have to pay attention to or I’ll suffer consequences”. Having a dependency on a non-popular crate is definitely an increasing score of criticality for that dependency. I may have misread but the fatal error in the metric to me is that popularity of a project increases its criticality when it should decrease. A popular project with lots of contributors means it has lots of stake holders already and a way to successfully manage that. What you want is the small independent projects that popular projects depend on. In other words, what’s the smallest, hardest to notice malicious change I can make in the supply chain of software development to enact the most disruptive change?
- rhencke 6y ago
  Your last sentence nails it: > What’s the smallest, hardest to notice malicious change I can make in the supply chain of software development to enact the most disruptive change?
- abharya 6y ago
  This is still an early project to start somewhere. We are working on this problem as well. Trying to identify dependency trees and which smaller projects are the most widely used (indirect deps) and will impact the most on the critical projects in the list.
  - vlovich123 6y ago
    Don’t get me wrong. I appreciate this work and it’s super important if you get it right. I’m worried the first step is drastically in the wrong philosophical direction as far as the metric itself goes. If you’re just proving out the tech then do that but the current publishing of the most “critical” projects is laughably wrong (as I’m sure you’ve heard from the comments on the page). The project isn’t a dead end but any attempt to use the data right now to drive decisions is a lost cause. Maybe you have some secret sauce that makes it work better for internal Google projects but if that’s the case that methodology should be published as well so that people can get an understanding of how Google uses this more effectively. I’m just contrasting this to a paper Google published recently about how to measure uptime more effectively in cloud. Every step of that paper was eminently approachable and didn’t have any obvious philosophical flaws that stood out to me (we can argue about them but any potential arguments were called out). Again, I applaud the attempt. This is a super important problem and trying to tackle it is laudable. Getting the right metrics is crucial and this is being released far too early with a presentation that makes it seem far more complete than it is.
effnorwood 6y ago
wget https://github.com https://github.com | /dev/null
lkcl 6y ago
btw, just one thing, having seen this slashdot comment: https://news.slashdot.org/comments.pl?sid=17816088&cid=60823564 https://news.slashdot.org/comments.pl?sid=17816088&cid=60823... what i strongly recommend that google do - instead of doing this work which is, as you've probably noticed from the comments, well-meaning but highly likely to be biased - is: *make a large donation to NLnet* don't tell them what to do with the money: leave that up to them. NLnet is extremely good at ensuring that money is used effectively, by requiring that they come up with a project plan involving milestones. they do not just "dump money at the developer", which has a known high historically-backed probability of causing more harm than good: they require the milestones to be completed, 100%, before the money is paid. note, here: they do not use algorithms to assess a project: they use people. they also assess the proposal against the usefulness of achieving key objectives.
tkhoven 6y ago
This url links to the top-level blog - for the specific post indicated by the title it needs to be https://opensource.googleblog.com/2020/12/finding-critical-open-source-projects.html https://opensource.googleblog.com/2020/12/finding-critical-o...

137 points by QuantumWasp 6y ago

sova 6y ago

Very interesting and I hope this leads to actual recompense/compensation for people "thanklessly maintaining" OSS. Not sure how it possibly fits into Google's newfound profit-at-all-costs motive, but the gesture is an important first step towards some semblance of justice.

PradeetPatel 6y ago
Can you elaborate more on the claimed "profit-at-all-costs" statement? It's been established that the purpose of most corporations are for profit, but iirc most large enterprises have a rather robust ethics framework when working with the OSS community.
- sova 6y ago
  I wish I had reasonable evidence to point to, but I'm not really an insider. Mainly, I think of the numerous applications Google invested lots of engineer time into to create an experience, say Hangouts, only to realize that it hampered sales in another corner of Google, say cell plans, and so one project must be discarded for profits in the other. The functionality is often difficult to replace for the end-user, and weakens the overall allure of the Google network/ecosystem of applications in favor of a bottom-line.

Svetlitski 6y ago

Sounds interesting in theory, but the results are somewhat bizarre. For example, their published list of the top 200 "most critical" open-source C projects includes Urbit and Stellar, and includes micropython but does not include CPython.

sitzkrieg 6y ago
seems like completely arbitrary garbage
- skinkestek 6y ago
  I.e. worse than ordinary search results?
abharya 6y ago
As per Github (see languages section on bottom right), Cpython is 63% Python, and 29% C code, so its api returns Python as main language. cpython is in top 10 in python_top_200.csv

harpiaharpyja 6y ago

Seems like a good direction. My main hope is that if metrics like these become more accepted and/or important, that they find a way to avoid running afoul of Goodhart's law.

cycloptic 6y ago

Lots of things to check out here, and it's limited only to projects that are on github, but unsurprisingly it looks like the top 5 are: - Node (0.984) - Tensorflow (0.969) - Git (0.945) - Linux (0.936) - PHP (0.919)

thundergolfer 6y ago
Tensorflow ranking higher than Linux is a weird one. Tensorflow debuted to a huge amount of hype, but it could fall over and nearly all our software systems would be totally fine. The rest in that list seem reasonable.

sien 6y ago

List of the top 200 is at : https://commondatastorage.googleapis.com/ossf-criticality-score/index.html https://commondatastorage.googleapis.com/ossf-criticality-sc... (from the repo) So gnucash is #15 . At #75 is gcc . This seems like a great idea, but perhaps some refinements are needed.

abharya 6y ago
World is not perfect. In case of gcc, it is only a mirror on github [https://github.com/gcc-mirror/gcc https://github.com/gcc-mirror/gcc], so we dont get all the metrics, we do plan to improve this part [but this will be slow to clone repo, know their custom issue trackers, etc]. Gnucash is still a popular project, has like ~7k downloads a week, see wikipedia page.
- deleted 6y ago
  [deleted]
- breakingcups 6y ago
  Strangely enough that's not mentioned anywhere on the announcement. In fact, it seems to announce that very real funding decisions are going to be made based on this score already. How can you make accurate decisions based on inaccurate data? Which reminds me of that Babbage quote: "On two occasions I have been asked, 'Pray, Mr. Babbage, if you put into the machine wrong figures, will the right answers come out?' I am not able rightly to apprehend the kind of confusion of ideas that could provoke such a question."
- jcelerier 6y ago
  > Gnucash is still a popular project, has like ~7k downloads a week, see wikipedia page. omg they're actually doubling down on this. Libreoffice has something like 50k downloads per day and isn't listed, while among "critical projects" there is... minetest ? tesseract ? battle for wesnoth ? dolphin ? citra ? There's qt creator (and a ton of Qt-using projects) but not Qt ?
  - abharya 6y ago

Buetol 6y ago

Top 10: - Python: salt, core (https://github.com/home-assistant/core https://github.com/home-assistant/core), pandas, scikit-learn, numpy, airflow, erpnext, matplotlib, pytest & pip - Rust: servo, cargo, rust-clippy, tokio, rust-analyzer, tock, tikv, alacritty, libc & substrate - JS: node, react-native, react, gatsby, three.js, bootstrap, material-ui, odoo, next.js & Rocket.Chat - Java: elasticsearch, flink, spring-boot, hadoop, netty, jenkins, beam, bazel, alluxio & pmd - C++: tensorflow, ceph, pytorch, bitcoin, electron, Marlin, Cataclysm-DDA, llvm-project, rocksdb & QGIS - C: git, linux, linux, php-src, openssl, systemd, curl, u-boot, qemu & mbed-os

EvilEy3 6y ago
In what universe Bazel is a critical project for Java ecosystem?
- lokedhs 6y ago
  The Google universe. Don't they use it for all their builds?
- tziki 6y ago
  In the Android universe?
- leegraham 6y ago
  It looks like it's in the Java section because it's written _in_ Java, not because it's critical _to_ Java. Similarly, php-src isn't critical to C.
- abharya 6y ago
  The list captures projects written in the Java language, not for Java ecosystem. See why it is critical here - https://github.com/bazelbuild/bazel/wiki/Bazel-Users https://github.com/bazelbuild/bazel/wiki/Bazel-Users
megous 6y ago
Which of these match the image in the article, where some random Nebraskan is maintaining some obscure dependency?

hn_throwaway_99 6y ago

As others have mentioned, while this may seem like a good idea, the results are often bizarre, and it's not hard to see why - the metrics and algorithm are here: https://github.com/ossf/criticality_score#criticality-score https://github.com/ossf/criticality_score#criticality-score. That algorithm seems unnecessarily complicated and includes somewhat dubious metrics when, in my mind, the only thing that really "counts" when it comes to "criticality" are "how many other things use/depend on me", and there are much easier ways to determine this: 1. For languages with a common package repo, how many other packages depend on me? With NPM, for example, it's pretty easy to figure out how many other packages depend on a given package. 2. For "top level" projects, look at downloads. My guess is just looking at either (or both) of those metrics would give you better results.

abharya 6y ago
- Other metrics such as how many contributors and organizations are involved, how many user feature requests and bugs getting reported, those are all important project importance and not just "dependency count". some projects can be standalone, so as per your algo, those should be very low. - Downloads data is not available for most repos, please find a reliable metric to use. - Package repo dependencies works, but it does not work for C/C++. At some point, we will integrate github's dependency count info as well.
- detaro 6y ago
  So basically the algorithm is designed to not find projects that are critical because they are deep in the foundations, depended on by nearly everyone but only worked on by a few people? (which is what "critical" would suggest at least to me) This seems to be mostly a "github marketing index"... I seriously hope no actual decisions about resource allocation etc are made based on this.
  - abharya 6y ago
    We are working on this problem, it is not simple. Identifying dependency trees reliably across languages is not straightforward [only nice for package manager ones]. Follow https://github.com/ossf/criticality_score/issues/8 https://github.com/ossf/criticality_score/issues/8

eh78ssxv2f 6y ago

I don't understand how gnucash is more critical than Arduino, wine or nginx. Is gnucash very widely used?

abharya 6y ago
It does seem to be popular w.r.t user downloads and other github metrics. E.g. https://sourceforge.net/projects/gnucash/files/stats/timeline?period=weekly https://sourceforge.net/projects/gnucash/files/stats/timelin... Wikipedia - "As of July 2018, SourceForge shows a count of over 6.3 million downloads of the stable releases starting from November 1999[24] Also, Sourceforge shows that current downloads are running at ~7,000 per week.[25] This does not include other software download sites as well as Linux distributions that provide download from their own repositories."
- schoen 6y ago
  > This does not include other software download sites as well as Linux distributions that provide download from their own repositories. Isn't that a pretty significant difficulty for the interpretation of this number, though? Couldn't there be some package that is installed by default in, say, Ubuntu (and that Ubuntu can't boot without), so therefore millions of users are using it -- but perhaps the "downloads" seen by GitHub or SourceForge are mostly by OS packagers or by developers contributing to that package. Whereas if there is something where, for some reason, end-users customarily or commonly get it from GitHub or Sourceforge, it might look more widely used even though it is actually less used overall, potentially even by orders of magnitude. (Like GNU coreutils, at an extreme, is probably not that popular at all for end users to download from a source control system, yet it might be running in tens or hundreds of millions of devices, which could not even boot without it.)
  - abharya 6y ago
    coreutils is in the top 200 list, c_top_200.csv:coreutils,https://github.com/coreutils/coreutils,C,90,0,240,5,2.2,50,1,6,0.7,1393,0.59927 https://github.com/coreutils/coreutils,C,90,0,240,5,2.2,50,1... There will always be edge cases and scenarios we are not taking into account, please provide feedback on issue tracker and provide any suggestions so we can account these.

gojomo 6y ago

Neither the blog post nor project README directly mentions whether the 0.0 to 1.0 score goes from most-critical to least, or least-critical to more. It appears 1.0 is most-critical, if you dig into ranking lists.

abharya 6y ago
Thanks, fixed in https://github.com/ossf/criticality_score/commit/fddcc093c7e11f097a447b057976391e1860ae85 https://github.com/ossf/criticality_score/commit/fddcc093c7e...

krzyk 6y ago

Strange list, PMD so high, quarkus also, but e.g. jackson is lower then those both. elasticsearch at the top? flink? It is first time I hear about it and I work in Java shops for quite some time.

watwut 6y ago
I mean, our company uses elasticsearch. But like, if it went away, it would not be critical or something. Some features would degrade a bit and no one would cared. But yeah, the list is bizarre. It is more of "list of cool things to read about that you dont really need" then "critical list".

cryptica 6y ago

There are some issues with the criticality score (https://github.com/ossf/criticality_score https://github.com/ossf/criticality_score). For example, it counts the number of issues as determining criticality. If there is any kind of money attached to this score (now or in the future), obviously this is going to encourage people to introduce more bugs into their projects.

rwmj 6y ago

It's nice that they're automating this even though the results seem currently a bit random. Fedora has a concept of Critical Path packages (https://fedoraproject.org/wiki/Critical_path_package https://fedoraproject.org/wiki/Critical_path_package). The concept is related to various activities, eg. a package is critical if is is needed when installing a graphical desktop. But as far as I know packages are picked manually and therefore we probably miss packages or over/underestimate criticality.

abharya 6y ago
Thanks for feedback, adding it for analysis in https://github.com/ossf/criticality_score/issues/23 https://github.com/ossf/criticality_score/issues/23

commoner 6y ago

The link is to the home page of the Google Open Source Blog, and not this specific blog post. It should be updated to: https://opensource.googleblog.com/2020/12/finding-critical-open-source-projects.html https://opensource.googleblog.com/2020/12/finding-critical-o...

defanor 6y ago

I like this idea, which pops up here and there occasionally, but this particular "criticality score" appears to measure popularity, rather than criticality. For one, it doesn't take into account whether there are common alternatives to a given project/package. Of course data like that is hard to gather automatically and from GitHub, but possibly that's just not a great source, and/or could be combined with others -- such as operating systems' repositories, which would contain information not just about alternatives, but also about packages required to run a minimal system, and to build it.

fsflover 6y ago

See also: https://www.fsf.org/campaigns/priority-projects https://www.fsf.org/campaigns/priority-projects.

vlovich123 6y ago

Have you heard of the WWII bullet holes problem that Wald solved? This project seems to fall into the trap that Wald’s colleagues fell into. The more popular a project the more critical it becomes when that’s not actually the case. You want to know where the bullet holes you don’t see are because that’s the thing that’s going to take you out. All the bullet holes you’re seeing and surviving are not great but survivable. Critical to me means “what’s the thing I have to pay attention to or I’ll suffer consequences”. Having a dependency on a non-popular crate is definitely an increasing score of criticality for that dependency. I may have misread but the fatal error in the metric to me is that popularity of a project increases its criticality when it should decrease. A popular project with lots of contributors means it has lots of stake holders already and a way to successfully manage that. What you want is the small independent projects that popular projects depend on. In other words, what’s the smallest, hardest to notice malicious change I can make in the supply chain of software development to enact the most disruptive change?

rhencke 6y ago
Your last sentence nails it: > What’s the smallest, hardest to notice malicious change I can make in the supply chain of software development to enact the most disruptive change?
abharya 6y ago
This is still an early project to start somewhere. We are working on this problem as well. Trying to identify dependency trees and which smaller projects are the most widely used (indirect deps) and will impact the most on the critical projects in the list.
- vlovich123 6y ago
  Don’t get me wrong. I appreciate this work and it’s super important if you get it right. I’m worried the first step is drastically in the wrong philosophical direction as far as the metric itself goes. If you’re just proving out the tech then do that but the current publishing of the most “critical” projects is laughably wrong (as I’m sure you’ve heard from the comments on the page). The project isn’t a dead end but any attempt to use the data right now to drive decisions is a lost cause. Maybe you have some secret sauce that makes it work better for internal Google projects but if that’s the case that methodology should be published as well so that people can get an understanding of how Google uses this more effectively. I’m just contrasting this to a paper Google published recently about how to measure uptime more effectively in cloud. Every step of that paper was eminently approachable and didn’t have any obvious philosophical flaws that stood out to me (we can argue about them but any potential arguments were called out). Again, I applaud the attempt. This is a super important problem and trying to tackle it is laudable. Getting the right metrics is crucial and this is being released far too early with a presentation that makes it seem far more complete than it is.

effnorwood 6y ago

wget https://github.com https://github.com | /dev/null

lkcl 6y ago

btw, just one thing, having seen this slashdot comment: https://news.slashdot.org/comments.pl?sid=17816088&cid=60823564 https://news.slashdot.org/comments.pl?sid=17816088&cid=60823... what i strongly recommend that google do - instead of doing this work which is, as you've probably noticed from the comments, well-meaning but highly likely to be biased - is: *make a large donation to NLnet* don't tell them what to do with the money: leave that up to them. NLnet is extremely good at ensuring that money is used effectively, by requiring that they come up with a project plan involving milestones. they do not just "dump money at the developer", which has a known high historically-backed probability of causing more harm than good: they require the milestones to be completed, 100%, before the money is paid. note, here: they do not use algorithms to assess a project: they use people. they also assess the proposal against the usefulness of achieving key objectives.

tkhoven 6y ago

This url links to the top-level blog - for the specific post indicated by the title it needs to be https://opensource.googleblog.com/2020/12/finding-critical-open-source-projects.html https://opensource.googleblog.com/2020/12/finding-critical-o...