> Do you have any suggestions perhaps? No, I don't. Our funding sources really wanted the number of users. The number of downloads was a reasonable proxy since

> Do you have any suggestions perhaps? No, I don't. Our funding sources really wanted the number of users. The number of downloads was a reasonable proxy since at the time each organization hosted its own downloads, so they were roughly comparable to each other. Over time it has became harder, and I don't know what the funding sources now want. There are no good measure of software quality. Even the number of users isn't useful except perhaps by comparing to alternatives in the same class. (The number of users of semi-empirical quantum mechanics software is certainly less than the number of users of BLAST, even when the QM software has been in heavy development for over 40 years.) My underlying point is that the text doesn't explain how the solution - a hub - solves the given use case any better than the current system. Readers have to accept your belief. There is, by the way, a long history of code hubs, at least in the form of aggregate archives. In my field, the oldest is the CCL archive, at http://ccl.net/chemistry/resources/software/index.shtml http://ccl.net/chemistry/resources/software/index.shtml . You can see from the dates that 1) it's old, and 2) few use it any more. There are also previous attempts at crowd-sourced software reviews. Here's one for molecular visualization software: http://molvis.sdsc.edu/visres/ http://molvis.sdsc.edu/visres/ . (The author uses the term 'Visitor-Maintained Indices' because it predates 'crowdsource'.) Why didn't similar earlier systems succeed? Was it simply poor metrics? Lack of automation? Or something else? How will it handle the churn of increasingly updated software and outdated information? > As long as you can read and parse the dependency tree I think what you are saying is that if a project references another project as an external dependency, then that can be captured and analyzed. However, in the case of VMD, STRIDE and SURF are included as part of the distribution. There is no dependency tree. This is quite common. Some of the other code I write include third-party packages as a submodule, in order to simplify installation, or because it requires a few tweaks. (Consider, for example, when the original developer has graduated and there is no replacement maintainer.) I've seen others incorporate my code the same way. > This happens more often than you would think and is something that regular citation statistics just don't cover, and the hub can tackle easily. Actually, I don't think you explained how the hub can tackle that problem. The hub might have more data, but that data might not be useful. How might a review of the hub data help the department head make a decision? > Perhaps that is true Really? My experience is that munging different formats is incredibly common, and unpublishable. I was asked just last week if I was interested in writing a program to call another program in order to explore parameter space. This is a call via the subprocess module, wrapped in a few loops, and some parser code to get the ouput from each run and put a summary in an output file. Not publishable. And they are willing to pay $1,000 for it. (I said I would rather teach them how to do it.) > what I saw from experience ... is that the software that is cited in papers is rarely generic Of course not. No one cites Microsoft Windows, or the Linux kernel, or the GNU tools. My statement was a different one. Most science software development goes completely unpublished, with neither a repository nor a published paper. My analogy is to the glass blower. Once upon a time all chemists blew their own glass, and a department would have its own glass shop. There could even be a specialist for making the more advanced glassware. Rather like how most researchers do some programming, but there are also research programmers. We see these in the names, like the Erlenmeyer flask, Schlenk tube, and Dewar flask. But most of the glassware is quite boring - enough that these days it's mostly outsourced and mass-produced. Most of the people who produce them, just like most other lab technicians, don't have their names in scientific publications. Most software is in the boring category of making a variation of something well known. A slightly different type of Schlenk tube, for example. But let's get back to the scenario, where someone distributes a world-class algorithm X. How does the department head judge the effectiveness? You said citations weren't sufficient, because they aren't always cited. However, presumably the same is true for every other program in the field, with about the same error rate. So the department head should be able to compare citations with similar programs and establish a ranking. Which, I'll note, is exactly what you suggest with 'Searching for FTOOL's citation on Google Scholar'. So the hub doesn't provide anything that couldn't be done by a meta-tool on top of Google Scholar, and so citations are sufficient, no?