4 ms·

I'm getting wildly different citation counts for some of the listed AI papers. For example, the paper "ColabFold: making protein folding accessible to all" is

I'm getting wildly different citation counts for some of the listed AI papers. For example, the paper "ColabFold: making protein folding accessible to all" is listed as having 1162 citations. I'm seeing that it was cited only by 899 publications on Scite: https://scite.ai/reports/colabfold-making-protein-folding-accessible-PQ3vGn3b https://scite.ai/reports/colabfold-making-protein-folding-ac... I'm wondering if Google Scholar is overestimating or Scite is underestimating.

zavrel 4y ago
We used the Google Scholar counts - and in particular a snapshot of end of february for this ranking. There is no perfect number, but at least this one is generally accepted as reasonable, and public, so easy to verify for everyone.
jltsiren 4y ago
Citation counts are always a bit arbitrary. Google Scholar usually overestimates, because it's basically a bunch of heuristics. Curated citation databases underestimate in the name of consistency. For example, they may ignore citations in conference proceedings, as conference papers are not considered legitimate publications in most fields.
godelski 4y ago
Semantic Scholar has 1,111. [0] I tend to trust Semantic more than GS. GS tends to overestimate. For example on GS I have 164 citations on one paper and semantic says 150. FWIW Scite says 49.[1] [0] https://www.semanticscholar.org/paper/ColabFold%3A-making-protein-folding-accessible-to-all-Mirdita-Ovchinnikov/0da93a948211b63c462582d7e5dbf8da9cdfbcf1 https://www.semanticscholar.org/paper/ColabFold%3A-making-pr... [1] I'll note that this paper is an arxiv paper and has not been accepted at a conference but I'd also argue that conference acceptance means little in ML. I'll explain if anyone is actually concerned with the claim.
- dr_dshiv 4y ago
  Please explain!
  - godelski 4y ago
    So there's a few things to consider: First, this peer review via conferences/journals/etc is relatively new in the scientific process. Really only the last 50 years has this paradigm been the main way for publishing. Prior to that scientists have just published in the open and and peer review happened by peers reading and responding. Not too different from what we see with arxiv, twitter, and blogging. Second, we need to talk about how good the review process actually is. There's been a lot of writing on the NeurIPS experiments [0] is the most famous one. But the Google paper[1] notes that reviewers are "good at identifying bad papers but not good at identifying good papers." I'll go a step further than them and suggest a plausible model that makes this statement true: reviewers are reject happy. We need a confusion matrix to really see this but if you reject every paper you'd have a 100% success rate of rejecting bad papers but a 0% success rate of approving good papers. We have a good demonstration that ML conferences (journals aren't our priority like other academic areas, conferences are. This is an oddity) are an extremely noisy process and not very meaningful. So how do we capture a signal in this noisy process? Citations are at least some signal. Obviously this isn't a fantastic signal either because big labs and companies are going to be able to popularize their work more and this will get more citations. But this still isn't any worse than we were 100 years ago. I'd argue that the noisy process of conferencing is worse than where we were 100 years ago (democratization of science aside). Unfortunately, the only way to identify if a paper is good is to have experts evaluate them. I don't think we have a good alternative for this and adding significantly noisy signals aren't helpful. [0] https://blog.mrtz.org/2014/12/15/the-nips-experiment.html https://blog.mrtz.org/2014/12/15/the-nips-experiment.html [1] https://arxiv.org/abs/2109.09774 https://arxiv.org/abs/2109.09774
  - fxtentacle 4y ago
    Facebook wav2vec2, for example, was also just uploaded to arxiv and then they formally published it months/years later after it was widely used. The traditional publication flow just isn't useful if a runnable demo on HuggingFace explains your work way better than 3 pages of formulas.

zavrel 4y ago

We used the Google Scholar counts - and in particular a snapshot of end of february for this ranking. There is no perfect number, but at least this one is generally accepted as reasonable, and public, so easy to verify for everyone.

jltsiren 4y ago

Citation counts are always a bit arbitrary. Google Scholar usually overestimates, because it's basically a bunch of heuristics. Curated citation databases underestimate in the name of consistency. For example, they may ignore citations in conference proceedings, as conference papers are not considered legitimate publications in most fields.

godelski 4y ago

Semantic Scholar has 1,111. [0] I tend to trust Semantic more than GS. GS tends to overestimate. For example on GS I have 164 citations on one paper and semantic says 150. FWIW Scite says 49.[1] [0] https://www.semanticscholar.org/paper/ColabFold%3A-making-protein-folding-accessible-to-all-Mirdita-Ovchinnikov/0da93a948211b63c462582d7e5dbf8da9cdfbcf1 https://www.semanticscholar.org/paper/ColabFold%3A-making-pr... [1] I'll note that this paper is an arxiv paper and has not been accepted at a conference but I'd also argue that conference acceptance means little in ML. I'll explain if anyone is actually concerned with the claim.

dr_dshiv 4y ago
Please explain!
- godelski 4y ago
  So there's a few things to consider: First, this peer review via conferences/journals/etc is relatively new in the scientific process. Really only the last 50 years has this paradigm been the main way for publishing. Prior to that scientists have just published in the open and and peer review happened by peers reading and responding. Not too different from what we see with arxiv, twitter, and blogging. Second, we need to talk about how good the review process actually is. There's been a lot of writing on the NeurIPS experiments [0] is the most famous one. But the Google paper[1] notes that reviewers are "good at identifying bad papers but not good at identifying good papers." I'll go a step further than them and suggest a plausible model that makes this statement true: reviewers are reject happy. We need a confusion matrix to really see this but if you reject every paper you'd have a 100% success rate of rejecting bad papers but a 0% success rate of approving good papers. We have a good demonstration that ML conferences (journals aren't our priority like other academic areas, conferences are. This is an oddity) are an extremely noisy process and not very meaningful. So how do we capture a signal in this noisy process? Citations are at least some signal. Obviously this isn't a fantastic signal either because big labs and companies are going to be able to popularize their work more and this will get more citations. But this still isn't any worse than we were 100 years ago. I'd argue that the noisy process of conferencing is worse than where we were 100 years ago (democratization of science aside). Unfortunately, the only way to identify if a paper is good is to have experts evaluate them. I don't think we have a good alternative for this and adding significantly noisy signals aren't helpful. [0] https://blog.mrtz.org/2014/12/15/the-nips-experiment.html https://blog.mrtz.org/2014/12/15/the-nips-experiment.html [1] https://arxiv.org/abs/2109.09774 https://arxiv.org/abs/2109.09774
- fxtentacle 4y ago
  Facebook wav2vec2, for example, was also just uploaded to arxiv and then they formally published it months/years later after it was widely used. The traditional publication flow just isn't useful if a runnable demo on HuggingFace explains your work way better than 3 pages of formulas.