5 ms·

Maybe I misunderstood something, but it seems to me their data demonstrate that less interesting papers that nobody bothered to download from SciHub tend to hav

by oefrha 6y ago

Maybe I misunderstood something, but it seems to me their data demonstrate that less interesting papers that nobody bothered to download from SciHub tend to have fewer citations than more interesting ones that are downloaded? How does that support the conclusion “Sci-Hub downloads lead to more article citations” (emphasis mine)? (Yes I read the paper and I’ve seen all the other factors they considered, but those factors simply aren’t as good predictors/indicators of interest as an organic download number from any channel, whether it’s ScienceDirect or SciHub.) Edit: To be clear, papers in their supposedly "limited access" (if we want to draw the same conclusion) control group appear to be available on Sci-Hub, i.e. as open access as their Sci-Hub group. Details in this comment: https://news.ycombinator.com/item?id=23710992 https://news.ycombinator.com/item?id=23710992

Noluris 6y ago
I think the title they chose is just misleading (wrong?). Otherwise, their conclusion is > we found that articles downloaded from Sci-hub were cited 1.72 times more than papers not downloaded from Sci-hub and that the number of downloads from Sci-hub was a robust predictor of future citations > The results suggest that limited access to publications may limit some scientific research from achieving its full impact.
- oefrha 6y ago
  It’s more than just the title. The conclusion from abstract reads “The results suggest that limited access to publications may limit some scientific research from achieving its full impact.” My uncharitable read is they slapped an interesting yet unsupported conclusion onto a rather boring observation. Edit: In particular, it seems to me that papers in their control group are available on SciHub (correct me if I’m wrong), just nobody bothered to download them in the time window they analyzed. Which directly contradicts the “limited access” part of their claim.
  - jaekash 6y ago
    > Edit: In particular, it seems to me that papers in their control group are available on SciHub (correct me if I’m wrong), just nobody bothered to download them in the time window they analyzed. Which directly contradicts the “limited access” part of their claim. I was not sure about this, but if this is the case then their conclusion seems to be unrelated to their findings.
    - oefrha 6y ago
      I'm pretty sure about this now. I just looked into the actual data set[1], and analyzed a few papers from it, e.g. [2] the first paper in scopus_nature.csv. They're definitely available on Sci-Hub at least at the moment [3], and there's no reason to believe they weren't available back then. Also, here's what they say about the dataset, emphasis mine: > As a quality control, we performed a random sampling of all the articles retrieved, excluding those already present in the first data set. As in this second data set, the number of Sci-hub downloads is precisely equal to zero, we regard it as a control group (nC = 4,015) from which we are going to estimate comparisons for our experimental group (nE = 4,646). Downloads equal to zero != not available. [1] https://osf.io/xb9yn/ https://osf.io/xb9yn/ [2] https://www.scopus.com/inward/record.url?eid=2-s2.0-85016141654&partnerID=40&md5=387eaeb70ac807a4fa4acb573e42b03a https://www.scopus.com/inward/record.url?eid=2-s2.0-85016141... [3] https://sci-hub.tw/10.1038/541123a https://sci-hub.tw/10.1038/541123a
      - amatic 6y ago
        > Downloads equal to zero != not available. I'm wondering about this too. What could be the possible other reason for the papers not being downloaded? Boring titles? Preprints easily available? Open access? > and there's no reason to believe they weren't available back then. Is it possible that the articles really were not available, as they used the data from sep-2015 to feb-2016, early version of SciHub?
        oefrha 6y ago
        > Is it possible that the articles really were not available, as they used the data from sep-2015 to feb-2016, early version of SciHub? That's within the realm of possibilities but sure doesn't sound like the case the way they described their methodology. That would have required a catalog of available papers on Sci-Hub too, but the dataset they used only contained download logs, so they couldn't have known whether a zero-download paper was available or not (unless I misunderstood what the first dataset is about).
- barbegal 6y ago
  I thought that too. But in the "Statistical analyses" section it says: "the interpretation of our robust estimates goes beyond claiming mere correlations or associations." which when you read through the statistical analysis is clearly not true.
- mistaken 6y ago
  I think what they're hinting at is that without SciHub, even a popular paper would have less exposure. Some papers which cite a popular publication can only do so, because they could download the referenced one through SciHub.
  - subroutine 6y ago
    I don't think anyone questions that some articles have been cited because they were downloaded from scihub. This was still a garbage attempt to 'prove' this conclusion. And I don't have any faith the authors measured the true magnitude of these effects.
deleted 6y ago
[deleted]
subroutine 6y ago
Yes this is a huge confound. Their methods state... The first data set contained all articles (nE = 4,646) that were downloaded from Sci-hub between September 2015 and February 2016 [24, 25]. The second data set was extracted from the Scopus database and contained all articles published in the selected journals within the same time period. From this data set, we excluded articles already present in the first data set. Their methods lend to circular reasoning since the conclusion could also be "papers of high interest (cited more often) are more likely to be downloaded from scihub"
chiefalchemist 6y ago
Same here. Simply put, this is correlation. The irony humorous.
Chris2048 6y ago
So basically, correlation, speculated (w/o basis) as causation?
pen2l 6y ago
As someone from a research lab from a big name school that already has access to every journal: we use sci-hub all the time just because it is so damn convenient (me, I have a chrome addon wherein I click a button on a page that has a doi link, and wahlah, I'm looking at the pdf. Firewall at work blocks scihub, but I find ways around it). As to why my lab might be preferring papers from big-name journal: I think 1) there's admittedly our own bias of wanting to cite big papers vs. small papers, 2) if we're doing some basic literature search, Google will also yield results that are already more popular. Really, at the moment, I struggle to see how our behavior changes about which papers we're citing with or without scihub.
- neutronicus 6y ago
  I think without Sci-hub I might have just said "fuck it" and never read certain papers from smaller journals that I later wound up citing One thing I would do a lot with Google Scholar is find a highly-cited, somewhat recent (past decade or so) paper and then click into its citations to find the very most recent related work. IIRC this interface was chronological so there was no big-journal bias and I found a lot of stuff in random journals. Low-friction quality-filtering through SciHub was very helpful for me in this process. I worked in a theoretical field, so it's actually possible to assess the quality of a paper just by reading it. If you're doing something experimental you may have to lean on the big-journal filter a little bit more.
- sitkack 6y ago
  > I struggle to see how our behavior changes about which papers we're citing with or without scihub. Under what conditions do you see that changing? You yourself say how useful it is, but discount the effect it has on the citation graph. Could we say the samething about arxiv? This is exactly why research needs to be open and freely available. The esoteric, rarely cited papers are locked behind paywalls and away from search engines. And in doing so, their impact fades over time as the flock centers around a single vein of focus. The future should look like a net not a river. I am not an academic and I read papers for their metalessons in areas that interest me. But without Sci-Hub, lots of research would not be accessible outside of its niche, esp if it was unlucky enough to not be in the right journal at the right time. On hilarious pattern I see, is when a bigshot in a field publishes a new paper, there is a race to cite it by some nobody. Basically the academic publishing version of "first post".
dekhn 6y ago
I find in nearly all cases, if you invert the causality and the results make more sense, you can ignore a paper. "Papers with more citations are downloaded more frequently on Sci-Hub" makes more sense to me than "Papers downloaded frequently from Sci-Hub get more citations".
credit_guy 6y ago
I guess what you are saying is that correlation does not imply causation, and in this case the causality arrow may go the other way round. In this case, there's something that actually can help establish the direction of the causality arrow: time. If the citations precede the downloads, then sure, it's more well known papers get downloaded more. This papers show that the downloads precede citations though: "number of downloads from Sci-hub was a robust predictor of future citations" While it's not a watertight argument that downloads lead to citations, the theory is at least plausible.

by oefrha 6y ago

Noluris 6y ago

I think the title they chose is just misleading (wrong?). Otherwise, their conclusion is > we found that articles downloaded from Sci-hub were cited 1.72 times more than papers not downloaded from Sci-hub and that the number of downloads from Sci-hub was a robust predictor of future citations > The results suggest that limited access to publications may limit some scientific research from achieving its full impact.

oefrha 6y ago
It’s more than just the title. The conclusion from abstract reads “The results suggest that limited access to publications may limit some scientific research from achieving its full impact.” My uncharitable read is they slapped an interesting yet unsupported conclusion onto a rather boring observation. Edit: In particular, it seems to me that papers in their control group are available on SciHub (correct me if I’m wrong), just nobody bothered to download them in the time window they analyzed. Which directly contradicts the “limited access” part of their claim.
- jaekash 6y ago
  > Edit: In particular, it seems to me that papers in their control group are available on SciHub (correct me if I’m wrong), just nobody bothered to download them in the time window they analyzed. Which directly contradicts the “limited access” part of their claim. I was not sure about this, but if this is the case then their conclusion seems to be unrelated to their findings.
  - oefrha 6y ago
    I'm pretty sure about this now. I just looked into the actual data set[1], and analyzed a few papers from it, e.g. [2] the first paper in scopus_nature.csv. They're definitely available on Sci-Hub at least at the moment [3], and there's no reason to believe they weren't available back then. Also, here's what they say about the dataset, emphasis mine: > As a quality control, we performed a random sampling of all the articles retrieved, excluding those already present in the first data set. As in this second data set, the number of Sci-hub downloads is precisely equal to zero, we regard it as a control group (nC = 4,015) from which we are going to estimate comparisons for our experimental group (nE = 4,646). Downloads equal to zero != not available. [1] https://osf.io/xb9yn/ https://osf.io/xb9yn/ [2] https://www.scopus.com/inward/record.url?eid=2-s2.0-85016141654&partnerID=40&md5=387eaeb70ac807a4fa4acb573e42b03a https://www.scopus.com/inward/record.url?eid=2-s2.0-85016141... [3] https://sci-hub.tw/10.1038/541123a https://sci-hub.tw/10.1038/541123a
    - amatic 6y ago
      > Downloads equal to zero != not available. I'm wondering about this too. What could be the possible other reason for the papers not being downloaded? Boring titles? Preprints easily available? Open access? > and there's no reason to believe they weren't available back then. Is it possible that the articles really were not available, as they used the data from sep-2015 to feb-2016, early version of SciHub?
      - oefrha 6y ago
        > Is it possible that the articles really were not available, as they used the data from sep-2015 to feb-2016, early version of SciHub? That's within the realm of possibilities but sure doesn't sound like the case the way they described their methodology. That would have required a catalog of available papers on Sci-Hub too, but the dataset they used only contained download logs, so they couldn't have known whether a zero-download paper was available or not (unless I misunderstood what the first dataset is about).
barbegal 6y ago
I thought that too. But in the "Statistical analyses" section it says: "the interpretation of our robust estimates goes beyond claiming mere correlations or associations." which when you read through the statistical analysis is clearly not true.
mistaken 6y ago
I think what they're hinting at is that without SciHub, even a popular paper would have less exposure. Some papers which cite a popular publication can only do so, because they could download the referenced one through SciHub.
- subroutine 6y ago
  I don't think anyone questions that some articles have been cited because they were downloaded from scihub. This was still a garbage attempt to 'prove' this conclusion. And I don't have any faith the authors measured the true magnitude of these effects.

deleted 6y ago

[deleted]

subroutine 6y ago

Yes this is a huge confound. Their methods state... The first data set contained all articles (nE = 4,646) that were downloaded from Sci-hub between September 2015 and February 2016 [24, 25]. The second data set was extracted from the Scopus database and contained all articles published in the selected journals within the same time period. From this data set, we excluded articles already present in the first data set. Their methods lend to circular reasoning since the conclusion could also be "papers of high interest (cited more often) are more likely to be downloaded from scihub"

chiefalchemist 6y ago

Same here. Simply put, this is correlation. The irony humorous.

Chris2048 6y ago

So basically, correlation, speculated (w/o basis) as causation?

pen2l 6y ago

As someone from a research lab from a big name school that already has access to every journal: we use sci-hub all the time just because it is so damn convenient (me, I have a chrome addon wherein I click a button on a page that has a doi link, and wahlah, I'm looking at the pdf. Firewall at work blocks scihub, but I find ways around it). As to why my lab might be preferring papers from big-name journal: I think 1) there's admittedly our own bias of wanting to cite big papers vs. small papers, 2) if we're doing some basic literature search, Google will also yield results that are already more popular. Really, at the moment, I struggle to see how our behavior changes about which papers we're citing with or without scihub.

neutronicus 6y ago
I think without Sci-hub I might have just said "fuck it" and never read certain papers from smaller journals that I later wound up citing One thing I would do a lot with Google Scholar is find a highly-cited, somewhat recent (past decade or so) paper and then click into its citations to find the very most recent related work. IIRC this interface was chronological so there was no big-journal bias and I found a lot of stuff in random journals. Low-friction quality-filtering through SciHub was very helpful for me in this process. I worked in a theoretical field, so it's actually possible to assess the quality of a paper just by reading it. If you're doing something experimental you may have to lean on the big-journal filter a little bit more.
sitkack 6y ago
> I struggle to see how our behavior changes about which papers we're citing with or without scihub. Under what conditions do you see that changing? You yourself say how useful it is, but discount the effect it has on the citation graph. Could we say the samething about arxiv? This is exactly why research needs to be open and freely available. The esoteric, rarely cited papers are locked behind paywalls and away from search engines. And in doing so, their impact fades over time as the flock centers around a single vein of focus. The future should look like a net not a river. I am not an academic and I read papers for their metalessons in areas that interest me. But without Sci-Hub, lots of research would not be accessible outside of its niche, esp if it was unlucky enough to not be in the right journal at the right time. On hilarious pattern I see, is when a bigshot in a field publishes a new paper, there is a race to cite it by some nobody. Basically the academic publishing version of "first post".

dekhn 6y ago

I find in nearly all cases, if you invert the causality and the results make more sense, you can ignore a paper. "Papers with more citations are downloaded more frequently on Sci-Hub" makes more sense to me than "Papers downloaded frequently from Sci-Hub get more citations".

credit_guy 6y ago

I guess what you are saying is that correlation does not imply causation, and in this case the causality arrow may go the other way round. In this case, there's something that actually can help establish the direction of the causality arrow: time. If the citations precede the downloads, then sure, it's more well known papers get downloaded more. This papers show that the downloads precede citations though: "number of downloads from Sci-hub was a robust predictor of future citations" While it's not a watertight argument that downloads lead to citations, the theory is at least plausible.