5 ms·

The econometric term for what this paper lacks is "identification strategy" -- a way for assessing the impact of x (downloads on sci-hub) on y (citations) which

The econometric term for what this paper lacks is "identification strategy" -- a way for assessing the impact of x (downloads on sci-hub) on y (citations) which _identifies_ a source of variation in x that is independent of y. A randomized controlled trial would be ideal. Second-best would be something that, say, suddenly caused some papers to be available or not available on Sci-Hub, or a "plausibly exogenous" (to y) cutoff around which papers would be either available on Sci-Hub or not. This paper offers nothing. I think we can say the relationship is "not identified." Doesn't mean the conclusion is wrong, just that this paper doesn't produce meaningful evidence on the question. For more see Angrist and Krueger (2001) https://economics.mit.edu/files/18 https://economics.mit.edu/files/18 (In Pearl/DAG language, the underlying quality or appeal of the paper is, as others have noted, an obvious confound/backdoor path).

barmstrong 6y ago
This is a great summary of how to show causal effects. Thx for pointing it out. I feel like we need a new way to do peer review, that is more real time - so that papers can be upvoted/downvoted, flaws can be pointed out - and we have some way to assess the truthiness of what the paper is claiming. Your comment is a step in this direction (but we're not capturing the wisdom of the crowds quantitatively around papers today - arxiv is great, but 1990's era web design). I'm working with a team that is trying to build better tools for science at https://www.researchhub.com/about https://www.researchhub.com/about Rethinking peer review is one item on the roadmap.
- thomasahle 6y ago
  > we need a new way to do peer review ... so that papers can be upvoted/downvoted This one managed to get 700 upvotes on HN and still be meaningless.
  - Melting_Harps 6y ago
    > I feel like we need a new way to do peer review, that is more real time - so that papers can be upvoted/downvoted, flaws can be pointed out - and we have some way to assess the truthiness of what the paper is claiming. Your comment is a step in this direction (but we're not capturing the wisdom of the crowds quantitatively around papers today - arxiv is great, but 1990's era web design). > This one managed to get 700 upvotes on HN and still be meaningless. Agreed. This is why I don't think voting with online peramters, like down voting, will ever be viable choice; because aside from things like Cybil attacks, the reality is that Social Media has normalized and acutely optimized group think and the tribalism that often follow. For those of that don't engage in it its incredibly alarming and disconcerting how many succumb to it's practices in real Life. What I will propose as an alternative is something that was first tried with Andreas Antonpolus' first book 'Mastering Bitcoin' wherein the chapters are each individually uploaded to Github and follows the same process that OSS does, wherein commits and corrections are submitted to alter the books ultimate maintained version, and never really being fully 'finished' and can be amended and annotated as needed to suit the updates that follow (eg: Segwit or Lightening Network) or in this case perhaps a replicated experiment that provides a larger sample size. Replication in Academic Papers is nearly non-existant, despite the notion that Peer Reviewed (especially in STEM) was to be a the critical component that made it invaluable. This could also effectively undo the walled-garden-extrotionist model that afflicts Academia's Peer Reviewed model and foster more interactive and International work without being present in a University. Samples or specimens may need to be more tightly controlled and transported, but this could effectively be done over night if the Will exists for little to no money in anything other than training. If the International University model is undergoing mass disruption and shifting toward a mainly online learning platform, this too could help mitigate these glaring problems and have a Global Repository for ongoing research. Its almost stupid not to do it at this point given the MANY pitfalls that its current model forces down our throats.
    - thomasahle 6y ago
      > commits and corrections are submitted to alter the books I think this is good idea fro books, and I wish more scholars would contribute to Wikipedia and other peer editable overviews. I'm not sure quite how it would work for new results though, for which it is still not quite clear how they fit into the current knowledge, or if they are worthwhile at all. People also tend to feel really strongly about their own original ideas, and may try to push them forward when they should rather be forgotten. I don't know how to prevent something like that, other than having a BDFL or individual publishing (as in the current system)
      - hunter-gatherer 6y ago
        Gitlab has an article on their website somewhere where they address some of your concerns. I'm on mobile now and can't find it, but the article advocates for a 'Handbook first' approach to documentation. We did this at my last job and I found it well worth the effort. The BDFL ended up being the quality of our work. In other words, the most correct version always wins out.
    - chrismatei 6y ago
      I agree with this. I'm currently doing research, and one factor that always makes go nuts was the actual time to go and find an article and then donwloadit. I can say that I have used or rather abused sci-hub so that I can find related papers. Having something like "github" style + the option to visually view the changes updates, what are the connection, who did what, where is coming fro, something in the way the site conectedpapers does is also very powerfull. Lastly, the only concern in here it is that if it's ever leving document how ca you asses as whole one document if its good quality at certain given time ? So my proposal wil follow under "feature request" type where even if the changes are made, having the ability to read the "metadata" or quality check done by lets say AI(plagarims, etc, etc,) + with the feedback from online reviewer = qood quality check.
  - on_and_off 6y ago
    Not to mention that it also needs to account for science advancing one funeral at a time. How to distinguish between real flaws or just a good theory that is not yet taken seriously ?
  - setgree 6y ago
    > This one managed to get 700 upvotes on HN and still be meaningless. On the other hand, the top/most salient comments on it are all pretty critical of its causal claims.
- setgree 6y ago
  I just uploaded the papers I’ve co-authored — all were licensed CC 4.0 — to Research Hub and emailed to say hello. Looks like a great platform!
jcorrean 6y ago
Thank you so much for this feedback. My team and I will be more than happy to discuss these ideas, if you want to partner up...
- setgree 6y ago
  Hi Dr. Correa, Thanks for writing. I've taken a quick look at your paper and its OSF page, and you all clearly know what you're doing and have some great work here. I am not your target audience (I am not an academic), but if I were peer reviewing this, I'd suggest one of two avenues. 1) As Angrist and Krueger put it, "In our view, good instruments often come from detailed knowledge of the economic mechanism and institutions determining the regressor of interest...progress comes from detailed institutional knowledge and the careful investigation and quantification of the forces at work in a particular setting." With this in mind, is it possible to contact the maintainer of Sci-Hub, email her your paper, and ask if she can think of any plausibly exogenous sources of variation in papers' availability on Sci-Hub? Some possibilities that come to mind for me: * In my own experience, papers without DOIs are often harder to find on Sci-Hub. Are there closed-access journals that don't mint DOIs on which you could use propensity score matching to create something of an apples-to-apples comparison? * Is there an identification strategy somewhere in journals with optional APCs for which only some of the papers are open access? * Were there ever any sci-hub outages that lasted periods of many months, and, however long down the line, did the citations of paywalled articles decline relative to their open access peers? 2) If none of this is possible, I suggest removing all causal language from your article and sticking to a predictive model. Observational research is appropriate for exploring and identifying causal hypotheses rather than confirming them; no statistical techniques that attempt to control for unobserved population heterogeneity will persuade me otherwise (I am admittedly an extremist on this position, but my own views hew closely to those of Gerber, Green and Kaplan (2003): http://www.donaldgreen.com/wp-content/uploads/2015/09/Gerber-Green-Kaplan-IllusionofLearning.pdf http://www.donaldgreen.com/wp-content/uploads/2015/09/Gerber...). The Lewbel (2012) paper you cite looks interesting but, to paraphrase Gerber et al., statistical techniques can't account for nonstatistical sources of uncertainty (the inherit unnkowability of whether you've specified the 'correct' model, in the absence of randomization or exogenous variation, means we can't say anything about the biasedness of your estimation procedure). Observational research is great! Generating hypotheses is as important as confirming them. It's just that some research designs license causal inference, and some do not. Best of luck.
setgree 6y ago
Just for the record my definition of an identification strategy is not quite right -- it should be > a way for assessing the impact of x (downloads on sci-hub) on y (citations) which _identifies_ a source of variation in x, that is independent of y, except through its impact on x. There are a bunch of different ways that people express this. One is that there are no backdoor paths between x and y. Another is that X should be uncorrelated with the error term of the model.

barmstrong 6y ago

This is a great summary of how to show causal effects. Thx for pointing it out. I feel like we need a new way to do peer review, that is more real time - so that papers can be upvoted/downvoted, flaws can be pointed out - and we have some way to assess the truthiness of what the paper is claiming. Your comment is a step in this direction (but we're not capturing the wisdom of the crowds quantitatively around papers today - arxiv is great, but 1990's era web design). I'm working with a team that is trying to build better tools for science at https://www.researchhub.com/about https://www.researchhub.com/about Rethinking peer review is one item on the roadmap.

thomasahle 6y ago
> we need a new way to do peer review ... so that papers can be upvoted/downvoted This one managed to get 700 upvotes on HN and still be meaningless.
- Melting_Harps 6y ago
  > I feel like we need a new way to do peer review, that is more real time - so that papers can be upvoted/downvoted, flaws can be pointed out - and we have some way to assess the truthiness of what the paper is claiming. Your comment is a step in this direction (but we're not capturing the wisdom of the crowds quantitatively around papers today - arxiv is great, but 1990's era web design). > This one managed to get 700 upvotes on HN and still be meaningless. Agreed. This is why I don't think voting with online peramters, like down voting, will ever be viable choice; because aside from things like Cybil attacks, the reality is that Social Media has normalized and acutely optimized group think and the tribalism that often follow. For those of that don't engage in it its incredibly alarming and disconcerting how many succumb to it's practices in real Life. What I will propose as an alternative is something that was first tried with Andreas Antonpolus' first book 'Mastering Bitcoin' wherein the chapters are each individually uploaded to Github and follows the same process that OSS does, wherein commits and corrections are submitted to alter the books ultimate maintained version, and never really being fully 'finished' and can be amended and annotated as needed to suit the updates that follow (eg: Segwit or Lightening Network) or in this case perhaps a replicated experiment that provides a larger sample size. Replication in Academic Papers is nearly non-existant, despite the notion that Peer Reviewed (especially in STEM) was to be a the critical component that made it invaluable. This could also effectively undo the walled-garden-extrotionist model that afflicts Academia's Peer Reviewed model and foster more interactive and International work without being present in a University. Samples or specimens may need to be more tightly controlled and transported, but this could effectively be done over night if the Will exists for little to no money in anything other than training. If the International University model is undergoing mass disruption and shifting toward a mainly online learning platform, this too could help mitigate these glaring problems and have a Global Repository for ongoing research. Its almost stupid not to do it at this point given the MANY pitfalls that its current model forces down our throats.
  - thomasahle 6y ago
    > commits and corrections are submitted to alter the books I think this is good idea fro books, and I wish more scholars would contribute to Wikipedia and other peer editable overviews. I'm not sure quite how it would work for new results though, for which it is still not quite clear how they fit into the current knowledge, or if they are worthwhile at all. People also tend to feel really strongly about their own original ideas, and may try to push them forward when they should rather be forgotten. I don't know how to prevent something like that, other than having a BDFL or individual publishing (as in the current system)
    - hunter-gatherer 6y ago
      Gitlab has an article on their website somewhere where they address some of your concerns. I'm on mobile now and can't find it, but the article advocates for a 'Handbook first' approach to documentation. We did this at my last job and I found it well worth the effort. The BDFL ended up being the quality of our work. In other words, the most correct version always wins out.
  - chrismatei 6y ago
    I agree with this. I'm currently doing research, and one factor that always makes go nuts was the actual time to go and find an article and then donwloadit. I can say that I have used or rather abused sci-hub so that I can find related papers. Having something like "github" style + the option to visually view the changes updates, what are the connection, who did what, where is coming fro, something in the way the site conectedpapers does is also very powerfull. Lastly, the only concern in here it is that if it's ever leving document how ca you asses as whole one document if its good quality at certain given time ? So my proposal wil follow under "feature request" type where even if the changes are made, having the ability to read the "metadata" or quality check done by lets say AI(plagarims, etc, etc,) + with the feedback from online reviewer = qood quality check.
- on_and_off 6y ago
  Not to mention that it also needs to account for science advancing one funeral at a time. How to distinguish between real flaws or just a good theory that is not yet taken seriously ?
- setgree 6y ago
  > This one managed to get 700 upvotes on HN and still be meaningless. On the other hand, the top/most salient comments on it are all pretty critical of its causal claims.
setgree 6y ago
I just uploaded the papers I’ve co-authored — all were licensed CC 4.0 — to Research Hub and emailed to say hello. Looks like a great platform!

jcorrean 6y ago

Thank you so much for this feedback. My team and I will be more than happy to discuss these ideas, if you want to partner up...

setgree 6y ago
Hi Dr. Correa, Thanks for writing. I've taken a quick look at your paper and its OSF page, and you all clearly know what you're doing and have some great work here. I am not your target audience (I am not an academic), but if I were peer reviewing this, I'd suggest one of two avenues. 1) As Angrist and Krueger put it, "In our view, good instruments often come from detailed knowledge of the economic mechanism and institutions determining the regressor of interest...progress comes from detailed institutional knowledge and the careful investigation and quantification of the forces at work in a particular setting." With this in mind, is it possible to contact the maintainer of Sci-Hub, email her your paper, and ask if she can think of any plausibly exogenous sources of variation in papers' availability on Sci-Hub? Some possibilities that come to mind for me: * In my own experience, papers without DOIs are often harder to find on Sci-Hub. Are there closed-access journals that don't mint DOIs on which you could use propensity score matching to create something of an apples-to-apples comparison? * Is there an identification strategy somewhere in journals with optional APCs for which only some of the papers are open access? * Were there ever any sci-hub outages that lasted periods of many months, and, however long down the line, did the citations of paywalled articles decline relative to their open access peers? 2) If none of this is possible, I suggest removing all causal language from your article and sticking to a predictive model. Observational research is appropriate for exploring and identifying causal hypotheses rather than confirming them; no statistical techniques that attempt to control for unobserved population heterogeneity will persuade me otherwise (I am admittedly an extremist on this position, but my own views hew closely to those of Gerber, Green and Kaplan (2003): http://www.donaldgreen.com/wp-content/uploads/2015/09/Gerber-Green-Kaplan-IllusionofLearning.pdf http://www.donaldgreen.com/wp-content/uploads/2015/09/Gerber...). The Lewbel (2012) paper you cite looks interesting but, to paraphrase Gerber et al., statistical techniques can't account for nonstatistical sources of uncertainty (the inherit unnkowability of whether you've specified the 'correct' model, in the absence of randomization or exogenous variation, means we can't say anything about the biasedness of your estimation procedure). Observational research is great! Generating hypotheses is as important as confirming them. It's just that some research designs license causal inference, and some do not. Best of luck.

setgree 6y ago

Just for the record my definition of an identification strategy is not quite right -- it should be > a way for assessing the impact of x (downloads on sci-hub) on y (citations) which _identifies_ a source of variation in x, that is independent of y, except through its impact on x. There are a bunch of different ways that people express this. One is that there are no backdoor paths between x and y. Another is that X should be uncorrelated with the error term of the model.