4 ms·

This is an excellent overview article on an important topic. It mentions many of the most influential authors on the topic of accuracy of scientific publication

This is an excellent overview article on an important topic. It mentions many of the most influential authors on the topic of accuracy of scientific publications. Hacker News readers may enjoy "Warning Signs in Experimental Design and Interpretation"[1] by Peter Norvig, a LISP hacker who is now director of research at Google, on how to interpret scientific research. Norvig's essay is my all-time favorite link to share in a Hacker News comment. That's because we see submissions here every day of preliminary studies that can be analyzed by Norvig's checklist on research issues to look for when reading about a scientific finding. I discuss psychology research weekly with a group of psychologists who study human behavior genetics in a "journal club" (graduate seminar course). Those researchers have told me about other researchers who are trying to clean up the published literature in psychology, for example Jelte Wicherts, whose article "Letting the daylight in: reviewing the reviewers and other ways to maximize transparency in science"[2] in an open-access journal suggests general procedures to improve scientific publishing, for example by changing the incentive structure around reviewing papers submitted for publication. Another helpful researcher on statistical tests to verify results is Uri Simonsohn. The papers he and his colleagues produce[3] are thought-provoking, pointed, and sometimes laugh-out-loud funny. [1] http://norvig.com/experiment-design.html http://norvig.com/experiment-design.html [2] Jelte M. Wicherts, Rogier A. Kievit, Marjan Bakker and Denny Borsboom. Letting the daylight in: reviewing the reviewers and other ways to maximize transparency in science. Front. Comput. Neurosci., 03 April 2012 doi: 10.3389/fncom.2012.00020 http://www.frontiersin.org/Computational_Neuroscience/10.3389/fncom.2012.00020/full http://www.frontiersin.org/Computational_Neuroscience/10.338... [3] http://opim.wharton.upenn.edu/~uws/ http://opim.wharton.upenn.edu/~uws/

sillysaurus2 13y ago
Norvig's essay is an absolute must-read. You may learn more counter-intuitive facts in the next hour than in the last week. For convenience to mobile users, here's the link again: http://norvig.com/experiment-design.html http://norvig.com/experiment-design.html
foxlisk 13y ago
Norving missed an excellent chance to title this paper "The structure and interpretation of science experiments"
shanusmagnus 13y ago
This beautiful (parent) comment, in contrast to the main article, actually contains links and references to the items it refers to. Is there some compelling reason I don't understand why _so_ many magazines / websites, even high profile reputable ones, cannot manage to link or cite?
- cousin_it 13y ago
  I suspect the reason is that outbound links drive traffic away from ads and internal links. The more visitors you have, the more profit you lose by adding outbound links... It's sad and I wish there was some way to fix the incentives.
  - sdoering 13y ago
    That's exactly what people in charge fear in my industry (working in online publishing). The do fear, that the people find the linked ressources better, so that they may never even return. Kind of like the fear of becoming obsolete. But instead of testing and maybe even becoming better, they just forbid us to link out.
- kybernetikos 13y ago
  I find this particularly galling since many of these publications have fact checkers who have gone to the effort of identifying claims in the original article, tracking down the evidence to prove/disprove it and checking that the article doesn't materially misrepresent the evidence. This is the internet, it shouldn't cost any more to let me see the fact checkers report, and it will save me a massive amount of time reproducing their work. Expandable margin notes on a per paragraph basis would be perfect.
- unhammer 13y ago
  Science Exchange: https://www.scienceexchange.com/reproducibility https://www.scienceexchange.com/reproducibility Ioannidis' article on false pos/neg and stat.power: http://www.plosmedicine.org/article/info:doi/10.1371/journal.pmed.0020124 http://www.plosmedicine.org/article/info:doi/10.1371/journal... Bohannon's "Who's afraid of a little peer review": http://www.sciencemag.org/content/342/6154/60.full http://www.sciencemag.org/content/342/6154/60.full (fake submissions) Also related I think is this: http://neurotheory.columbia.edu/~ken/cargo_cult.html http://neurotheory.columbia.edu/~ken/cargo_cult.html ;-)
deleted 13y ago
[deleted]
Alex3917 13y ago
> Warning Sign I5: Taking p too Seriously What does it say about X when you have 10 studies saying X is true with p < .05, and another 10 studies saying that X is false with p < .05? It seems like the closer you come to actually taking p values at face value given conflicting results, the amount of things that can be considered to be 'scientifically proven' rapidly approaches zero. Also, while I do like norvig's essay, all the errors he goes through are fairly pedestrian. If you read the modern scientific literature on research, the rabbit hole goes vastly deeper. (Assuming researchers researching researchers are more trustworthy than researchers.)
- einhverfr 13y ago
  > If you read the modern scientific literature on research, the rabbit hole goes vastly deeper. On indeed. Heisenberg used to say, "data does not imply theory."[1] Once you deeply internalize that, you realize you can't really be that, ahem, certain about anything in scientific thought outside of the limited domain of how scientific truth is defined (namely that the model has predictive value, which is entirely different from whether it has ontological value). [1] http://archive.org/details/PhysicsPhilosophy http://archive.org/details/PhysicsPhilosophy
  - skylan_q 13y ago
    Try telling that to economists who claim to discover macroeconomic theories through econometric models and call it "empirical"
- dekhn 13y ago
  Nothing is ever "proven". We merely make statements that become increasingly challenging to show are likely to be false based on human judgement of experimental methods. P values are estimates of the probability that a given observation is likely to have come about by chance, rather than due to a hypothetical model. They have many problems and it is best to think about them with a solid understanding of probability. If there was a case where you have as many conflicting results as you say, then it would be indicative of a very active scientific community and problem. People would be working hard to resolve the problem!
deleted 13y ago
[deleted]
unhammer 13y ago
For people interested in reproducability in computational linguistics and NLP, I recommend http://lingpipe-blog.com/2013/09/25/fokkens-et-al-on-reproduction-replication-failure/ http://lingpipe-blog.com/2013/09/25/fokkens-et-al-on-reprodu... and the article linked within. One of the authors of "Replication Failure" has another, slightly older but really awesome paper: http://www.d.umn.edu/~tpederse/Pubs/pedersen-last-word-2008.pdf http://www.d.umn.edu/~tpederse/Pubs/pedersen-last-word-2008....

sillysaurus2 13y ago

Norvig's essay is an absolute must-read. You may learn more counter-intuitive facts in the next hour than in the last week. For convenience to mobile users, here's the link again: http://norvig.com/experiment-design.html http://norvig.com/experiment-design.html

foxlisk 13y ago

Norving missed an excellent chance to title this paper "The structure and interpretation of science experiments"

shanusmagnus 13y ago

This beautiful (parent) comment, in contrast to the main article, actually contains links and references to the items it refers to. Is there some compelling reason I don't understand why _so_ many magazines / websites, even high profile reputable ones, cannot manage to link or cite?

cousin_it 13y ago
I suspect the reason is that outbound links drive traffic away from ads and internal links. The more visitors you have, the more profit you lose by adding outbound links... It's sad and I wish there was some way to fix the incentives.
- sdoering 13y ago
  That's exactly what people in charge fear in my industry (working in online publishing). The do fear, that the people find the linked ressources better, so that they may never even return. Kind of like the fear of becoming obsolete. But instead of testing and maybe even becoming better, they just forbid us to link out.
kybernetikos 13y ago
I find this particularly galling since many of these publications have fact checkers who have gone to the effort of identifying claims in the original article, tracking down the evidence to prove/disprove it and checking that the article doesn't materially misrepresent the evidence. This is the internet, it shouldn't cost any more to let me see the fact checkers report, and it will save me a massive amount of time reproducing their work. Expandable margin notes on a per paragraph basis would be perfect.
unhammer 13y ago
Science Exchange: https://www.scienceexchange.com/reproducibility https://www.scienceexchange.com/reproducibility Ioannidis' article on false pos/neg and stat.power: http://www.plosmedicine.org/article/info:doi/10.1371/journal.pmed.0020124 http://www.plosmedicine.org/article/info:doi/10.1371/journal... Bohannon's "Who's afraid of a little peer review": http://www.sciencemag.org/content/342/6154/60.full http://www.sciencemag.org/content/342/6154/60.full (fake submissions) Also related I think is this: http://neurotheory.columbia.edu/~ken/cargo_cult.html http://neurotheory.columbia.edu/~ken/cargo_cult.html ;-)

deleted 13y ago

[deleted]

Alex3917 13y ago

> Warning Sign I5: Taking p too Seriously What does it say about X when you have 10 studies saying X is true with p < .05, and another 10 studies saying that X is false with p < .05? It seems like the closer you come to actually taking p values at face value given conflicting results, the amount of things that can be considered to be 'scientifically proven' rapidly approaches zero. Also, while I do like norvig's essay, all the errors he goes through are fairly pedestrian. If you read the modern scientific literature on research, the rabbit hole goes vastly deeper. (Assuming researchers researching researchers are more trustworthy than researchers.)

einhverfr 13y ago
> If you read the modern scientific literature on research, the rabbit hole goes vastly deeper. On indeed. Heisenberg used to say, "data does not imply theory."[1] Once you deeply internalize that, you realize you can't really be that, ahem, certain about anything in scientific thought outside of the limited domain of how scientific truth is defined (namely that the model has predictive value, which is entirely different from whether it has ontological value). [1] http://archive.org/details/PhysicsPhilosophy http://archive.org/details/PhysicsPhilosophy
- skylan_q 13y ago
  Try telling that to economists who claim to discover macroeconomic theories through econometric models and call it "empirical"
dekhn 13y ago
Nothing is ever "proven". We merely make statements that become increasingly challenging to show are likely to be false based on human judgement of experimental methods. P values are estimates of the probability that a given observation is likely to have come about by chance, rather than due to a hypothetical model. They have many problems and it is best to think about them with a solid understanding of probability. If there was a case where you have as many conflicting results as you say, then it would be indicative of a very active scientific community and problem. People would be working hard to resolve the problem!

deleted 13y ago

[deleted]

unhammer 13y ago

For people interested in reproducability in computational linguistics and NLP, I recommend http://lingpipe-blog.com/2013/09/25/fokkens-et-al-on-reproduction-replication-failure/ http://lingpipe-blog.com/2013/09/25/fokkens-et-al-on-reprodu... and the article linked within. One of the authors of "Replication Failure" has another, slightly older but really awesome paper: http://www.d.umn.edu/~tpederse/Pubs/pedersen-last-word-2008.pdf http://www.d.umn.edu/~tpederse/Pubs/pedersen-last-word-2008....