3 ms·

It would, however, be easy for a non-peer reviewed amateur to make a mistake. It's an appeal to authority, but in this case I don't believe it's a fallacy. Fur

by jakevn 12y ago

It would, however, be easy for a non-peer reviewed amateur to make a mistake. It's an appeal to authority, but in this case I don't believe it's a fallacy. Furthermore, since peer review is fallible, I don't believe Clausen's identical mistake making it through peer review proves the process any less important. Perhaps Clausen's criticisms were not well intended, but wouldn't any respectable scientist demand rigor before seriously considering such a hypothesis? After all, the burden of proof is not on Clausen.

notacoward 12y ago
The mirror twins of ad hominem and ad verecundiam (appeal to authority) are always fallacies, regardless of person or circumstances. To say that certain people should get away with them is itself an appeal to authority. In logic, science, or debate, the identity of the person making or disputing a claim is irrelevant. If Clausen had a concern about the data, he should have said so. If he had a concern about the reasoning applied to that data, he should have said so. That would have been "demanding rigor" as you put it, but he did neither. Instead he addressed identity, and that's where he came off as snotty.
- deleted 12y ago
  [deleted]
- jakevn 12y ago
  He did have a concern about the data. His concern that was it was not peer-reviewed. Had the data come from someone with a proven track record in the field, he may have found it worth his time to peer review it himself. Appeal to authority is not always a fallacy. It would have been fallacious if he was outright denying or accepting a hypothesis as being the logical conclusion based on one's standing. All that I read from it, however, was that he had a hard time believing it, and would not spend the time to look into the data personally, but that it should be reviewed. I don't think it's illogical to take more seriously the hypothesis of someone with a proven track record of good research.
  - notacoward 12y ago
    If he couldn't spend the time to examine it, he should have just said that instead of taking the time to blather about journalistic vs. academic review processes. Clearly he did have the time; he just used it poorly.
    - sanderjd 12y ago
      Your argument is interesting, but it seems to boil down to dismissing the entire system of peer reviewed journals. If they can't be appealed to, and everything needs to be independently researched based on personal expertise, then they have no purpose. Would you agree with that?
      - mmusson 12y ago
        I took his meaning to be that appealing to peer reviewed journals as an "authority" is what is wrong. The value of the peer review system is the rigorous process and the continuing search for the truth.
- aamar 12y ago
  > In logic, science, or debate, the identity of the person making or disputing a claim is irrelevant. Are you describing logic/science/debate as you believe it to be or as you believe it should be? If the latter, are you suggesting ending peer review?
- Jach 12y ago
  Appeal to authority may be a fallacy in strictly formal logical reasoning, but it's trivially shown in probabilistic reasoning (which is the core of scientific reasoning -- reasoning under uncertainty) that if the data supports experts being right in disagreements on claims more often than non-experts, then an expert making a claim provides at least weak evidence in favor compared to a non-expert's opposition, knowing nothing else about the claim's veracity. Unless you're arguing by trading Coq proofs, it might help to consider a probabilistic approach for natural language arguments with degrees of informality and stop fighting over the various fallacies that don't really apply outside of strict formal logic arguments and instead spend time seeking the truth. For more, see: http://www.gwern.net/docs/statistics/2003-korb.pdf http://www.gwern.net/docs/statistics/2003-korb.pdf http://www.researchgate.net/profile/Mike_Oaksford/publication/6199092_The_rationality_of_informal_argumentation_a_Bayesian_approach_to_reasoning_fallacies/links/0c96051e9a3d941970000000.pdf http://www.researchgate.net/profile/Mike_Oaksford/publicatio... http://www.pfeifer-research.de/pdf/pfeifer08rh.pdf http://www.pfeifer-research.de/pdf/pfeifer08rh.pdf I can see how Clausen comes off as snotty. But his response is pretty neutral, even if he praises peer review as a filter beyond its actual merits. He sees a lack of appropriate effort to establish the claim from a nobody, why should he not be skeptical of such claims in general? It's the same heuristic people like Aaronson make on being highly skeptical of the latest P==NP or P!=NP proof uploaded to arxiv.
saalweachter 12y ago
I'd like to chime in here. The valuable thing isn't individual peer-reviewed papers, it's bodies of peer-reviewed works. "Peer-review" as most people think of it -- the vetting of papers by a few reviewers before publication -- is the start of the process, not the end. Peer-review is what happens after your paper is published and your peers -- the people who read the papers you publish -- read your paper and discuss your paper and try and succeed or fail to replicate your results. A journal isn't "peer-reviewed" because a couple of schmucks read the paper before publication; that's just the scientific equivalent of Fizz-Buzz to avoid wasting everyone's time with obvious crap. It's "peer-reviewed" because it is widely and critically read by a large number of practitioners in a field. You should never regard an individual paper uncritically just because it is published. Later, in light of a larger body of research, you might regard a paper as particularly insightful or seminal.
- lotharbot 12y ago
  This is a wonderfully clear explanation. Well said. To generalize a little more: science isn't about single pieces of data or single experiments. Science is about the process. Experiment/data -> idea -> experiment/data -> refined idea -> ... until the idea and the data match to the greatest possible degree (a temporary state of affairs, until science in other areas makes it possible to refine measurements even more tightly.) A single paper, even one with solid methodology, extensive data, and clear patterns, is just one part of science. Replication, refinement, methodological improvements, identification of weaknesses, etc. are what make the whole institution of science so powerful.

by jakevn 12y ago

notacoward 12y ago

The mirror twins of ad hominem and ad verecundiam (appeal to authority) are always fallacies, regardless of person or circumstances. To say that certain people should get away with them is itself an appeal to authority. In logic, science, or debate, the identity of the person making or disputing a claim is irrelevant. If Clausen had a concern about the data, he should have said so. If he had a concern about the reasoning applied to that data, he should have said so. That would have been "demanding rigor" as you put it, but he did neither. Instead he addressed identity, and that's where he came off as snotty.

deleted 12y ago
[deleted]
jakevn 12y ago
He did have a concern about the data. His concern that was it was not peer-reviewed. Had the data come from someone with a proven track record in the field, he may have found it worth his time to peer review it himself. Appeal to authority is not always a fallacy. It would have been fallacious if he was outright denying or accepting a hypothesis as being the logical conclusion based on one's standing. All that I read from it, however, was that he had a hard time believing it, and would not spend the time to look into the data personally, but that it should be reviewed. I don't think it's illogical to take more seriously the hypothesis of someone with a proven track record of good research.
- notacoward 12y ago
  If he couldn't spend the time to examine it, he should have just said that instead of taking the time to blather about journalistic vs. academic review processes. Clearly he did have the time; he just used it poorly.
  - sanderjd 12y ago
    Your argument is interesting, but it seems to boil down to dismissing the entire system of peer reviewed journals. If they can't be appealed to, and everything needs to be independently researched based on personal expertise, then they have no purpose. Would you agree with that?
    - mmusson 12y ago
      I took his meaning to be that appealing to peer reviewed journals as an "authority" is what is wrong. The value of the peer review system is the rigorous process and the continuing search for the truth.
aamar 12y ago
> In logic, science, or debate, the identity of the person making or disputing a claim is irrelevant. Are you describing logic/science/debate as you believe it to be or as you believe it should be? If the latter, are you suggesting ending peer review?
Jach 12y ago
Appeal to authority may be a fallacy in strictly formal logical reasoning, but it's trivially shown in probabilistic reasoning (which is the core of scientific reasoning -- reasoning under uncertainty) that if the data supports experts being right in disagreements on claims more often than non-experts, then an expert making a claim provides at least weak evidence in favor compared to a non-expert's opposition, knowing nothing else about the claim's veracity. Unless you're arguing by trading Coq proofs, it might help to consider a probabilistic approach for natural language arguments with degrees of informality and stop fighting over the various fallacies that don't really apply outside of strict formal logic arguments and instead spend time seeking the truth. For more, see: http://www.gwern.net/docs/statistics/2003-korb.pdf http://www.gwern.net/docs/statistics/2003-korb.pdf http://www.researchgate.net/profile/Mike_Oaksford/publication/6199092_The_rationality_of_informal_argumentation_a_Bayesian_approach_to_reasoning_fallacies/links/0c96051e9a3d941970000000.pdf http://www.researchgate.net/profile/Mike_Oaksford/publicatio... http://www.pfeifer-research.de/pdf/pfeifer08rh.pdf http://www.pfeifer-research.de/pdf/pfeifer08rh.pdf I can see how Clausen comes off as snotty. But his response is pretty neutral, even if he praises peer review as a filter beyond its actual merits. He sees a lack of appropriate effort to establish the claim from a nobody, why should he not be skeptical of such claims in general? It's the same heuristic people like Aaronson make on being highly skeptical of the latest P==NP or P!=NP proof uploaded to arxiv.

saalweachter 12y ago

I'd like to chime in here. The valuable thing isn't individual peer-reviewed papers, it's bodies of peer-reviewed works. "Peer-review" as most people think of it -- the vetting of papers by a few reviewers before publication -- is the start of the process, not the end. Peer-review is what happens after your paper is published and your peers -- the people who read the papers you publish -- read your paper and discuss your paper and try and succeed or fail to replicate your results. A journal isn't "peer-reviewed" because a couple of schmucks read the paper before publication; that's just the scientific equivalent of Fizz-Buzz to avoid wasting everyone's time with obvious crap. It's "peer-reviewed" because it is widely and critically read by a large number of practitioners in a field. You should never regard an individual paper uncritically just because it is published. Later, in light of a larger body of research, you might regard a paper as particularly insightful or seminal.

lotharbot 12y ago
This is a wonderfully clear explanation. Well said. To generalize a little more: science isn't about single pieces of data or single experiments. Science is about the process. Experiment/data -> idea -> experiment/data -> refined idea -> ... until the idea and the data match to the greatest possible degree (a temporary state of affairs, until science in other areas makes it possible to refine measurements even more tightly.) A single paper, even one with solid methodology, extensive data, and clear patterns, is just one part of science. Replication, refinement, methodological improvements, identification of weaknesses, etc. are what make the whole institution of science so powerful.