3 ms·

"And yes, there is certainly real critique that can happen in climate science..." The kind of critique that routine happens within the scientific community, ov

"And yes, there is certainly real critique that can happen in climate science..." The kind of critique that routine happens within the scientific community, over time. Scientists are highly motivated to prove the majority wrong, because that's how you make your career, by finding something new that everyone else missed. By contrast, the EPA 'critique' will be a political charade, a circus of crackpots and shills assembled by the administration to rationalize its predetermined policy decisions.

gpm 9y ago
If that kind of critique was happening routinely we wouldn't have 50% of scientists not being able to reproduce their own paper [0]. The sad truth is that incentives have aligned so that sufficient critique doesn't happen, and that showing some paper doesn't reproduce isn't going to make your career. Government funding to encourage such critique would be great. I don't disagree on this likely be(com)ing a political charade. [0] http://www.nature.com/news/1-500-scientists-lift-the-lid-on-reproducibility-1.19970 http://www.nature.com/news/1-500-scientists-lift-the-lid-on-...
- trendia 9y ago
  > 50% of scientists not being able to reproduce their own paper In the case of the psychology replication crisis (note that I'm NOT talking about climate science here), the problem was the result of a few factors: 1. the lack of incentives to replicate others' work 2. the use of p-values to determine what gets published 3. the low probability of a true positive These factors are not part of the critique process of peer review. That is, reviewers are looking at whether the methodology is valid. "Did the scientists follow proper procedures? Do the results match up with what they claim?" While these questions can identify methodological errors, they cannot identify when an experiment is a statistical fluke. Suppose the true positive rate is 5% and the false positive rate is 5%. That means that only 10% of experiments will give results that suggest publishing. But if you look at published papers only, the probability that the paper is false is not 5%, it's 50%! The solution to this problem is not to throw out peer review, it's to fix the three issues above. We must create a system that incentivizes replication. Perhaps 1st-year grad students will be required to attempt to replicate existing work. (This would be more interesting for the grad student than taking an additional class!) Perhaps we should move away from p-values (but what do we replace that with?) However, the replication crisis is not really the problem of bad science or corruption -- it's mostly the result of bad incentives and false positives.
  - pas 9y ago
    P-values have the feature as being "simple", but here's the/a better/Bayesian approach to hypothesis testing: https://stats.stackexchange.com/a/20393 https://stats.stackexchange.com/a/20393
- nostrademons 9y ago
  I think your take-away from the reproducibility crisis shouldn't be that science is corrupt, peer review is stuffed with cronies, or scientists don't know what they're doing, it should be that science is really fucking hard: https://fivethirtyeight.com/features/science-isnt-broken/#part1 https://fivethirtyeight.com/features/science-isnt-broken/#pa... My old boss at Google used to run the data science program for Search - this is basic business intelligence, not even the natural sciences. He gave an hour and a half long presentation once on all the ways our experiments can and have gone wrong. It included everything from browser bugs disrupting logging, to service outages that happened in the middle of the experiment, to current events that change the normal traffic pattern, to software bugs that trigger failsafes that shut off the experiment but don't affect the control. By the end of the presentation, engineers were asking "So how is it that we actually get anything right?" My brother-in-law is an actual scientist - he does cancer research, studying gene expression within head & neck tumors. The time it takes to get one rigorous, publishable result is measured in years. And it can be derailed by months by something as simple as a tech mixing up lab samples. The thing is, when people use examples of these techniques going wrong as examples of why science is untrustworthy, they forget that normal human reasoning doesn't even bother trying. It's basically a given that most of the stuff you "know" is false. What would be the reproducibility stats for an everyday human belief like "vaccines cause autism" or "the economy does better under Democratic presidents" or "vitamin C prevents colds"? The virtue of science is in being less wrong, it's not in being right. Which, given how often the average human is wrong, is something I'll happily take.
  - babyrainbow 9y ago
    > everyday human belief like "vaccines cause autism".. A bit funny that you have to tell this, considering where this originated from..
- vannevar 9y ago
  I did qualify my statement by adding "over time." Note Feynman's famous observation about scientific progress in a commencement address at CalTech: "We have learned a lot from experience about how to handle some of the ways we fool ourselves. One example: Millikan measured the charge on an electron by an experiment with falling oil drops, and got an answer which we now know not to be quite right. It's a little bit off because he had the incorrect value for the viscosity of air. It's interesting to look at the history of measurements of the charge of an electron, after Millikan. If you plot them as a function of time, you find that one is a little bit bigger than Millikan's, and the next one's a little bit bigger than that, and the next one's a little bit bigger than that, until finally they settle down to a number which is higher. Why didn't they discover the new number was higher right away? It's a thing that scientists are ashamed of—this history—because it's apparent that people did things like this: When they got a number that was too high above Millikan's, they thought something must be wrong—and they would look for and find a reason why something might be wrong. When they got a number close to Millikan's value they didn't look so hard. And so they eliminated the numbers that were too far off, and did other things like that ..." But the number did eventually converge to a consensus, though it took many years to get there.
neutronicus 9y ago
> Scientists are highly motivated to prove the majority wrong, because that's how you make your career, by finding something new that everyone else missed. Don't romanticize the scientific establishment too much. Yes, you want to find something that everyone else missed and publish it. However, you don't really want to "prove the majority wrong" (they, after all, review your paper and speak often with people on funding committees) in any kind of funding- and career-invalidating way. What you really want is a detail your colleagues overlooked in their largely-correct approach to a problem. This kind of thing - generating results and techniques that your colleagues can integrate into their own research, citing you - is what makes careers. Partly this is because (I honestly believe) the majority of people in my field (plasma physics) are competent and conscientious researchers, so "proving them all wrong" is the futile quest of a crank. However, my point remains - as a junior scientist I do not want to stir up any shit with senior scientists, nor do I want to challenge any narratives that funding agencies like. I mean, I might want to, but it would be counter to my own interests in a big way.
- vannevar 9y ago
  "Proving them wrong" is a bit strong, I admit. Proving something they don't already know is closer to the mark. And sure, in any field there are people who don't want to rock the boat. They are necessary to do the grunt work for the people who do want to rock it. Not everyone needs to be a pioneer, but enough regularly appear that science continues to make great progress.

gpm 9y ago

If that kind of critique was happening routinely we wouldn't have 50% of scientists not being able to reproduce their own paper [0]. The sad truth is that incentives have aligned so that sufficient critique doesn't happen, and that showing some paper doesn't reproduce isn't going to make your career. Government funding to encourage such critique would be great. I don't disagree on this likely be(com)ing a political charade. [0] http://www.nature.com/news/1-500-scientists-lift-the-lid-on-reproducibility-1.19970 http://www.nature.com/news/1-500-scientists-lift-the-lid-on-...

trendia 9y ago
> 50% of scientists not being able to reproduce their own paper In the case of the psychology replication crisis (note that I'm NOT talking about climate science here), the problem was the result of a few factors: 1. the lack of incentives to replicate others' work 2. the use of p-values to determine what gets published 3. the low probability of a true positive These factors are not part of the critique process of peer review. That is, reviewers are looking at whether the methodology is valid. "Did the scientists follow proper procedures? Do the results match up with what they claim?" While these questions can identify methodological errors, they cannot identify when an experiment is a statistical fluke. Suppose the true positive rate is 5% and the false positive rate is 5%. That means that only 10% of experiments will give results that suggest publishing. But if you look at published papers only, the probability that the paper is false is not 5%, it's 50%! The solution to this problem is not to throw out peer review, it's to fix the three issues above. We must create a system that incentivizes replication. Perhaps 1st-year grad students will be required to attempt to replicate existing work. (This would be more interesting for the grad student than taking an additional class!) Perhaps we should move away from p-values (but what do we replace that with?) However, the replication crisis is not really the problem of bad science or corruption -- it's mostly the result of bad incentives and false positives.
- pas 9y ago
  P-values have the feature as being "simple", but here's the/a better/Bayesian approach to hypothesis testing: https://stats.stackexchange.com/a/20393 https://stats.stackexchange.com/a/20393
nostrademons 9y ago
I think your take-away from the reproducibility crisis shouldn't be that science is corrupt, peer review is stuffed with cronies, or scientists don't know what they're doing, it should be that science is really fucking hard: https://fivethirtyeight.com/features/science-isnt-broken/#part1 https://fivethirtyeight.com/features/science-isnt-broken/#pa... My old boss at Google used to run the data science program for Search - this is basic business intelligence, not even the natural sciences. He gave an hour and a half long presentation once on all the ways our experiments can and have gone wrong. It included everything from browser bugs disrupting logging, to service outages that happened in the middle of the experiment, to current events that change the normal traffic pattern, to software bugs that trigger failsafes that shut off the experiment but don't affect the control. By the end of the presentation, engineers were asking "So how is it that we actually get anything right?" My brother-in-law is an actual scientist - he does cancer research, studying gene expression within head & neck tumors. The time it takes to get one rigorous, publishable result is measured in years. And it can be derailed by months by something as simple as a tech mixing up lab samples. The thing is, when people use examples of these techniques going wrong as examples of why science is untrustworthy, they forget that normal human reasoning doesn't even bother trying. It's basically a given that most of the stuff you "know" is false. What would be the reproducibility stats for an everyday human belief like "vaccines cause autism" or "the economy does better under Democratic presidents" or "vitamin C prevents colds"? The virtue of science is in being less wrong, it's not in being right. Which, given how often the average human is wrong, is something I'll happily take.
- babyrainbow 9y ago
  > everyday human belief like "vaccines cause autism".. A bit funny that you have to tell this, considering where this originated from..
vannevar 9y ago
I did qualify my statement by adding "over time." Note Feynman's famous observation about scientific progress in a commencement address at CalTech: "We have learned a lot from experience about how to handle some of the ways we fool ourselves. One example: Millikan measured the charge on an electron by an experiment with falling oil drops, and got an answer which we now know not to be quite right. It's a little bit off because he had the incorrect value for the viscosity of air. It's interesting to look at the history of measurements of the charge of an electron, after Millikan. If you plot them as a function of time, you find that one is a little bit bigger than Millikan's, and the next one's a little bit bigger than that, and the next one's a little bit bigger than that, until finally they settle down to a number which is higher. Why didn't they discover the new number was higher right away? It's a thing that scientists are ashamed of—this history—because it's apparent that people did things like this: When they got a number that was too high above Millikan's, they thought something must be wrong—and they would look for and find a reason why something might be wrong. When they got a number close to Millikan's value they didn't look so hard. And so they eliminated the numbers that were too far off, and did other things like that ..." But the number did eventually converge to a consensus, though it took many years to get there.

neutronicus 9y ago

> Scientists are highly motivated to prove the majority wrong, because that's how you make your career, by finding something new that everyone else missed. Don't romanticize the scientific establishment too much. Yes, you want to find something that everyone else missed and publish it. However, you don't really want to "prove the majority wrong" (they, after all, review your paper and speak often with people on funding committees) in any kind of funding- and career-invalidating way. What you really want is a detail your colleagues overlooked in their largely-correct approach to a problem. This kind of thing - generating results and techniques that your colleagues can integrate into their own research, citing you - is what makes careers. Partly this is because (I honestly believe) the majority of people in my field (plasma physics) are competent and conscientious researchers, so "proving them all wrong" is the futile quest of a crank. However, my point remains - as a junior scientist I do not want to stir up any shit with senior scientists, nor do I want to challenge any narratives that funding agencies like. I mean, I might want to, but it would be counter to my own interests in a big way.

vannevar 9y ago
"Proving them wrong" is a bit strong, I admit. Proving something they don't already know is closer to the mark. And sure, in any field there are people who don't want to rock the boat. They are necessary to do the grunt work for the people who do want to rock it. Not everyone needs to be a pioneer, but enough regularly appear that science continues to make great progress.