7 ms·

Then I think there is a certain disconnect between how scientists understand science and how it is presented to the public at large. It certainly doesn't help i

Then I think there is a certain disconnect between how scientists understand science and how it is presented to the public at large. It certainly doesn't help in this regard that many scientists involved in Environmental Science are also passionate political activists. Maybe it is understood by scientists that peer review has flaws, but when they appear on morning talk shows the term "peer review" is wielded as a holy sword. Regardless of the state of the current process, how could review become anything but better from increased openness? >"And since reviewers know that they can't verify correctness, they tend to look for theoretical papers that very closely reproduce experimental results." This seems to be part of the problem. "We'll use this data series for the 60 years that it agrees with everybody else, then throw it out for the 40 years where it doesn't, then not show anybody the 40 years we threw out". But then again, I'm no climate scientist. Although, one would think that no harm would be done by releasing the data, even if there were legitimate statistical reasons for truncating it. >"Ultimately, you have a system where a lot of genuinely smart, skeptical people are attacking ideas from all sides until either a consensus emerges, or the idea is destroyed." If you're a smart, skeptical person attacking a paper that shows global warming to be severe, Michael Mann will get you fired. Or ignore your ideas. Mann just published a paper that appears to use an inverted data series after McIntyre already correctly pointed out that it was inverted, out of pride I suppose. Fine science, that.

DanielBMarkham 17y ago
It certainly doesn't help in this regard that many scientists involved in Environmental Science are also passionate political activists. Absolutely. I think we're going to have to have some kind of new certification system for research that affects public policy. Scientists simply can not be activists -- the conflict of interest is too great. If you find a meteor is going to strike the earth next year, you'd better be spending your time on orbital calculations and mass estimates, not on the Today show. Leave that to the politicians. Mind your knitting. In the code arena I'll be charitable and say that mistakes were made at CRU. Public-funded science should run from a public wiki or source control system where the scientists are the only authors but the public at large are readers. Yes, I know that will drive some scientists mad compared to the current secretive system but it's time to let some daylight in, folks. You'll never, ever win an argument with an honest skeptic if you're keeping secrets. As for the dishonest ones? They're not your job. Mind your knitting.
- DanielBMarkham 17y ago
  EDIT: I see some comments that are already talking about huge data volumes. To those guys, simply take a look at Google, or Amazon, or the credit-card processors. Lots of places deal with huge datasets in a peer-reviewed, transparent manner. Of course,use common sense here.
  - frig 17y ago
    In what meaningful sense do Google or Amazon have peer-reviewed and transparent data sets? Please be specific.
    - DanielBMarkham 17y ago
      I'll pick on Google, and use memory. 1) Google is notorious for peer-reviewing code. 2) IIRC, Google also has some functional/provable code In general, large, critical datasets are routinely processed by businesses. Businesses who use separate testing groups, code reviews, and tiger teams to validate each others' work. None of this is new information (at least to me) Of course, transparency doesn't extend beyond the organization's walls in many instances. If you thought I was saying that it did then I mis-communicated.
      - frig 17y ago
        Yes, you did miscommunicate; even allowing for the miscommunication you're making an extremely poorly-considered analogy. The kind of peer review that matters for purposes of scientific integrity is review by outsiders; eg, a paper is "peer-reviewed" when experts in the field not involved in writing it or conducting its research give it a going-over and see if it appears to hold up. The kind of transparency that matters for purposes of scientific integrity is making data available as-is to outsiders, so that they can meaningfully replicate your results ab initio (or, perhaps, not!). Neither Google nor Amazon conducts meaningful amounts of peer review in the scientific sense nor are they transparent in the scientific sense (nor should they be, the last thing I want is any old anyone seeing the raw data backing someone else's gmail account or search history). So you're making a useless assertion in the context of the issue at hand: neither Google nor Amazon does much "peer review" (to my knowledge Microsoft in fact does to some limited extent with shared source and hiring 3rd party auditors for some important code chunks); neither is "transparent". In this thread you'll find multiple people speaking from positions of actual experience working on large-scale endeavors of scientific computation who've commented at length upon their internal practices. I'm not going to repeat what they've typed but if you read those comments you will see that at least those commenting here were in fact engaging in what you're calling "peer review" and "transparency" wrt their development practices; their reports match my experience concerning scientific computation with large annual budgets but I won't claim any authority for my anecdotal experiences. I will close out by posting a protip here; you might benefit from it but it's not specifically aimed at you. If a blog with a name like "chicagoboyz" makes a bold, sweeping, and somewhat shocking assertion about an entire area of human endeavor -- that it itself has no plausible claim to expertise in (it's an ec blog, not a large-scale computation blog, and no claim of direct experience was made that I saw) -- and you find yourself nodding your head and thinking "yeah, that sounds plausible", proceed to do the following: - slow down, step away from the computer, and count to 20 backwards in greek - ask yourself: do I have any concrete knowledge, at all, about the area this claim is being made? Any involvement in a project in that area, or business involving that area, etc.? (In this case: do you have any direct experience with large-scale computational efforts in science? do you know anyone who's been involved in such a project? anything beyond the flamebait du jour?) - if you do have any concrete knowledge: great, you have at least some nonzero evidence base from which your initial "yeah, that's right feeling" may or may not be substantiated. Think carefully about what you already know and see if the "yeah that's right" feeling holds up. - if you don't have any concrete knowledge: you've given yourself an awesome opportunity for self-discovery and personal growth. Clearly there's something that makes you want to uncritically believe this specific sweeping claim about some area about which you literally know nothing concrete; we generally consider who believe sweeping claims without evidence suckers, and we've found an area where your preexisting biases leave you a sucker, and therefore at the mercy of others. You might still have the right intuition about the sweeping claim, but at least take the opportunity to de-suckerify yourself on this front before drawing your conclusion.
timr 17y ago
"Then I think there is a certain disconnect between how scientists understand science and how it is presented to the public at large." There's clearly a disconnect between how science is performed, and how it is perceived. It's not clear that this is the fault of scientists. It's also very clear that certain political groups intentionally cultivate this misunderstanding, so that they can convince the public that an entire field of science is wrong. "It certainly doesn't help in this regard that many scientists involved in Environmental Science are also passionate political activists." I have unpleasant news for you, then: the "skeptical" side is composed almost exclusively of politicians and economists. "This seems to be part of the problem. 'We'll use this data series for the 60 years that it agrees with everybody else, then throw it out for the 40 years where it doesn't, then not show anybody the 40 years we threw out'." That's not actually what they did, and the fact that you don't understand the nuances of the subject goes back to what I was saying about spending more time looking for "smoking guns" than actually understanding the science. The tree-ring data wasn't omitted to make models fit better (that paper wasn't even about a model) -- it was omitted because it was unreliable data. If you ask me, the tree-ring data should never have been included in that paper at all. And if it weren't, as far as I can tell, it wouldn't have affected the results -- except that there would be an obvious (but largely irrelevant) gap in modern temperature data -- and the "skeptics" would no doubt be attacking some other absurd corner of the climate change research.
- jacoblyles 17y ago
  >"It's not clear that this is the fault of scientists. It's also very clear that certain political groups intentionally cultivate this misunderstanding, so that they can convince the public that an entire field of science is wrong." It sounds like you're excusing bad behavior by saying that other people sometimes behave badly. There is no "certain political group" that is over-representing the value of "peer review" to the public, unless we are talking about environmentalists: http://www.reuters.com/article/latestCrisis/idUSGEE5AP1Y5 http://www.reuters.com/article/latestCrisis/idUSGEE5AP1Y5 No "certain political group" forced the scientists involved in this scandal to act in unethical and secretive manners. Just because politicians act in unethical and dishonest ways doesn't mean that scientists are excused for sinking to their level.
tel 17y ago
Regardless of the state of the current process, how could review become anything but better from increased openness? Increased openness is definitely a direction scientific computing needs to move toward. The trick is conflating that with "peer review" which often, for all its merit, is strongly tied to red tape and frustration. It's impossible to peer review software not because it's impossible to verify techniques but because without some level of trust somewhere it becomes impossible to publish, to communicate. There's definitely a startup in there. There are tons of opportunities to kill off scientific journals as they are. The challenge is that you can't kill peer review at the same time. Peer review is still, at the end of the day, critical. It's not a holy sword of truth but instead more like rinsing your plates in the sink before you throw them in the washing machine.

DanielBMarkham 17y ago

It certainly doesn't help in this regard that many scientists involved in Environmental Science are also passionate political activists. Absolutely. I think we're going to have to have some kind of new certification system for research that affects public policy. Scientists simply can not be activists -- the conflict of interest is too great. If you find a meteor is going to strike the earth next year, you'd better be spending your time on orbital calculations and mass estimates, not on the Today show. Leave that to the politicians. Mind your knitting. In the code arena I'll be charitable and say that mistakes were made at CRU. Public-funded science should run from a public wiki or source control system where the scientists are the only authors but the public at large are readers. Yes, I know that will drive some scientists mad compared to the current secretive system but it's time to let some daylight in, folks. You'll never, ever win an argument with an honest skeptic if you're keeping secrets. As for the dishonest ones? They're not your job. Mind your knitting.

DanielBMarkham 17y ago
EDIT: I see some comments that are already talking about huge data volumes. To those guys, simply take a look at Google, or Amazon, or the credit-card processors. Lots of places deal with huge datasets in a peer-reviewed, transparent manner. Of course,use common sense here.
- frig 17y ago
  In what meaningful sense do Google or Amazon have peer-reviewed and transparent data sets? Please be specific.
  - DanielBMarkham 17y ago
    I'll pick on Google, and use memory. 1) Google is notorious for peer-reviewing code. 2) IIRC, Google also has some functional/provable code In general, large, critical datasets are routinely processed by businesses. Businesses who use separate testing groups, code reviews, and tiger teams to validate each others' work. None of this is new information (at least to me) Of course, transparency doesn't extend beyond the organization's walls in many instances. If you thought I was saying that it did then I mis-communicated.
    - frig 17y ago
      Yes, you did miscommunicate; even allowing for the miscommunication you're making an extremely poorly-considered analogy. The kind of peer review that matters for purposes of scientific integrity is review by outsiders; eg, a paper is "peer-reviewed" when experts in the field not involved in writing it or conducting its research give it a going-over and see if it appears to hold up. The kind of transparency that matters for purposes of scientific integrity is making data available as-is to outsiders, so that they can meaningfully replicate your results ab initio (or, perhaps, not!). Neither Google nor Amazon conducts meaningful amounts of peer review in the scientific sense nor are they transparent in the scientific sense (nor should they be, the last thing I want is any old anyone seeing the raw data backing someone else's gmail account or search history). So you're making a useless assertion in the context of the issue at hand: neither Google nor Amazon does much "peer review" (to my knowledge Microsoft in fact does to some limited extent with shared source and hiring 3rd party auditors for some important code chunks); neither is "transparent". In this thread you'll find multiple people speaking from positions of actual experience working on large-scale endeavors of scientific computation who've commented at length upon their internal practices. I'm not going to repeat what they've typed but if you read those comments you will see that at least those commenting here were in fact engaging in what you're calling "peer review" and "transparency" wrt their development practices; their reports match my experience concerning scientific computation with large annual budgets but I won't claim any authority for my anecdotal experiences. I will close out by posting a protip here; you might benefit from it but it's not specifically aimed at you. If a blog with a name like "chicagoboyz" makes a bold, sweeping, and somewhat shocking assertion about an entire area of human endeavor -- that it itself has no plausible claim to expertise in (it's an ec blog, not a large-scale computation blog, and no claim of direct experience was made that I saw) -- and you find yourself nodding your head and thinking "yeah, that sounds plausible", proceed to do the following: - slow down, step away from the computer, and count to 20 backwards in greek - ask yourself: do I have any concrete knowledge, at all, about the area this claim is being made? Any involvement in a project in that area, or business involving that area, etc.? (In this case: do you have any direct experience with large-scale computational efforts in science? do you know anyone who's been involved in such a project? anything beyond the flamebait du jour?) - if you do have any concrete knowledge: great, you have at least some nonzero evidence base from which your initial "yeah, that's right feeling" may or may not be substantiated. Think carefully about what you already know and see if the "yeah that's right" feeling holds up. - if you don't have any concrete knowledge: you've given yourself an awesome opportunity for self-discovery and personal growth. Clearly there's something that makes you want to uncritically believe this specific sweeping claim about some area about which you literally know nothing concrete; we generally consider who believe sweeping claims without evidence suckers, and we've found an area where your preexisting biases leave you a sucker, and therefore at the mercy of others. You might still have the right intuition about the sweeping claim, but at least take the opportunity to de-suckerify yourself on this front before drawing your conclusion.

timr 17y ago

"Then I think there is a certain disconnect between how scientists understand science and how it is presented to the public at large." There's clearly a disconnect between how science is performed, and how it is perceived. It's not clear that this is the fault of scientists. It's also very clear that certain political groups intentionally cultivate this misunderstanding, so that they can convince the public that an entire field of science is wrong. "It certainly doesn't help in this regard that many scientists involved in Environmental Science are also passionate political activists." I have unpleasant news for you, then: the "skeptical" side is composed almost exclusively of politicians and economists. "This seems to be part of the problem. 'We'll use this data series for the 60 years that it agrees with everybody else, then throw it out for the 40 years where it doesn't, then not show anybody the 40 years we threw out'." That's not actually what they did, and the fact that you don't understand the nuances of the subject goes back to what I was saying about spending more time looking for "smoking guns" than actually understanding the science. The tree-ring data wasn't omitted to make models fit better (that paper wasn't even about a model) -- it was omitted because it was unreliable data. If you ask me, the tree-ring data should never have been included in that paper at all. And if it weren't, as far as I can tell, it wouldn't have affected the results -- except that there would be an obvious (but largely irrelevant) gap in modern temperature data -- and the "skeptics" would no doubt be attacking some other absurd corner of the climate change research.

jacoblyles 17y ago
>"It's not clear that this is the fault of scientists. It's also very clear that certain political groups intentionally cultivate this misunderstanding, so that they can convince the public that an entire field of science is wrong." It sounds like you're excusing bad behavior by saying that other people sometimes behave badly. There is no "certain political group" that is over-representing the value of "peer review" to the public, unless we are talking about environmentalists: http://www.reuters.com/article/latestCrisis/idUSGEE5AP1Y5 http://www.reuters.com/article/latestCrisis/idUSGEE5AP1Y5 No "certain political group" forced the scientists involved in this scandal to act in unethical and secretive manners. Just because politicians act in unethical and dishonest ways doesn't mean that scientists are excused for sinking to their level.

tel 17y ago

Regardless of the state of the current process, how could review become anything but better from increased openness? Increased openness is definitely a direction scientific computing needs to move toward. The trick is conflating that with "peer review" which often, for all its merit, is strongly tied to red tape and frustration. It's impossible to peer review software not because it's impossible to verify techniques but because without some level of trust somewhere it becomes impossible to publish, to communicate. There's definitely a startup in there. There are tons of opportunities to kill off scientific journals as they are. The challenge is that you can't kill peer review at the same time. Peer review is still, at the end of the day, critical. It's not a holy sword of truth but instead more like rinsing your plates in the sink before you throw them in the washing machine.