12 ms·

Seems to me this issue is getting to the point where it could become an existential threat to the credibility of science in general. Note how climate-change den

Seems to me this issue is getting to the point where it could become an existential threat to the credibility of science in general. Note how climate-change deniers have recently used these sorts of arguments to challenge the consensus - is it really so far-fetched to argue that perhaps climate scientists are as biased as researchers in areas such as medicine and linguistics? The pay-wall, blind peer review process seems broken beyond repair. There needs to be a better, robust method to publish every relevant study that is not utter crank, and get some sort of crowd-sourced consensus from researchers with credible reputations.

fchopin 10y ago
> and get some sort of crowd-sourced consensus from researchers with credible reputations. Would be nice, but where would the money come from? Even if you live in a country with high taxes that fund ridiculous projects, they are still often accountable to the public. Would the public want to fund new research or try to replicate or disprove other studies? The only way to make it work would be to build-in a certain percentage of funding for such a purpose in reaction to some gross oversight, so you could get the public behind it, or perhaps get some wealthy donors behind the cause, but I think you'll have a difficult time. The better option might be having research students be required to replicate at least one study that hasn't been replicated in addition to doing their own research. That way you get some free labor!
- apathy 10y ago
  Depends. I would like to think that "the public" would notice all those silly cancer patients showing up to clinic (don't they know that cancer is cured twice a week at every major hype factory, err, university?) but, no. Pie in the sky it is, then! If only a field existed that attempted to quantify the expected variation from a given size and design of experiment. That field would be of great use to major nation states. So much so, you might call it statistics. The practitioners would probably be big party poopers, always referring to weird concepts like "regression to the mean", "power", "effect size" and other unexciting yet critical details. I imagine there would be funny videos on YouTube about these people, and "real" researchers would happily ignore them while they jerk each other off in P01 study sections. Anyways, let's keep science the way it is. There's no way the process or its results could possibly be improved!
  - fchopin 10y ago
    > Anyways, let's keep science the way it is. There's no way the process or its results could possibly be improved! Let it be know that this is sarcasm, for those that have trouble determining that. Note: I was being completely serious about having graduate students being required to replicate studies that have not been replicated, or to disprove them. That was not sarcasm.
    - apathy 10y ago
      Graduate students are effectively required to replicate studies already. They just don't get any credit for the work when it turns out that the flashy paper didn't replicate, and the journal editors hate to look foolish, so it's exceedingly rare for failures to replicate to see a journal. So the next graduate student to work on the technique also gets to waste months or years on it. Really a splendid system, isn't it? Sometimes (rarely) I sympathize with industry types who want grant claw-back processes. Then I remember that those are the same crooks that pushed Vioxx and 510k "equivalent" medical devices. And the sympathy evaporates, because they're even worse.
Animats 10y ago
It may take a toll of the social sciences and biology. It's not a big problem in physics or chemistry.
- apathy 10y ago
  Oh, "Hollywood" chemistry is pretty fucking bad, but for different reasons. I miss being a real scientist sometimes.
nostrademons 10y ago
> There needs to be a better, robust method to publish every relevant study that is not utter crank, and get some sort of crowd-sourced consensus from researchers with credible reputations. Isn't that the Internet? I guess the problem on the Internet is that you also get every irrelevant study that is utter crank, and a crowd-sourced consensus from laypeople with no credibility whatsoever.
- apathy 10y ago
  Peer review is boosting with three weak learners. If you think that has much credibility (after STAP, arsenic life, LaCour, etc) you clearly haven't been paying attention. Nb. I review for various journals, but the process is far from foolproof. I do the best I can, but editors are free to override us in the interest of "impact". At least with preprints and PubMedCommons anyone with a cogent rebuttal can present it. Cell Press must hate that...
  - devinhelton 10y ago
    Curious question -- when you or someone else does the peer review, is it standard practice to actually double-check the math behind the paper, or audit any of the data?
    - mattkrause 10y ago
      No. At least not in neuroscience/psychology. I've never seen a review request that was accompanied by raw data--you typically get an unformatted version of the manuscript, along with the tables and figures. That's it. The reviewers can comment on anything, but they tend to be pretty conceptual. For example, one might say that the manuscript claims X, but the authors need to rule out competing hypothesis Y. Good reviewers will either suggest ways to do that (e.g., by doing a specific control experiment or citing a paper that rules Y out). They might ask you to comment on why your data claims A, but some previously published work indicates that !A. To the extent that statistics get reviewed (not enough), reviewers typically comment whether they think the methods are suitable for the specific application or whether their output is being interpreted correctly. However, it's exceedingly unlikely that someone will actually check whether a t-test on the data in Table #2 actually gives the p-value printed at the bottom (or whatever).
the_watcher 10y ago
> is it really so far-fetched to argue that perhaps climate scientists are as biased as researchers in areas such as medicine and linguistics? No, it isn't, unless you believe that climate scientists are a definitionally more moral group than others. The incentives remain the same, in all branches of science, so manipulation is likely the same in all branches.
- selectron 10y ago
  Physicists have a better understanding of statistics and better data to work with than pyschologists. There is a very clear causal explanation for why greenhouse gases do what they do. With that said, there is not enough discussion about the cost and effectiveness of proposed solutions to global warming - a lot of the "solutions" are more feel-good measures, which are not worth the cost for what they do. Global warming is a global problem - if the US stopped all of its contributions to climate change the earth would keep warming up. As a physics grad student I have seen plenty of "negative" results published, and the standard of positive results for instance 3 or even 5 sigma is a much tighter standard than p > 0.05. Science is a big field, the problems in one domain do not necessarily translate into all domains. However there are other fundamental problems with physics as a field.
  - gd1 10y ago
    >There is a very clear causal explanation for why greenhouse gases do what they do I'm sorry, but it doesn't work that way. We have very clear chemistry/physics on the composition of food, how it is digested, etc. But nutrition science is still an embarrassing shitshow. We have clear physics on how neurons fire, etc. but neuroscience is still in its infancy and psychology... well, it is psychology. Earth's climate is a large scale multi-variable control system with thousands of feedback loops. A good understanding of the physics that drive a single forcing doesn't really tell us shit I'm afraid. It is just as open to manipulation as these other fields.
    - selectron 10y ago
      This sentiment is not correct - causal explanations do matter. The statistical evidence for global warming is not particularly strong, as temperature data is very noisy. If we didn't have a causal explanation there would not be a scientific consensus behind climate change. That adding greenhouse gases to the atmosphere changes the climate is more like gravity - we have clear physics on why things fall when you drop them, and if we somehow were adding mass to the center of the earth we know what effect this would have. More generally you are correct, we don't know exactly how much the earth will warm or if there are complicated feedback mechanisms in place that could cause this warming to speed up or reverse course. We can't even reliably say that next year will be hotter than this year (actually it probably will be cooler because this year has been unusually hot).
kriro 10y ago
Another point that is briefly touched upon is the availability of data. What good is a study I can't verify? I wish there was a tag that could be applied to published papers along the lines of "the original data is not available or the author is unwilling to share it, treat this very septically". If you ever feel like having a sad couple of weeks, pick 50 empirical papers and send the authors emails along the lines of "I just read your paper X, very interesting. Would you kindly provide me with access to the data so that I can Y". Call me naive but reproducibility is one of the hallmarks of a good scientific publication. I'm also fairly skeptical of studies where the used questionnaires are not openly available...the list goes on (I'd also argue that closed source software should be avoided because "we plugged our data into the not checkable algorithm X and got result Y which we interpreted according to what the authors of the software say we should interpret it as" isn't great either)
- Someone 10y ago
  s/septically/sceptically/ (probably :-)) And preparing data for publication is lots of work [1] that unfortunately doesn't 'pay' in today's scientific culture.mit even runs the risk of others 'mining' publications from your data before you have the chance to do so. The best reply to questions you can expect is "that's a nice follow-up idea. Let's work on it together". Risk there is that the scientists may be working in some other direction that they find more promising. [1] for example, you have 100 sets of data, but only used 78 in _this_ publication for various reasons (chemical used was applied at incorrect time, for too long, was out of date, subject was too old, had an existing condition, etc). You explain that in the publication in a small paragraph. In your dataset, you probably did it by copying the dataset and removing the ones not suitable for the task at hand. Did you keep that selection after the paper was published? Probably not. Even if you did, sending the selected data set out isn't sufficient. Others might want to see all data, so that they can verify the correctness of your selection. Source control systems could help here, but as I said, providing data unfortunately career-wise doesn't pay at the moment for scientists.
robotresearcher 10y ago
Every few days I read that science is broken or losing credibility. And yet somehow the torrent of new technology, medicine and understanding of the world just keeps coming, and often makes my life better. So maybe not completely broken. CRISPR, Higgs' boson, gravity waves, deep networks, self-driving cars. Not broken.
- praptak 10y ago
  It's not about technology. Bullshit science affects policies. Also you don't see what did not get discovered/invented because of people having wasted time trying to build on bad theories. Also, nobody's saying it's completely broken.
- return0 10y ago
  Those are major advances that happen very very rarely. The bulk of science is small incremental steps in science. It is extremely frustrating not being able to trust the results published even in high impact journals. This is a particularly glaring problem for theoreticists/modelers. Each theory/model must rely on potentially hundreds of studies, and it's really hard to assign a confidence level to each of them.
- T-A 10y ago
  CRISPR is a genuinely novel scientific advance. The other items in your list are not. The Higgs mechanism was concocted half a century ago; gravity waves are even older (general relativity is more than a century old). Their experimental confirmations were engineering feats. Neural networks are from the 50s. Add fast computers and you get deep ones. Put them in cars and they learn to drive themselves.
peter303 10y ago
Science is ultimately self-correcting, but that can take decades. Sometimes you have to wait for distinguished asshole scientist to retire or die before happens. For example the Harvard Geology Department was the laughingstock of the world for taking so long to accept the thoery of plate tectonics. But eventually it did after rotating in a new generation of modern faculty.

fchopin 10y ago

> and get some sort of crowd-sourced consensus from researchers with credible reputations. Would be nice, but where would the money come from? Even if you live in a country with high taxes that fund ridiculous projects, they are still often accountable to the public. Would the public want to fund new research or try to replicate or disprove other studies? The only way to make it work would be to build-in a certain percentage of funding for such a purpose in reaction to some gross oversight, so you could get the public behind it, or perhaps get some wealthy donors behind the cause, but I think you'll have a difficult time. The better option might be having research students be required to replicate at least one study that hasn't been replicated in addition to doing their own research. That way you get some free labor!

apathy 10y ago
Depends. I would like to think that "the public" would notice all those silly cancer patients showing up to clinic (don't they know that cancer is cured twice a week at every major hype factory, err, university?) but, no. Pie in the sky it is, then! If only a field existed that attempted to quantify the expected variation from a given size and design of experiment. That field would be of great use to major nation states. So much so, you might call it statistics. The practitioners would probably be big party poopers, always referring to weird concepts like "regression to the mean", "power", "effect size" and other unexciting yet critical details. I imagine there would be funny videos on YouTube about these people, and "real" researchers would happily ignore them while they jerk each other off in P01 study sections. Anyways, let's keep science the way it is. There's no way the process or its results could possibly be improved!
- fchopin 10y ago
  > Anyways, let's keep science the way it is. There's no way the process or its results could possibly be improved! Let it be know that this is sarcasm, for those that have trouble determining that. Note: I was being completely serious about having graduate students being required to replicate studies that have not been replicated, or to disprove them. That was not sarcasm.
  - apathy 10y ago
    Graduate students are effectively required to replicate studies already. They just don't get any credit for the work when it turns out that the flashy paper didn't replicate, and the journal editors hate to look foolish, so it's exceedingly rare for failures to replicate to see a journal. So the next graduate student to work on the technique also gets to waste months or years on it. Really a splendid system, isn't it? Sometimes (rarely) I sympathize with industry types who want grant claw-back processes. Then I remember that those are the same crooks that pushed Vioxx and 510k "equivalent" medical devices. And the sympathy evaporates, because they're even worse.

Animats 10y ago

It may take a toll of the social sciences and biology. It's not a big problem in physics or chemistry.

apathy 10y ago
Oh, "Hollywood" chemistry is pretty fucking bad, but for different reasons. I miss being a real scientist sometimes.

nostrademons 10y ago

> There needs to be a better, robust method to publish every relevant study that is not utter crank, and get some sort of crowd-sourced consensus from researchers with credible reputations. Isn't that the Internet? I guess the problem on the Internet is that you also get every irrelevant study that is utter crank, and a crowd-sourced consensus from laypeople with no credibility whatsoever.

apathy 10y ago
Peer review is boosting with three weak learners. If you think that has much credibility (after STAP, arsenic life, LaCour, etc) you clearly haven't been paying attention. Nb. I review for various journals, but the process is far from foolproof. I do the best I can, but editors are free to override us in the interest of "impact". At least with preprints and PubMedCommons anyone with a cogent rebuttal can present it. Cell Press must hate that...
- devinhelton 10y ago
  Curious question -- when you or someone else does the peer review, is it standard practice to actually double-check the math behind the paper, or audit any of the data?
  - mattkrause 10y ago
    No. At least not in neuroscience/psychology. I've never seen a review request that was accompanied by raw data--you typically get an unformatted version of the manuscript, along with the tables and figures. That's it. The reviewers can comment on anything, but they tend to be pretty conceptual. For example, one might say that the manuscript claims X, but the authors need to rule out competing hypothesis Y. Good reviewers will either suggest ways to do that (e.g., by doing a specific control experiment or citing a paper that rules Y out). They might ask you to comment on why your data claims A, but some previously published work indicates that !A. To the extent that statistics get reviewed (not enough), reviewers typically comment whether they think the methods are suitable for the specific application or whether their output is being interpreted correctly. However, it's exceedingly unlikely that someone will actually check whether a t-test on the data in Table #2 actually gives the p-value printed at the bottom (or whatever).

the_watcher 10y ago

> is it really so far-fetched to argue that perhaps climate scientists are as biased as researchers in areas such as medicine and linguistics? No, it isn't, unless you believe that climate scientists are a definitionally more moral group than others. The incentives remain the same, in all branches of science, so manipulation is likely the same in all branches.

selectron 10y ago
Physicists have a better understanding of statistics and better data to work with than pyschologists. There is a very clear causal explanation for why greenhouse gases do what they do. With that said, there is not enough discussion about the cost and effectiveness of proposed solutions to global warming - a lot of the "solutions" are more feel-good measures, which are not worth the cost for what they do. Global warming is a global problem - if the US stopped all of its contributions to climate change the earth would keep warming up. As a physics grad student I have seen plenty of "negative" results published, and the standard of positive results for instance 3 or even 5 sigma is a much tighter standard than p > 0.05. Science is a big field, the problems in one domain do not necessarily translate into all domains. However there are other fundamental problems with physics as a field.
- gd1 10y ago
  >There is a very clear causal explanation for why greenhouse gases do what they do I'm sorry, but it doesn't work that way. We have very clear chemistry/physics on the composition of food, how it is digested, etc. But nutrition science is still an embarrassing shitshow. We have clear physics on how neurons fire, etc. but neuroscience is still in its infancy and psychology... well, it is psychology. Earth's climate is a large scale multi-variable control system with thousands of feedback loops. A good understanding of the physics that drive a single forcing doesn't really tell us shit I'm afraid. It is just as open to manipulation as these other fields.
  - selectron 10y ago
    This sentiment is not correct - causal explanations do matter. The statistical evidence for global warming is not particularly strong, as temperature data is very noisy. If we didn't have a causal explanation there would not be a scientific consensus behind climate change. That adding greenhouse gases to the atmosphere changes the climate is more like gravity - we have clear physics on why things fall when you drop them, and if we somehow were adding mass to the center of the earth we know what effect this would have. More generally you are correct, we don't know exactly how much the earth will warm or if there are complicated feedback mechanisms in place that could cause this warming to speed up or reverse course. We can't even reliably say that next year will be hotter than this year (actually it probably will be cooler because this year has been unusually hot).

kriro 10y ago

Another point that is briefly touched upon is the availability of data. What good is a study I can't verify? I wish there was a tag that could be applied to published papers along the lines of "the original data is not available or the author is unwilling to share it, treat this very septically". If you ever feel like having a sad couple of weeks, pick 50 empirical papers and send the authors emails along the lines of "I just read your paper X, very interesting. Would you kindly provide me with access to the data so that I can Y". Call me naive but reproducibility is one of the hallmarks of a good scientific publication. I'm also fairly skeptical of studies where the used questionnaires are not openly available...the list goes on (I'd also argue that closed source software should be avoided because "we plugged our data into the not checkable algorithm X and got result Y which we interpreted according to what the authors of the software say we should interpret it as" isn't great either)

Someone 10y ago
s/septically/sceptically/ (probably :-)) And preparing data for publication is lots of work [1] that unfortunately doesn't 'pay' in today's scientific culture.mit even runs the risk of others 'mining' publications from your data before you have the chance to do so. The best reply to questions you can expect is "that's a nice follow-up idea. Let's work on it together". Risk there is that the scientists may be working in some other direction that they find more promising. [1] for example, you have 100 sets of data, but only used 78 in _this_ publication for various reasons (chemical used was applied at incorrect time, for too long, was out of date, subject was too old, had an existing condition, etc). You explain that in the publication in a small paragraph. In your dataset, you probably did it by copying the dataset and removing the ones not suitable for the task at hand. Did you keep that selection after the paper was published? Probably not. Even if you did, sending the selected data set out isn't sufficient. Others might want to see all data, so that they can verify the correctness of your selection. Source control systems could help here, but as I said, providing data unfortunately career-wise doesn't pay at the moment for scientists.

robotresearcher 10y ago

Every few days I read that science is broken or losing credibility. And yet somehow the torrent of new technology, medicine and understanding of the world just keeps coming, and often makes my life better. So maybe not completely broken. CRISPR, Higgs' boson, gravity waves, deep networks, self-driving cars. Not broken.

praptak 10y ago
It's not about technology. Bullshit science affects policies. Also you don't see what did not get discovered/invented because of people having wasted time trying to build on bad theories. Also, nobody's saying it's completely broken.
return0 10y ago
Those are major advances that happen very very rarely. The bulk of science is small incremental steps in science. It is extremely frustrating not being able to trust the results published even in high impact journals. This is a particularly glaring problem for theoreticists/modelers. Each theory/model must rely on potentially hundreds of studies, and it's really hard to assign a confidence level to each of them.
T-A 10y ago
CRISPR is a genuinely novel scientific advance. The other items in your list are not. The Higgs mechanism was concocted half a century ago; gravity waves are even older (general relativity is more than a century old). Their experimental confirmations were engineering feats. Neural networks are from the 50s. Add fast computers and you get deep ones. Put them in cars and they learn to drive themselves.

peter303 10y ago

Science is ultimately self-correcting, but that can take decades. Sometimes you have to wait for distinguished asshole scientist to retire or die before happens. For example the Harvard Geology Department was the laughingstock of the world for taking so long to accept the thoery of plate tectonics. But eventually it did after rotating in a new generation of modern faculty.