4 ms·

Science is often flawed. It's time we embraced that

38 points by simonsarris 11y ago

stonogo 11y ago
This entire article is a criticism of the scientific journal industry, not "science." It is disturbing that the authors appear to have a difficult time compartmentalizing those concepts, but still consider themselves authoritative on the matter.
- deleted 11y ago
  [deleted]
  - deleted 11y ago
    [deleted]
denzil_correa 11y ago
I think the issue stems from a fact that most people overestimate the worth of a peer reviewed article. I like to look at it a bit differently. A research problem is a like a discussion where each peer reviewed article provides one discussion point towards the solution. It by no means is entirely self encompassed but at the same time, it doesn't mean one should discard it. The solution to the research problem isn't entirely contained in a peer reviewed article. Rather, it is one data point towards making an impact towards the larger problem. A collection of research articles is many such discussion points. The final conclusion has to be drawn by collecting all such arguments on the table and synthesizing them.
nosideeffects 11y ago
How about we just ban "journalists" from publishing articles about the results of recently published studies?
- 6t6t6 11y ago
  > ban "journalists" from publishing Maybe not.
hoopd 11y ago
> When people talk about flaws in science, they're often focusing on medical and life sciences, as Horton is. But that might simply be because these fields are furthest along in auditing their own problems. Many of the structural problems in medical science could well apply to other fields, too. Pure FUD; those fields have unusually low levels of empiricism. Rule of thumb: as a science has greater trouble gathering enough data to settle disputes the scientists of that field become more and more like philosophers. They decide what's true based on arguments or personal feelings and in the worst cases they fabricate data to fill the vacuum. The problem is that the core element of science, that of rigorous experimentation, is missing from the process. The question is if a field is only able to follow the aesthetics of science but unable to adopt the rigor to what extent should we consider it science?
- nice1 11y ago
  You'd think climate studies should have a higher standard; yet, as the climategate materials showed, those people suffer from the same "goal oriented" data fudging practices as described in the article.
bitwize 11y ago
The problem is that admitting science is flawed means that people will propose astrology or alkaline diets or antivaccinationism as alternatives. So yes, let's all embrace that science is flawed and then work together to fix science, and not pretend that Christianity or traditional vitalism are better alternatives to science.
- zenogais 11y ago
  This kind of black and white thinking is exactly the problem though. Either science or mysticism. Science will never be infallible, and in fact has strict limits on what it can know [1][2] - there's nothing to fix here, except the perception that fetishizes the scientific process as an infallible method for producing universal truths. This black and white perspective fails to recognize social and psychological biases that underlie scientific production and also lie outside its full understanding. The scientific method is a tool. That being said, we should of course strive to improve this tool. The problem here is one of human perception. What goes for science goes for all things - don't believe the hype. [1]: http://undsci.berkeley.edu/article/0_0_0/whatisscience_12 http://undsci.berkeley.edu/article/0_0_0/whatisscience_12 [2]: http://moreintelligentlife.com/content/ideas/anthony-gottlieb/limits-science http://moreintelligentlife.com/content/ideas/anthony-gottlie...
- neap24 11y ago
  It seems unlikely that admitting science is flawed would immediately cause people to jump to the opposite extreme. Perhaps those who already trumpet those ideas would feel emboldened, but I doubt they would have much influence.
- xj9 11y ago
  Science is not a religion, lets not compare apples to oranges.
  - briantakita 11y ago
    Belief in Science is a religion.
- 13years 11y ago
deadgrey19 11y ago
Academic publishing is a conversation between academics. No one paper is truly authoritative and only with time will the highly cited well regarded contributions make themselves known within the academic community (and eventually outside of it too). The time horizon between publication and "impact" for most academic work is usually (or should be) in the range of 10-20years. In the rush for funding and "impact" university PR machines (and scientists too) are forced to make flashy statements about the latest new sensational, tenuous at best implications of their findings. Boring, rigorous, incremental work has no place in a media driven funding cycle. This is a real problem and one we need to think about how to resolve. Having said that, scientists are not perfect either. As an author, problems of honesty and reproducibility in science trouble me greatly. That's why on our recent papers (http://www.cl.cam.ac.uk/research/srg/netos/qjump/pubs/2015-nsdi-qjump.pdf http://www.cl.cam.ac.uk/research/srg/netos/qjump/pubs/2015-n... and http://www.cl.cam.ac.uk/research/srg/netos/musketeer/pubs/2015-eurosys-musketeer.pdf http://www.cl.cam.ac.uk/research/srg/netos/musketeer/pubs/20...) my coauthors and I have gone out of our way to make the original unprocessed data sets and detailed experimental descriptions easily available. If you take a look inside of our papers, you'll find that all of the substantial figures in the papers are clickable giving direct access to original datasets on our websites. Although not perfect, this is a start down the road of more easily testable and reproducible work. To our great surprise, only a few weeks after publication we've already got requests from people trying to reproduce the results in different environments. This suggests to me that this kind of approach both can work, and really should become the minimum standard in academic publishing.
- wnevets 11y ago
  > In the rush for funding and "impact" university PR machines (and scientists too) are forced to make flashy statements about the latest new sensational, tenuous at best implications of their findings. in a lot of cases its worse than just wanting funding. A lot of companies are funding "studies" and getting them published so they can sell their latest revolutionary product.
return0 11y ago
We should obviously not embrace that. Science should never be flawed knowingly, if we break this contract, then it's best for the public to withdraw all academic funding earlier than later and seek answers elsewhere. I understand that too many people are working in science nowadays, and they all have to publish something, but that's no excuse to justify "flawed" science.
analog31 11y ago
One humble idea: Include the proposed press release in the peer review process. Maybe require two abstracts, one for the academic journal, and the other for the lay press. And there is no loss of honor from saying that a study is of purely intellectual interest.

38 points by simonsarris 11y ago

This entire article is a criticism of the scientific journal industry, not "science." It is disturbing that the authors appear to have a difficult time compartmentalizing those concepts, but still consider themselves authoritative on the matter.

deleted 11y ago
[deleted]
- deleted 11y ago
  [deleted]

denzil_correa 11y ago

I think the issue stems from a fact that most people overestimate the worth of a peer reviewed article. I like to look at it a bit differently. A research problem is a like a discussion where each peer reviewed article provides one discussion point towards the solution. It by no means is entirely self encompassed but at the same time, it doesn't mean one should discard it. The solution to the research problem isn't entirely contained in a peer reviewed article. Rather, it is one data point towards making an impact towards the larger problem. A collection of research articles is many such discussion points. The final conclusion has to be drawn by collecting all such arguments on the table and synthesizing them.

nosideeffects 11y ago

How about we just ban "journalists" from publishing articles about the results of recently published studies?

6t6t6 11y ago
> ban "journalists" from publishing Maybe not.

hoopd 11y ago

> When people talk about flaws in science, they're often focusing on medical and life sciences, as Horton is. But that might simply be because these fields are furthest along in auditing their own problems. Many of the structural problems in medical science could well apply to other fields, too. Pure FUD; those fields have unusually low levels of empiricism. Rule of thumb: as a science has greater trouble gathering enough data to settle disputes the scientists of that field become more and more like philosophers. They decide what's true based on arguments or personal feelings and in the worst cases they fabricate data to fill the vacuum. The problem is that the core element of science, that of rigorous experimentation, is missing from the process. The question is if a field is only able to follow the aesthetics of science but unable to adopt the rigor to what extent should we consider it science?

nice1 11y ago
You'd think climate studies should have a higher standard; yet, as the climategate materials showed, those people suffer from the same "goal oriented" data fudging practices as described in the article.

bitwize 11y ago

The problem is that admitting science is flawed means that people will propose astrology or alkaline diets or antivaccinationism as alternatives. So yes, let's all embrace that science is flawed and then work together to fix science, and not pretend that Christianity or traditional vitalism are better alternatives to science.

zenogais 11y ago
This kind of black and white thinking is exactly the problem though. Either science or mysticism. Science will never be infallible, and in fact has strict limits on what it can know [1][2] - there's nothing to fix here, except the perception that fetishizes the scientific process as an infallible method for producing universal truths. This black and white perspective fails to recognize social and psychological biases that underlie scientific production and also lie outside its full understanding. The scientific method is a tool. That being said, we should of course strive to improve this tool. The problem here is one of human perception. What goes for science goes for all things - don't believe the hype. [1]: http://undsci.berkeley.edu/article/0_0_0/whatisscience_12 http://undsci.berkeley.edu/article/0_0_0/whatisscience_12 [2]: http://moreintelligentlife.com/content/ideas/anthony-gottlieb/limits-science http://moreintelligentlife.com/content/ideas/anthony-gottlie...
neap24 11y ago
It seems unlikely that admitting science is flawed would immediately cause people to jump to the opposite extreme. Perhaps those who already trumpet those ideas would feel emboldened, but I doubt they would have much influence.
xj9 11y ago
Science is not a religion, lets not compare apples to oranges.
- briantakita 11y ago
  Belief in Science is a religion.
13years 11y ago

deadgrey19 11y ago

Academic publishing is a conversation between academics. No one paper is truly authoritative and only with time will the highly cited well regarded contributions make themselves known within the academic community (and eventually outside of it too). The time horizon between publication and "impact" for most academic work is usually (or should be) in the range of 10-20years. In the rush for funding and "impact" university PR machines (and scientists too) are forced to make flashy statements about the latest new sensational, tenuous at best implications of their findings. Boring, rigorous, incremental work has no place in a media driven funding cycle. This is a real problem and one we need to think about how to resolve. Having said that, scientists are not perfect either. As an author, problems of honesty and reproducibility in science trouble me greatly. That's why on our recent papers (http://www.cl.cam.ac.uk/research/srg/netos/qjump/pubs/2015-nsdi-qjump.pdf http://www.cl.cam.ac.uk/research/srg/netos/qjump/pubs/2015-n... and http://www.cl.cam.ac.uk/research/srg/netos/musketeer/pubs/2015-eurosys-musketeer.pdf http://www.cl.cam.ac.uk/research/srg/netos/musketeer/pubs/20...) my coauthors and I have gone out of our way to make the original unprocessed data sets and detailed experimental descriptions easily available. If you take a look inside of our papers, you'll find that all of the substantial figures in the papers are clickable giving direct access to original datasets on our websites. Although not perfect, this is a start down the road of more easily testable and reproducible work. To our great surprise, only a few weeks after publication we've already got requests from people trying to reproduce the results in different environments. This suggests to me that this kind of approach both can work, and really should become the minimum standard in academic publishing.

wnevets 11y ago
> In the rush for funding and "impact" university PR machines (and scientists too) are forced to make flashy statements about the latest new sensational, tenuous at best implications of their findings. in a lot of cases its worse than just wanting funding. A lot of companies are funding "studies" and getting them published so they can sell their latest revolutionary product.

return0 11y ago

We should obviously not embrace that. Science should never be flawed knowingly, if we break this contract, then it's best for the public to withdraw all academic funding earlier than later and seek answers elsewhere. I understand that too many people are working in science nowadays, and they all have to publish something, but that's no excuse to justify "flawed" science.

analog31 11y ago

One humble idea: Include the proposed press release in the peer review process. Maybe require two abstracts, one for the academic journal, and the other for the lay press. And there is no loss of honor from saying that a study is of purely intellectual interest.