4 ms·

This is an incredibly important problem that Ioannidis is working on. Not only could it increase our confidence in scientific results, it can help institutional

This is an incredibly important problem that Ioannidis is working on. Not only could it increase our confidence in scientific results, it can help institutionalize a bunch of practices (reproducibility, publishing negative results, post-publication review) that we know to be good ideas. But it might be worth it to take a step back and recognize what a brilliant process science is. After all, even with such waste, such bias, and such low rates of reproduction, science still gives us amazing insights into the world, enriches our understanding of the universe and our place in it, and of course births incredible technology. Through fraud exists, though scientists are flawed humans, though funding agencies can be so conservative or wrong-headed—still, science chugs along. It's so robust, so powerful, and yet so simple! It's truly amazing to live in our world.

return0 13y ago
The problem is that science for many scientists is the business of persuading reviewers and editors, publishing positive results on a regular basis and spending the rest of the time writing grant proposals. This has to do with the publishing process itself, and the poor peer review (closed, sloppy, even for "respectable" journals) of the current publishing system. Science nowadays generates large amounts of knowledge, not all of which is useful, but we lack even rudimentary tools to catalog and analyze that information. Very few people have suggested ways to catalog science in ways that are useful to inform future research[1,2]. [1] http://www.kurzweilai.net/new-tools-to-manage-information-overload-threatening-neuroscience http://www.kurzweilai.net/new-tools-to-manage-information-ov... [2] http://www.silvalab.com.cnchost.com/silvapapers/S2Neuron2013.pdf http://www.silvalab.com.cnchost.com/silvapapers/S2Neuron2013...
- waltherg 13y ago
  Amen. In my experience academic research is very different from what the wider public thinks it is. You sometimes ask yourself: Would the public continue throwing their taxes at scientific research if they knew that most of their cash is just consumed to fuel the egos of professors?
  - mathattack 13y ago
    This is true above and beyond just science. I get the need for research. I get the idea of pushing the boundary of knowledge, checked by peer review. The current system just seems very broken. People respond well to incentives, and they're incented to publish novel results no matter what. All this on the back of a lot of grad students, most of which whom will struggle to get jobs.
- fnl 13y ago
  I could not agree more to your description of what the life after your PhD seems to be. The only other workday activities you are missing out are probably (a) travelling, meetings, and conferences [lots] and (b) reading papers [hundreds/year]. However, I fail to see how "research maps" would improve our way to catalog and analyze information. What is shown in your ref [1] is a regulatory network. These models have been used for years to drive research into diseases ranging from cancer to neurodegenerative diseases. And, there is (at least in the biomedical sciences) a very vibrant community dedicated to that very job, the "bio-curators" [http://www.biocurator.org/ http://www.biocurator.org/] that directly or indirectly fill (relational) bio-repositories. "Research maps" like STRING, Reactome, PIR, BioCyc, or KEGG, to name a few, have been around for quite a while, too. So I think the claim in [1] that Silva has "invented" "research maps" is a quite some PR exaggeration, as this can be simply described as a "Reactome limited to Neuroscientists only".
  - return0 13y ago
    Agree it's not a better solution, it's just that there is little talk of it in general. Paywalled science plays a big role here, it basically makes it impossible to gather even paper titles for further analysis.
- pron 13y ago
  I would propose an even more "meta" categorization. Research should be categorized into 'A' level and 'B' level. B level would mean that the purpose of this research is to explore an hypothesis and suggest directions for further research, while A level would mean an attempt at conclusive (or as close to conclusive as possible) results after enough B level research has been done in a certain field. B research would encompass what we suspect and/or want to examine; A research would encompass what we claim to know.
spfoos 13y ago
Public policy based on such science impacts millions of people. The lipid hypothesis is a classic example.
pessimizer 13y ago
>After all, even with such waste, such bias, and such low rates of reproduction, science still gives us amazing insights into the world, enriches our understanding of the universe and our place in it Not most science, it looks like. Most science gives us false insights into the world, reenforcing our prejudices and the sales of dubious products. >and of course births incredible technology. Or at least it once did. We're at the end of riding Moore's law, which confused us into thinking that science was progressing, when it was just transistor density that was progressing.
Retric 13y ago
It's an incentive problem. People tend to game any system based on how there measured. And PHD's are generally measured by how much they publish as well as how quickly they publish. Take two academics that publish the same finding 3 months apart and it's the guy who published first who becomes famous. But, there are a lot more PHD's out there than famous ones so for the most part it's #of papers that's most important. Ideally you want a lot of incremental research(easy), using other peoples data (cheap), and showing something novel(fame lottery). While most of this is practically useless it still gets published because there are plenty of Journals and they all need to publish something every month. That's not to say people are not still doing high quality research but rather the signal to noise ratio often low for a reason.
- mathattack 13y ago
  How do you fix this, assuming you're constrained by the same amount of resources? My sense is the end game should be encouraging long term research where there is a decent enough chance for success relative to the gains from it - the chances are hard to determine but somewhat based on the idea, and the team chasing it. Essentially this is a VC model for ideas that are too far out to generate attractive returns.
  - Retric 13y ago
    #1 is probably require Journals to agree and publish research before the results are known based on the experimental design and require scientists to include there raw data. Preferably as they collect it. This is to counter: http://xkcd.com/882/ http://xkcd.com/882/ Require Journals to Publish the first 2 studies that try and reproduce a given result. Note: Publish need not mean include in the paper copy's just indexed and searchable online. This let's you create a reproducibility index which might not mean much until they have done a fair bit of research but still something there going to keep in mind. Funding agency's would need to set aside a large amount to validate any study. However, as there is a lot of null results that the percentage is probably not all that high. More importantly as not doing so means the results are practically worthless in the first place it's still a good idea even if it takes 1/2 the overall budget.

return0 13y ago

The problem is that science for many scientists is the business of persuading reviewers and editors, publishing positive results on a regular basis and spending the rest of the time writing grant proposals. This has to do with the publishing process itself, and the poor peer review (closed, sloppy, even for "respectable" journals) of the current publishing system. Science nowadays generates large amounts of knowledge, not all of which is useful, but we lack even rudimentary tools to catalog and analyze that information. Very few people have suggested ways to catalog science in ways that are useful to inform future research[1,2]. [1] http://www.kurzweilai.net/new-tools-to-manage-information-overload-threatening-neuroscience http://www.kurzweilai.net/new-tools-to-manage-information-ov... [2] http://www.silvalab.com.cnchost.com/silvapapers/S2Neuron2013.pdf http://www.silvalab.com.cnchost.com/silvapapers/S2Neuron2013...

waltherg 13y ago
Amen. In my experience academic research is very different from what the wider public thinks it is. You sometimes ask yourself: Would the public continue throwing their taxes at scientific research if they knew that most of their cash is just consumed to fuel the egos of professors?
- mathattack 13y ago
  This is true above and beyond just science. I get the need for research. I get the idea of pushing the boundary of knowledge, checked by peer review. The current system just seems very broken. People respond well to incentives, and they're incented to publish novel results no matter what. All this on the back of a lot of grad students, most of which whom will struggle to get jobs.
fnl 13y ago
I could not agree more to your description of what the life after your PhD seems to be. The only other workday activities you are missing out are probably (a) travelling, meetings, and conferences [lots] and (b) reading papers [hundreds/year]. However, I fail to see how "research maps" would improve our way to catalog and analyze information. What is shown in your ref [1] is a regulatory network. These models have been used for years to drive research into diseases ranging from cancer to neurodegenerative diseases. And, there is (at least in the biomedical sciences) a very vibrant community dedicated to that very job, the "bio-curators" [http://www.biocurator.org/ http://www.biocurator.org/] that directly or indirectly fill (relational) bio-repositories. "Research maps" like STRING, Reactome, PIR, BioCyc, or KEGG, to name a few, have been around for quite a while, too. So I think the claim in [1] that Silva has "invented" "research maps" is a quite some PR exaggeration, as this can be simply described as a "Reactome limited to Neuroscientists only".
- return0 13y ago
  Agree it's not a better solution, it's just that there is little talk of it in general. Paywalled science plays a big role here, it basically makes it impossible to gather even paper titles for further analysis.
pron 13y ago
I would propose an even more "meta" categorization. Research should be categorized into 'A' level and 'B' level. B level would mean that the purpose of this research is to explore an hypothesis and suggest directions for further research, while A level would mean an attempt at conclusive (or as close to conclusive as possible) results after enough B level research has been done in a certain field. B research would encompass what we suspect and/or want to examine; A research would encompass what we claim to know.

spfoos 13y ago

Public policy based on such science impacts millions of people. The lipid hypothesis is a classic example.

pessimizer 13y ago

>After all, even with such waste, such bias, and such low rates of reproduction, science still gives us amazing insights into the world, enriches our understanding of the universe and our place in it Not most science, it looks like. Most science gives us false insights into the world, reenforcing our prejudices and the sales of dubious products. >and of course births incredible technology. Or at least it once did. We're at the end of riding Moore's law, which confused us into thinking that science was progressing, when it was just transistor density that was progressing.

Retric 13y ago

It's an incentive problem. People tend to game any system based on how there measured. And PHD's are generally measured by how much they publish as well as how quickly they publish. Take two academics that publish the same finding 3 months apart and it's the guy who published first who becomes famous. But, there are a lot more PHD's out there than famous ones so for the most part it's #of papers that's most important. Ideally you want a lot of incremental research(easy), using other peoples data (cheap), and showing something novel(fame lottery). While most of this is practically useless it still gets published because there are plenty of Journals and they all need to publish something every month. That's not to say people are not still doing high quality research but rather the signal to noise ratio often low for a reason.

mathattack 13y ago
How do you fix this, assuming you're constrained by the same amount of resources? My sense is the end game should be encouraging long term research where there is a decent enough chance for success relative to the gains from it - the chances are hard to determine but somewhat based on the idea, and the team chasing it. Essentially this is a VC model for ideas that are too far out to generate attractive returns.
- Retric 13y ago
  #1 is probably require Journals to agree and publish research before the results are known based on the experimental design and require scientists to include there raw data. Preferably as they collect it. This is to counter: http://xkcd.com/882/ http://xkcd.com/882/ Require Journals to Publish the first 2 studies that try and reproduce a given result. Note: Publish need not mean include in the paper copy's just indexed and searchable online. This let's you create a reproducibility index which might not mean much until they have done a fair bit of research but still something there going to keep in mind. Funding agency's would need to set aside a large amount to validate any study. However, as there is a lot of null results that the percentage is probably not all that high. More importantly as not doing so means the results are practically worthless in the first place it's still a good idea even if it takes 1/2 the overall budget.