4 ms·

Maybe if academics were engaged in real work that can't be faked this wouldn't be a problem. For instance, why not only accept papers that come along with cold

Maybe if academics were engaged in real work that can't be faked this wouldn't be a problem. For instance, why not only accept papers that come along with cold, hard evidence for what the paper is saying? If it's a chemistry paper there should be a video of all the lab work and the results being shown. If it's a computer science paper then have a runnable docker image of the program. If it's a linguistics paper then have a program that includes the corpus search and the searches you performed along with your statistical analysis files. So long as academics don't have to prove they did anything, there will be fakers and posers.

denom 12y ago
If no one is going to review the paper, no one is going to examine the additional artifacts that "prove" science was being done.
whyaduck 12y ago
The attached article was discussing open, "pay to publish" journals that don't care about the contents of submitted papers. They will publish just about anything that sounds vaguely scientific as long as they get paid. The purpose of peer reviewed journals is to disseminate scientific research that's been vetted so that readers don't have to attempt to reproduce the results to have some comfort level with the paper. That's not to say that published articles in quality journals shouldn't be approached with some degree of skepticism, but the sheer volume of science publishing precludes combing through the data of every article you read. High quality papers will include enough information to reproduce any results referenced in the paper - video is no substitute for actually running the experiment.
tikhonj 12y ago
Good venues do: that's a big part of peer review in any decent journal or conference. While reproducibility could be improved, it's not nearly as large a problem as what the article is talking about: publication venues that don't even try to vet incoming papers. These are places that publish papers without even reading the title. You could submit obvious nonsense—randomly generated gibberish—and get published. You can't do that in respected venues in most fields, especially in science. But for people outside of the field, it's really hard to tell which venues are legitimate and which ones aren't, and incentives are misaligned so that quantity of publishing matters far more than anything else.
- sseveran 12y ago
  I would submit that these venues are not doing an adequate job. The Nature case and the Duke cancer case are examples of how even the most reputable journals can be scammed with fabricated data. We weren't even talking about questionable practices with subtle problems. They were outright fabrications. The most damning comment is near the end where 2% admit to fabricating and 30% to questionable practices. And that was just the self reported results.
EnochSquareRoot 12y ago
What is "real work that can't be faked"? I think it much better to define clear criteria by which a journal should accept research. Further, I wouldn't single-out academics when there are corporate funded studies which cherry-pick results and data to support their products/services.
tjradcliffe 12y ago
Since outside of math the goal of science is not proof but plausibility, this would not be an effective approach. Science is the discipline of publicly testing ideas by systematic observation, controlled experiment and Bayesian inference, and as such is aimed at changing the posterior plausibility of some proposition, not "proving" anything. Proof and certainty are the Alchemist's Stone: philosophers sought after them for thousands of years the way alchemists sought after the secret of turning base metals into gold, never realizing that the fundamental problem wasn't in their methods (although their methods had problems) but in their goal, which was impossible and wrong. We should seek knowledge, not certainty. The range of means by which that can be done is huge (though considerably less than "anything goes") and there is a definite role for work that is exploratory and speculative, up to and including stuff that is almost certainly wrong but worth publishing because a) the error is not obvious and b) publishing creates the opportunity for others to respond to it, hopefully putting the error to rest for good and all. Some of the early work on the "no cloning" theorem was motivated by publications of this type (it turns out if you could clone a quantum state you could use entanglement to communicate faster than light, and there was a series of papers in Physics Letters in the late '80's proposing to do just that.) So trusting in the self-correcting ability of the discipline of science is fundamental to its progress, and therefore insisting on some alchemical standard of "proof" as the goal for publication would fatally cripple the scientific enterprise. For science to work we have to be tolerant of the publication of error. But we need to keep the rate of erroneous publications down to a manageable level. Peer review and society membership were ways of doing this in the past. They have broken down today, and we are still casting about for new ways to keep the error rate well above zero, but not so high as to swamp everything else.
- semi-extrinsic 12y ago
  I think the OP is saying that you should prove that you actually did the science, not implying you have to prove that you are correct.

denom 12y ago

If no one is going to review the paper, no one is going to examine the additional artifacts that "prove" science was being done.

whyaduck 12y ago

The attached article was discussing open, "pay to publish" journals that don't care about the contents of submitted papers. They will publish just about anything that sounds vaguely scientific as long as they get paid. The purpose of peer reviewed journals is to disseminate scientific research that's been vetted so that readers don't have to attempt to reproduce the results to have some comfort level with the paper. That's not to say that published articles in quality journals shouldn't be approached with some degree of skepticism, but the sheer volume of science publishing precludes combing through the data of every article you read. High quality papers will include enough information to reproduce any results referenced in the paper - video is no substitute for actually running the experiment.

tikhonj 12y ago

Good venues do: that's a big part of peer review in any decent journal or conference. While reproducibility could be improved, it's not nearly as large a problem as what the article is talking about: publication venues that don't even try to vet incoming papers. These are places that publish papers without even reading the title. You could submit obvious nonsense—randomly generated gibberish—and get published. You can't do that in respected venues in most fields, especially in science. But for people outside of the field, it's really hard to tell which venues are legitimate and which ones aren't, and incentives are misaligned so that quantity of publishing matters far more than anything else.

sseveran 12y ago
I would submit that these venues are not doing an adequate job. The Nature case and the Duke cancer case are examples of how even the most reputable journals can be scammed with fabricated data. We weren't even talking about questionable practices with subtle problems. They were outright fabrications. The most damning comment is near the end where 2% admit to fabricating and 30% to questionable practices. And that was just the self reported results.

EnochSquareRoot 12y ago

What is "real work that can't be faked"? I think it much better to define clear criteria by which a journal should accept research. Further, I wouldn't single-out academics when there are corporate funded studies which cherry-pick results and data to support their products/services.

tjradcliffe 12y ago

Since outside of math the goal of science is not proof but plausibility, this would not be an effective approach. Science is the discipline of publicly testing ideas by systematic observation, controlled experiment and Bayesian inference, and as such is aimed at changing the posterior plausibility of some proposition, not "proving" anything. Proof and certainty are the Alchemist's Stone: philosophers sought after them for thousands of years the way alchemists sought after the secret of turning base metals into gold, never realizing that the fundamental problem wasn't in their methods (although their methods had problems) but in their goal, which was impossible and wrong. We should seek knowledge, not certainty. The range of means by which that can be done is huge (though considerably less than "anything goes") and there is a definite role for work that is exploratory and speculative, up to and including stuff that is almost certainly wrong but worth publishing because a) the error is not obvious and b) publishing creates the opportunity for others to respond to it, hopefully putting the error to rest for good and all. Some of the early work on the "no cloning" theorem was motivated by publications of this type (it turns out if you could clone a quantum state you could use entanglement to communicate faster than light, and there was a series of papers in Physics Letters in the late '80's proposing to do just that.) So trusting in the self-correcting ability of the discipline of science is fundamental to its progress, and therefore insisting on some alchemical standard of "proof" as the goal for publication would fatally cripple the scientific enterprise. For science to work we have to be tolerant of the publication of error. But we need to keep the rate of erroneous publications down to a manageable level. Peer review and society membership were ways of doing this in the past. They have broken down today, and we are still casting about for new ways to keep the error rate well above zero, but not so high as to swamp everything else.

semi-extrinsic 12y ago
I think the OP is saying that you should prove that you actually did the science, not implying you have to prove that you are correct.