5 ms·

Note well. This is for clinical research. This says nothing about C.S., Math, Physics, etc... It's harder to publish a big result in Physics, for example, and

Note well. This is for clinical research. This says nothing about C.S., Math, Physics, etc... It's harder to publish a big result in Physics, for example, and have it stand for long without being questioned. Just ask Jan Hendrik Schön.

johndcook 16y ago
Math and CS are better off than experimental sciences. Theoretical paper often contain errors, but these don't usually invalidate the conclusions. In experimental sciences, hard sciences like physics do have a lower error rate than sciences that rely heavily on statistics. But that's changing. See the second page of this article: http://www-stat.stanford.edu/~ckirby/brad/papers/2005NEWModernScience.pdf http://www-stat.stanford.edu/~ckirby/brad/papers/2005NEWMode... The author argues that "Nature has gotten more tight-fisted with modern physicists" and that physics is now more dependent on statistics.
logjam 16y ago
> Note well. This is for clinical research. My own experience is that there is much constant questioning about published clinical research. In addition, decision-making of clinicians based on clinical research is tempered with other considerations, sometimes quite complex. Stephen Paulker commented in response to the original paper: "From the perspective of an epidemiologist or a statistician, the relevant question is whether the study's hypothesis is true - i.e., is the probability of H1 greater than 0.5. For clinicians and their patients, the relevant question is whether a particular strategy should be followed in an individual patient or a subset of similar patients. That decision (or recommendation to the patient) will depend on the pre-study likelihood of benefit in that patient and on the relative magnitude of benefits and risks of that strategy, if the diagnosis in that patient is uncertain. For many such decisions, the "more likely true than false" criterion may not be the best decision rule. For serious diseases and treatments of only modest risk, post-study probabilities of considerably less than 0.5 may be sufficient to justify treatment."
Lewisham 16y ago
From what I've seen, I wouldn't say CS is immune to this at all. CS seems to only be accepting papers that illustrate some big win. In my short PhD career, I've an article in the drawer where the result was inconclusive. This result, in and of itself, was fairly valuable. It spoke much about all the ways things went wrong. However, wrong was not acceptable for publication, it had to be right. Much hand-wringing was done to identify the bits that could be pulled out from it, polished, and then submitted (I was not especially pleased about this). The paper was, rightly, rejected, but all the valuable findings are still in my drawer. The pressure to produce Good findings rather than Bad findings does not meet my expectation of scientific pursuit. I took great joy in seeing a history of science exhibition recently, and reading the notebooks of people like Newton and Darwin, who they themselves spend a lot of time writing out all the things that went wrong, all the concerns they have about their "results", and general humbleness. It speaks volumes that the greatest scientific minds that have ever lived have put more caveats and concerns in their published work than 98% of what is published today does. To get published, it seems you have to be 110% sure that your work is 120% amazing. I don't think this is a good state of affairs.

johndcook 16y ago

Math and CS are better off than experimental sciences. Theoretical paper often contain errors, but these don't usually invalidate the conclusions. In experimental sciences, hard sciences like physics do have a lower error rate than sciences that rely heavily on statistics. But that's changing. See the second page of this article: http://www-stat.stanford.edu/~ckirby/brad/papers/2005NEWModernScience.pdf http://www-stat.stanford.edu/~ckirby/brad/papers/2005NEWMode... The author argues that "Nature has gotten more tight-fisted with modern physicists" and that physics is now more dependent on statistics.

logjam 16y ago

> Note well. This is for clinical research. My own experience is that there is much constant questioning about published clinical research. In addition, decision-making of clinicians based on clinical research is tempered with other considerations, sometimes quite complex. Stephen Paulker commented in response to the original paper: "From the perspective of an epidemiologist or a statistician, the relevant question is whether the study's hypothesis is true - i.e., is the probability of H1 greater than 0.5. For clinicians and their patients, the relevant question is whether a particular strategy should be followed in an individual patient or a subset of similar patients. That decision (or recommendation to the patient) will depend on the pre-study likelihood of benefit in that patient and on the relative magnitude of benefits and risks of that strategy, if the diagnosis in that patient is uncertain. For many such decisions, the "more likely true than false" criterion may not be the best decision rule. For serious diseases and treatments of only modest risk, post-study probabilities of considerably less than 0.5 may be sufficient to justify treatment."

Lewisham 16y ago

From what I've seen, I wouldn't say CS is immune to this at all. CS seems to only be accepting papers that illustrate some big win. In my short PhD career, I've an article in the drawer where the result was inconclusive. This result, in and of itself, was fairly valuable. It spoke much about all the ways things went wrong. However, wrong was not acceptable for publication, it had to be right. Much hand-wringing was done to identify the bits that could be pulled out from it, polished, and then submitted (I was not especially pleased about this). The paper was, rightly, rejected, but all the valuable findings are still in my drawer. The pressure to produce Good findings rather than Bad findings does not meet my expectation of scientific pursuit. I took great joy in seeing a history of science exhibition recently, and reading the notebooks of people like Newton and Darwin, who they themselves spend a lot of time writing out all the things that went wrong, all the concerns they have about their "results", and general humbleness. It speaks volumes that the greatest scientific minds that have ever lived have put more caveats and concerns in their published work than 98% of what is published today does. To get published, it seems you have to be 110% sure that your work is 120% amazing. I don't think this is a good state of affairs.