3 ms·

And what happens when a new fact-checking algorithm comes out? What I said in my comment.

by wyager 10y ago

dahart 10y ago
Only if they're also rude about it and run it on a huge set of old papers, and do a big PR campaign to get attention. Otherwise, the only thing that happens is papers quietly get better and everyone on all sides is happy.
- int_19h 10y ago
  So they shouldn't run it on the old papers, so as to not offend anyone's sensibilities? Do we just declare said papers completely useless then? Or do they keep getting cited by new papers, and used to guide policy? If the latter, then not vetting them using the newer and better methodology would be unethical.
  - dahart 10y ago
    That would be jumping to a conclusion I didn't state. I think I already address this comment above, I don't think what they did was wrong, and I don't care if people were offended. But, since you brought it up - old papers are already dead, they cannot be fixed. They can only be referenced as prior work, or retracted in extreme cases. We're not talking about extreme cases here. Statcheck can only help new papers, it cannot help old papers. Running it on old papers was done as a publicity stunt for statcheck, and nothing more. The authors said so.
  - dahart 10y ago
    > If the latter, then not vetting them using the newer and better methodology would be unethical. On the contrary, it would be unethical to hold people to new standards that didn't exist when the work was done. If you have a beer and then next week the laws change the drinking age to 65 years old, should you go to jail? This is simply not how things are done, in science or in society. Laws and standards change all the time, you are only subject to the laws or standards that exist when the work or action you do is performed & evaluated. For the purposes of scientific publication, we do not and will never revisit all prior work and formally re-judge whenever standards or policies change. You may be conflating publication policy with general scientific understanding. Old papers will always be informally evaluated under the thinking of the day. But that doesn't help the old papers, nothing can be done about the old papers, they are part of a fixed record that can't change, it only allows us to publish new papers. What will and does happen now is new papers will be published that refute old papers. The new papers are subject to the new methodologies.
    - int_19h 10y ago
      You're not holding people to new standards. You're holding their research to new standards. Which is the only sensible course of action. And no-one suggested that those papers should be rewritten. But insofar as there's some known problem with them, why is it a bad thing to have a public record stating that much?
      - dahart 10y ago
        We may be agreeing and mis-communicating, or agreeing violently, as I've heard it called, so let's get specific. What are you suggesting should happen to a paper when it's shown to have errors? Nothing at all is wrong with adding new information to the public record stating the issues, that's what I mentioned is already happening -- new papers reference and demonstrate the weaknesses of old papers. In my field, as I suspect most, it's a time honored tradition to do a small literature review in the introduction section of a paper that mainly dismisses all related previous work as not good enough to solve the problem you're about to blow everyone's mind's with. In my mind, nothing is wrong with what the statcheck authors did either. My one and only point at the top was that it's not surprising it ruffled some feathers, and that it didn't have to ruffle any feathers. That only happened because the results were made public without solicitation. @wyager was trying to paint the situation as a dichotomy between rude or unscientific, that rude was the only option. Rude is not the only option. If statcheck hadn't published the review of old papers and contacted all the old authors, then I'm pretty sure two things would have happened: 1- this wouldn't have ruffled any feathers, and 2- it wouldn't have gotten much attention, and we wouldn't be talking about it.
    - Cogito 10y ago
      I don't get this argument. There is no new standard here. This is just a tool that says, "is there anything that looks like a mathematical/statistical mistake". The expectation that a paper's calculations are correct and error free is one that has always existed. There is no "thinking of the day", there is just mathematical correctness or not. At most you could argue that we might learn that some thing we thought mathematically true is no longer so, and warrants us reviewing papers where it had previously been used, but that is not the case here. It is not unreasonable to hold published research to the standard of correctness, and if a paper contains errors within its calculations these should be fixed - regardless of when or who published it.

And what happens when a new fact-checking algorithm comes out? What I said in my comment.

by wyager 10y ago

And what happens when a new fact-checking algorithm comes out? What I said in my comment.

Only if they're also rude about it and run it on a huge set of old papers, and do a big PR campaign to get attention. Otherwise, the only thing that happens is papers quietly get better and everyone on all sides is happy.

int_19h 10y ago
So they shouldn't run it on the old papers, so as to not offend anyone's sensibilities? Do we just declare said papers completely useless then? Or do they keep getting cited by new papers, and used to guide policy? If the latter, then not vetting them using the newer and better methodology would be unethical.
- dahart 10y ago
  That would be jumping to a conclusion I didn't state. I think I already address this comment above, I don't think what they did was wrong, and I don't care if people were offended. But, since you brought it up - old papers are already dead, they cannot be fixed. They can only be referenced as prior work, or retracted in extreme cases. We're not talking about extreme cases here. Statcheck can only help new papers, it cannot help old papers. Running it on old papers was done as a publicity stunt for statcheck, and nothing more. The authors said so.
- dahart 10y ago
  > If the latter, then not vetting them using the newer and better methodology would be unethical. On the contrary, it would be unethical to hold people to new standards that didn't exist when the work was done. If you have a beer and then next week the laws change the drinking age to 65 years old, should you go to jail? This is simply not how things are done, in science or in society. Laws and standards change all the time, you are only subject to the laws or standards that exist when the work or action you do is performed & evaluated. For the purposes of scientific publication, we do not and will never revisit all prior work and formally re-judge whenever standards or policies change. You may be conflating publication policy with general scientific understanding. Old papers will always be informally evaluated under the thinking of the day. But that doesn't help the old papers, nothing can be done about the old papers, they are part of a fixed record that can't change, it only allows us to publish new papers. What will and does happen now is new papers will be published that refute old papers. The new papers are subject to the new methodologies.
  - int_19h 10y ago
    You're not holding people to new standards. You're holding their research to new standards. Which is the only sensible course of action. And no-one suggested that those papers should be rewritten. But insofar as there's some known problem with them, why is it a bad thing to have a public record stating that much?
    - dahart 10y ago
      We may be agreeing and mis-communicating, or agreeing violently, as I've heard it called, so let's get specific. What are you suggesting should happen to a paper when it's shown to have errors? Nothing at all is wrong with adding new information to the public record stating the issues, that's what I mentioned is already happening -- new papers reference and demonstrate the weaknesses of old papers. In my field, as I suspect most, it's a time honored tradition to do a small literature review in the introduction section of a paper that mainly dismisses all related previous work as not good enough to solve the problem you're about to blow everyone's mind's with. In my mind, nothing is wrong with what the statcheck authors did either. My one and only point at the top was that it's not surprising it ruffled some feathers, and that it didn't have to ruffle any feathers. That only happened because the results were made public without solicitation. @wyager was trying to paint the situation as a dichotomy between rude or unscientific, that rude was the only option. Rude is not the only option. If statcheck hadn't published the review of old papers and contacted all the old authors, then I'm pretty sure two things would have happened: 1- this wouldn't have ruffled any feathers, and 2- it wouldn't have gotten much attention, and we wouldn't be talking about it.
  - Cogito 10y ago
    I don't get this argument. There is no new standard here. This is just a tool that says, "is there anything that looks like a mathematical/statistical mistake". The expectation that a paper's calculations are correct and error free is one that has always existed. There is no "thinking of the day", there is just mathematical correctness or not. At most you could argue that we might learn that some thing we thought mathematically true is no longer so, and warrants us reviewing papers where it had previously been used, but that is not the case here. It is not unreasonable to hold published research to the standard of correctness, and if a paper contains errors within its calculations these should be fixed - regardless of when or who published it.