4 ms·

> My pet idea for making science less wrong is to maintain a citation graph of all papers. Then when a problem is found with any paper, every other one downstre

> My pet idea for making science less wrong is to maintain a citation graph of all papers. Then when a problem is found with any paper, every other one downstream from it is automatically flagged as at risk of being wrong. Now all those authors (or others) have to go back and re-evaluate how that citation was used and how it affects their result. Once they decide it's OK, they update their paper and remove the flag. If it's not OK then their paper is retracted or flagged as wrong or simply keeps its at-risk flag if it's not important enough for anybody to bother re-checking it. This is an interesting idea, but I don't know what real-world problem it would actually solve. As someone who reads a lot of scientific papers, I can't think of a single one whose correctness is actually dependent on the correctness of the results of a paper it cites, as though the paper in question is "buggy" because it "calls another paper as a subroutine" and that subroutine is buggy. Could you give an example or two of this? I could see this making sense in mathematics since mathematical proofs really do "call other papers as subroutines" (theorems), but not so much in science.

nonbel 9y ago
>"As someone who reads a lot of scientific papers, I can't think of a single one whose correctness is actually dependent on the correctness of the results of a paper it cites" What field are you reading where each paper is independent of all the others? As the most obvious example, in biomed pretty much every paper says something like "as done previously[ref x, y]" in the methods section. It is an extremely annoying, but very common, practice (usually refs x and y contain somewhat different methods descriptions and/or also contain the "as done previously" phrase). There are many other ways the papers depend on each other too ("we used these cells because they don't express this protein[ref z]", etc)
- peteretep 9y ago
  > > whose correctness is actually > > dependent on the correctness of the > > results of *a* paper it cites > where each paper is independent of > *all* the others These are not describing the same thing
  - nonbel 9y ago
    I think they are, unless you are selectively citing (irrelevant) papers to avoid depending on any...
  - mekarpeles 9y ago
    If I understand, the author is suggesting something like pagerank for axiomatic correctness where a paper's score can be interpreted per paper (in terms of some binary promise of "correctness") even if the score is computed by the weighted correctness of the papers it represents. Really, the information should be abstracted away from the paper, a la freebase or wikidata, and there should be strict rules (e.g. the protocols stackoverflow enforces) such that no cycles can occur, such that propositions can be resolved to axioms, and such that it is practical to compute scores in a finite period of time with reasonable resource constraints... See also gitxiv.org (at attempt at literate research for computer science and physics papers that can be paired with code). It doesn't address the problem described, but if statements of a paper are codified (e.g. curry howard isomorphism) in a strict way, it makes it easier for "linked statements" to have a quantifiable value.
- throwaway287391 9y ago
  I more or less exclusively read CS papers, which sometimes contain similar statements. I don't see how in your example, "ref x" being invalid would make the paper citing it invalid. Can you give an actual example of a paper that does this that could be invalidated in this way? Basically, to me, an empirical result arrived at using the scientific method is, well, an empirical result arrived at using the scientific method. And it seems to me that the only thing that would invalidate such a result is if it were found that the researchers did not follow the protocol they described in their paper. The fact that the researchers might have merely cited another paper, or used the method described in another paper, in which the researchers were found to have "cheated" in this way, doesn't seem to invalidate the paper in question. It's possible that the researchers in question could have happened to also cheat, but I don't see any reason to assume it's any more likely because it happened with a paper they cited (unless of course they're the same researchers building on their own previous work).
  - nonbel 9y ago
    For the methods citation example, the citation chain (is supposed to) go all the way back to the paper devoted to originally describing the method. In that paper there (is supposed to be) a bunch of checks (compare to other methods, etc) that the method is working as claimed. These checks will not be repeated every time.
GolDDranks 9y ago
I mainly read papers of second language acquisition which is an empirical science that performs language learning experiments. Often the "inspiration" for the research in newer papers stems from the results of earlier ones. The empirical results of the newer ones are, of course, not directly invalidated if there was a problem with an earlier one. However, some of the papers tend to discuss the results in the light of the current empirical results AND the earlier empirical results ("from this result, we now see that A → B, and from the earlier results we already know that B → C, so there's some evidence that A → C"); this of course biases the discussion and may lead to wrong conclusions. By the way, the largest problems I see in the papers are not directly "wrong" results, but like the "Chinese whispers" or "broken telephone" game. The paper A makes some assumptions and uses certain test protocol, measurements etc, and then paper B refers the result but misrepresents it slightly, which may cause overgeneralisation. A concrete example: paper A tests the learning of some phoneme with a training period and before-after test and publishes the result. Then the paper B later references the paper A mentioning that A has shown that the learners "learn" the phoneme. However, represented this way – "unqualified" – this is an overgeneralisation: the students haven't actually shown to have learned the phoneme in any other way than using the test. Indeed – other studies have shown that using certain kind of tests are generally very poorly correlated with what people usually mean by "having learnt" something – being able to use it in spontaneous communication.

nonbel 9y ago

>"As someone who reads a lot of scientific papers, I can't think of a single one whose correctness is actually dependent on the correctness of the results of a paper it cites" What field are you reading where each paper is independent of all the others? As the most obvious example, in biomed pretty much every paper says something like "as done previously[ref x, y]" in the methods section. It is an extremely annoying, but very common, practice (usually refs x and y contain somewhat different methods descriptions and/or also contain the "as done previously" phrase). There are many other ways the papers depend on each other too ("we used these cells because they don't express this protein[ref z]", etc)

peteretep 9y ago
> > whose correctness is actually > > dependent on the correctness of the > > results of *a* paper it cites > where each paper is independent of > *all* the others These are not describing the same thing
- nonbel 9y ago
  I think they are, unless you are selectively citing (irrelevant) papers to avoid depending on any...
- mekarpeles 9y ago
  If I understand, the author is suggesting something like pagerank for axiomatic correctness where a paper's score can be interpreted per paper (in terms of some binary promise of "correctness") even if the score is computed by the weighted correctness of the papers it represents. Really, the information should be abstracted away from the paper, a la freebase or wikidata, and there should be strict rules (e.g. the protocols stackoverflow enforces) such that no cycles can occur, such that propositions can be resolved to axioms, and such that it is practical to compute scores in a finite period of time with reasonable resource constraints... See also gitxiv.org (at attempt at literate research for computer science and physics papers that can be paired with code). It doesn't address the problem described, but if statements of a paper are codified (e.g. curry howard isomorphism) in a strict way, it makes it easier for "linked statements" to have a quantifiable value.
throwaway287391 9y ago
I more or less exclusively read CS papers, which sometimes contain similar statements. I don't see how in your example, "ref x" being invalid would make the paper citing it invalid. Can you give an actual example of a paper that does this that could be invalidated in this way? Basically, to me, an empirical result arrived at using the scientific method is, well, an empirical result arrived at using the scientific method. And it seems to me that the only thing that would invalidate such a result is if it were found that the researchers did not follow the protocol they described in their paper. The fact that the researchers might have merely cited another paper, or used the method described in another paper, in which the researchers were found to have "cheated" in this way, doesn't seem to invalidate the paper in question. It's possible that the researchers in question could have happened to also cheat, but I don't see any reason to assume it's any more likely because it happened with a paper they cited (unless of course they're the same researchers building on their own previous work).
- nonbel 9y ago
  For the methods citation example, the citation chain (is supposed to) go all the way back to the paper devoted to originally describing the method. In that paper there (is supposed to be) a bunch of checks (compare to other methods, etc) that the method is working as claimed. These checks will not be repeated every time.

GolDDranks 9y ago

I mainly read papers of second language acquisition which is an empirical science that performs language learning experiments. Often the "inspiration" for the research in newer papers stems from the results of earlier ones. The empirical results of the newer ones are, of course, not directly invalidated if there was a problem with an earlier one. However, some of the papers tend to discuss the results in the light of the current empirical results AND the earlier empirical results ("from this result, we now see that A → B, and from the earlier results we already know that B → C, so there's some evidence that A → C"); this of course biases the discussion and may lead to wrong conclusions. By the way, the largest problems I see in the papers are not directly "wrong" results, but like the "Chinese whispers" or "broken telephone" game. The paper A makes some assumptions and uses certain test protocol, measurements etc, and then paper B refers the result but misrepresents it slightly, which may cause overgeneralisation. A concrete example: paper A tests the learning of some phoneme with a training period and before-after test and publishes the result. Then the paper B later references the paper A mentioning that A has shown that the learners "learn" the phoneme. However, represented this way – "unqualified" – this is an overgeneralisation: the students haven't actually shown to have learned the phoneme in any other way than using the test. Indeed – other studies have shown that using certain kind of tests are generally very poorly correlated with what people usually mean by "having learnt" something – being able to use it in spontaneous communication.