3 ms·

My pet idea for making science less wrong is to maintain a citation graph of all papers. Then when a problem is found with any paper, every other one downstream

My pet idea for making science less wrong is to maintain a citation graph of all papers. Then when a problem is found with any paper, every other one downstream from it is automatically flagged as at risk of being wrong. Now all those authors (or others) have to go back and re-evaluate how that citation was used and how it affects their result. Once they decide it's OK, they update their paper and remove the flag. If it's not OK then their paper is retracted or flagged as wrong or simply keeps its at-risk flag if it's not important enough for anybody to bother re-checking it. This way, people will be reluctant to cite too many useless papers for their friends, and will be reluctant to depend on unreliable ones. Work with good methodology would be more popular to cite or rely on. Maybe peer review could even become optional this way. If other researchers trust your work enough to risk their own papers by citing it, that would act as a peer approval in the long term. Actual peer review would only be needed to give an immediate indication of the quality. We would first need to solve the versioning problem where there's no way to update a paper when mistakes are discovered or simply to improve it.

throwaway287391 9y ago
> My pet idea for making science less wrong is to maintain a citation graph of all papers. Then when a problem is found with any paper, every other one downstream from it is automatically flagged as at risk of being wrong. Now all those authors (or others) have to go back and re-evaluate how that citation was used and how it affects their result. Once they decide it's OK, they update their paper and remove the flag. If it's not OK then their paper is retracted or flagged as wrong or simply keeps its at-risk flag if it's not important enough for anybody to bother re-checking it. This is an interesting idea, but I don't know what real-world problem it would actually solve. As someone who reads a lot of scientific papers, I can't think of a single one whose correctness is actually dependent on the correctness of the results of a paper it cites, as though the paper in question is "buggy" because it "calls another paper as a subroutine" and that subroutine is buggy. Could you give an example or two of this? I could see this making sense in mathematics since mathematical proofs really do "call other papers as subroutines" (theorems), but not so much in science.
- nonbel 9y ago
  >"As someone who reads a lot of scientific papers, I can't think of a single one whose correctness is actually dependent on the correctness of the results of a paper it cites" What field are you reading where each paper is independent of all the others? As the most obvious example, in biomed pretty much every paper says something like "as done previously[ref x, y]" in the methods section. It is an extremely annoying, but very common, practice (usually refs x and y contain somewhat different methods descriptions and/or also contain the "as done previously" phrase). There are many other ways the papers depend on each other too ("we used these cells because they don't express this protein[ref z]", etc)
  - peteretep 9y ago
    > > whose correctness is actually > > dependent on the correctness of the > > results of *a* paper it cites > where each paper is independent of > *all* the others These are not describing the same thing
    - nonbel 9y ago
      I think they are, unless you are selectively citing (irrelevant) papers to avoid depending on any...
    - mekarpeles 9y ago
      If I understand, the author is suggesting something like pagerank for axiomatic correctness where a paper's score can be interpreted per paper (in terms of some binary promise of "correctness") even if the score is computed by the weighted correctness of the papers it represents. Really, the information should be abstracted away from the paper, a la freebase or wikidata, and there should be strict rules (e.g. the protocols stackoverflow enforces) such that no cycles can occur, such that propositions can be resolved to axioms, and such that it is practical to compute scores in a finite period of time with reasonable resource constraints... See also gitxiv.org (at attempt at literate research for computer science and physics papers that can be paired with code). It doesn't address the problem described, but if statements of a paper are codified (e.g. curry howard isomorphism) in a strict way, it makes it easier for "linked statements" to have a quantifiable value.
rexpop 9y ago
Surely this exists.
IanCal 9y ago
You could prototype this with the data in crossref at the moment. They don't have full citation coverage (basically nobody does, it's a bugger of a problem), but there should be enough there to try out some ideas with real world data (and free data with an API). > We would first need to solve the versioning problem where there's no way to update a paper when mistakes are discovered or simply to improve it. You can certainly update what a DOI points to.
mekarpeles 9y ago
@averagewall, check out openknowledgemaps.org Also, citeseerx and crossref are your friends. Pubmed central similarly has the ability to return much of this citation data programatically (for certain scientific subdomains) If you're passionate about prototyping such a service, let me know and we can get you an invite to the Archive Labs slack (we're a non-profit, volunteer-run community which incubates open access and public good projects). Literate, reproducible research and knowledge + citation graphs are two of our emphases. Good luck
- mekarpeles 9y ago
  P.S. One way you might describe this project is "an axiomatic Euclid's Elements for all scholarly knowledge". You might also like metacademy.org (one of my favourite projects -- it's a package manager for knowledge built on DAGs)
CogDisco 9y ago
What would be your plan for papers that cite another one because they are doing a literature review or something akin to "Paper X is wrong"? There's also a trickier problem in your simple binary flag: papers can have incorrect proofs of correct statements. I'd love to see a Semantic Web approach where you don't cite papers but results in papers. So while a result in a paper might be wrong, you don't have to throw out all the results in all the papers citing this paper.

throwaway287391 9y ago

> My pet idea for making science less wrong is to maintain a citation graph of all papers. Then when a problem is found with any paper, every other one downstream from it is automatically flagged as at risk of being wrong. Now all those authors (or others) have to go back and re-evaluate how that citation was used and how it affects their result. Once they decide it's OK, they update their paper and remove the flag. If it's not OK then their paper is retracted or flagged as wrong or simply keeps its at-risk flag if it's not important enough for anybody to bother re-checking it. This is an interesting idea, but I don't know what real-world problem it would actually solve. As someone who reads a lot of scientific papers, I can't think of a single one whose correctness is actually dependent on the correctness of the results of a paper it cites, as though the paper in question is "buggy" because it "calls another paper as a subroutine" and that subroutine is buggy. Could you give an example or two of this? I could see this making sense in mathematics since mathematical proofs really do "call other papers as subroutines" (theorems), but not so much in science.

nonbel 9y ago
>"As someone who reads a lot of scientific papers, I can't think of a single one whose correctness is actually dependent on the correctness of the results of a paper it cites" What field are you reading where each paper is independent of all the others? As the most obvious example, in biomed pretty much every paper says something like "as done previously[ref x, y]" in the methods section. It is an extremely annoying, but very common, practice (usually refs x and y contain somewhat different methods descriptions and/or also contain the "as done previously" phrase). There are many other ways the papers depend on each other too ("we used these cells because they don't express this protein[ref z]", etc)
- peteretep 9y ago
  > > whose correctness is actually > > dependent on the correctness of the > > results of *a* paper it cites > where each paper is independent of > *all* the others These are not describing the same thing
  - nonbel 9y ago
    I think they are, unless you are selectively citing (irrelevant) papers to avoid depending on any...
  - mekarpeles 9y ago
    If I understand, the author is suggesting something like pagerank for axiomatic correctness where a paper's score can be interpreted per paper (in terms of some binary promise of "correctness") even if the score is computed by the weighted correctness of the papers it represents. Really, the information should be abstracted away from the paper, a la freebase or wikidata, and there should be strict rules (e.g. the protocols stackoverflow enforces) such that no cycles can occur, such that propositions can be resolved to axioms, and such that it is practical to compute scores in a finite period of time with reasonable resource constraints... See also gitxiv.org (at attempt at literate research for computer science and physics papers that can be paired with code). It doesn't address the problem described, but if statements of a paper are codified (e.g. curry howard isomorphism) in a strict way, it makes it easier for "linked statements" to have a quantifiable value.

rexpop 9y ago

Surely this exists.

IanCal 9y ago

You could prototype this with the data in crossref at the moment. They don't have full citation coverage (basically nobody does, it's a bugger of a problem), but there should be enough there to try out some ideas with real world data (and free data with an API). > We would first need to solve the versioning problem where there's no way to update a paper when mistakes are discovered or simply to improve it. You can certainly update what a DOI points to.

mekarpeles 9y ago

@averagewall, check out openknowledgemaps.org Also, citeseerx and crossref are your friends. Pubmed central similarly has the ability to return much of this citation data programatically (for certain scientific subdomains) If you're passionate about prototyping such a service, let me know and we can get you an invite to the Archive Labs slack (we're a non-profit, volunteer-run community which incubates open access and public good projects). Literate, reproducible research and knowledge + citation graphs are two of our emphases. Good luck

mekarpeles 9y ago
P.S. One way you might describe this project is "an axiomatic Euclid's Elements for all scholarly knowledge". You might also like metacademy.org (one of my favourite projects -- it's a package manager for knowledge built on DAGs)

CogDisco 9y ago

What would be your plan for papers that cite another one because they are doing a literature review or something akin to "Paper X is wrong"? There's also a trickier problem in your simple binary flag: papers can have incorrect proofs of correct statements. I'd love to see a Semantic Web approach where you don't cite papers but results in papers. So while a result in a paper might be wrong, you don't have to throw out all the results in all the papers citing this paper.