4 ms·

Have they solved the issue where papers that cite research already invalidated are still being cited?

cogman10 9mo ago
AFAIK, no, but I could see there being cause to push citations to also cite the validations. It'd be good if standard practice turned into something like Paper A, by bob, bill, brad. Validated by Paper B by carol, clare, charlotte. or Paper A, by bob, bill, brad. Unvalidated.
- gcr 9mo ago
  Academics typically use citation count and popularity as a rough proxy for validation. It's certainly not perfect, but it is something that people think about. Semantic Scholar in particular is doing great work in this area, making it easy to see who cites who: https://www.semanticscholar.org/ https://www.semanticscholar.org/ Google Scholar's PDF reader extension turns every hyperlinked citation into a popout card that shows citation counts inline in the PDF: https://chromewebstore.google.com/detail/google-scholar-pdf-reader/dahenjhkoodjbpjheillcadbppiidmhp?hl=en https://chromewebstore.google.com/detail/google-scholar-pdf-...
  - rtkwe 9mo ago
    That is a factor most people miss when thinking about the replication crisis. For the harder physical sciences a wrong paper will fairly quickly be found because as people go to expand on the ideas/use that data and get results that don't match the model informed by paper X they're going to eventually figure out that X is wrong. There might be issues with getting incentives to write and publish that negative result but each paper where the results of a previous paper are actually used in the new paper is a form of replication.
reliabilityguy 9mo ago
Nope. I am still reviewing papers that propose solutions based on a technique X, conveniently ignoring research from two years ago that shows that X cannot be used on its own. Both the paper I reviewed and the research showing X cannot be used are in the same venue!
- b00ty4breakfast 9mo ago
  does it seem to be legitimate ignorance or maybe folks pushing ahead regardless of x being disproved?
  - freedomben 9mo ago
    IMHO, It's mostly ignorance coming a push/drive to "publish or perish." When the stakes are so high and output is so valued, and when reproducability isn't required, it disincentivizes thorough work. The system is set up in a way that is making it fail. There is also the reality that "one paper" or "one study" can be found contradicted almost anything, so if you just went with "some other paper/study debunks my premise" then you'd end up producing nothing. Plus many inside know that there's a lot of slop out there that gets published, so they can (sometimes reasonably IMHO) dismiss that "one paper" even when they do know about it. It's (mostly) not fraud or malicious intent or ignorance, it's (mostly) humans existing in the system in which they must live.
  - reliabilityguy 9mo ago
    Poor scholarship. However, given the feedback by other reviewers, I was the only one who knew that X doesn’t work. I am not sure how these people mark themselves as “experts” in the field if they are not following the literature themselves.

AFAIK, no, but I could see there being cause to push citations to also cite the validations. It'd be good if standard practice turned into something like Paper A, by bob, bill, brad. Validated by Paper B by carol, clare, charlotte. or Paper A, by bob, bill, brad. Unvalidated.

gcr 9mo ago
Academics typically use citation count and popularity as a rough proxy for validation. It's certainly not perfect, but it is something that people think about. Semantic Scholar in particular is doing great work in this area, making it easy to see who cites who: https://www.semanticscholar.org/ https://www.semanticscholar.org/ Google Scholar's PDF reader extension turns every hyperlinked citation into a popout card that shows citation counts inline in the PDF: https://chromewebstore.google.com/detail/google-scholar-pdf-reader/dahenjhkoodjbpjheillcadbppiidmhp?hl=en https://chromewebstore.google.com/detail/google-scholar-pdf-...
- rtkwe 9mo ago
  That is a factor most people miss when thinking about the replication crisis. For the harder physical sciences a wrong paper will fairly quickly be found because as people go to expand on the ideas/use that data and get results that don't match the model informed by paper X they're going to eventually figure out that X is wrong. There might be issues with getting incentives to write and publish that negative result but each paper where the results of a previous paper are actually used in the new paper is a form of replication.

reliabilityguy 9mo ago

Nope. I am still reviewing papers that propose solutions based on a technique X, conveniently ignoring research from two years ago that shows that X cannot be used on its own. Both the paper I reviewed and the research showing X cannot be used are in the same venue!

b00ty4breakfast 9mo ago
does it seem to be legitimate ignorance or maybe folks pushing ahead regardless of x being disproved?
- freedomben 9mo ago
  IMHO, It's mostly ignorance coming a push/drive to "publish or perish." When the stakes are so high and output is so valued, and when reproducability isn't required, it disincentivizes thorough work. The system is set up in a way that is making it fail. There is also the reality that "one paper" or "one study" can be found contradicted almost anything, so if you just went with "some other paper/study debunks my premise" then you'd end up producing nothing. Plus many inside know that there's a lot of slop out there that gets published, so they can (sometimes reasonably IMHO) dismiss that "one paper" even when they do know about it. It's (mostly) not fraud or malicious intent or ignorance, it's (mostly) humans existing in the system in which they must live.
- reliabilityguy 9mo ago
  Poor scholarship. However, given the feedback by other reviewers, I was the only one who knew that X doesn’t work. I am not sure how these people mark themselves as “experts” in the field if they are not following the literature themselves.