4 ms·

What if you had a webpage with the paper at the top and a comments section at the bottom where anyone in the world could post a rich-media post of their finding

What if you had a webpage with the paper at the top and a comments section at the bottom where anyone in the world could post a rich-media post of their findings that cast doubt on some findings with regard to the original paper. We currently treat academic research by a metric of "citations", why not have a new metric where the world can be cautioned of potential issues with the result. You could use the bug bounty model and invite crowdfunded contributions to pay out bounties to those who participate in debunkings. This does not need to be accepted as canonical by the scientific community at large.

pjc50 6y ago
> What if you had a webpage with the paper at the top and a comments section at the bottom where anyone in the world could post a rich-media post of their findings that cast doubt on some findings with regard to the original paper. That just enables your field to be destroyed by cranks who themselves have no accountability.
ramraj07 6y ago
That's already a thing - https://pubpeer.com/static/about https://pubpeer.com/static/about It has definitely brought out many demons in papers but it's still not widely prescribed to or refered from in academia.
throwaway2245 6y ago
> ...anyone in the world could post... This would be ultimately filled with trolls, nonexperts, and people with an ulterior motive or grudge: so any such effort would be less trustworthy than the original publication. If you want to find mistakes, you can find one in every paper. The character of the mistakes is important - is it fraud, is it incompetence, does it negate the results? Or is it good science being done by a human? An open website doesn't seem to me likely to be able to draw that out.
- grumple 6y ago
  > any such effort would be less trustworthy than the original publication. The idea that scrutiny would be less reliable than blind trust is absurd. The question in the OP, for example, could have been in the comments sections of these papers.
  - throwaway2245 6y ago
    I understood that this was about scrutinising papers in academic journals - the academic journal's value is quite literally its trustworthiness. (The journal exists and is employed to do the scrutiny) A comment from any random person (in general) holds a lower level of trustworthiness.
    - grumple 6y ago
      This post - and many other conversations we’ve had on the subject on HN - are about the lack of integrity of academic journals. More broadly, this contributes to discussions about fraud in academia, the reproducibility crisis, and the pressure to publish. “Any random person” includes many researchers, including phd holders or just random people with time on their hands, but whose commentary could be judged on its own merits, not by some credentials or stamp of approval from journals that don’t even examine the data used by studies they publish. This does not mean that comments should go completely unmoderated. As far as I’ve seen, no journal does a thorough examination of data referenced by studies it publishes. Credentials, papers, citations, and studies do little to increase the levels of trustworthiness precisely because papers like these are not publicly scrutinized.
renewiltord 6y ago
Well, I suspect that comments section could be autogenerated by a bot that randomly generates comments that say "Correlation != causation" and "Sample size only 11". The bot needn't even parse out the true sample size. It can always use a constant number.

pjc50 6y ago

> What if you had a webpage with the paper at the top and a comments section at the bottom where anyone in the world could post a rich-media post of their findings that cast doubt on some findings with regard to the original paper. That just enables your field to be destroyed by cranks who themselves have no accountability.

ramraj07 6y ago

That's already a thing - https://pubpeer.com/static/about https://pubpeer.com/static/about It has definitely brought out many demons in papers but it's still not widely prescribed to or refered from in academia.

throwaway2245 6y ago

> ...anyone in the world could post... This would be ultimately filled with trolls, nonexperts, and people with an ulterior motive or grudge: so any such effort would be less trustworthy than the original publication. If you want to find mistakes, you can find one in every paper. The character of the mistakes is important - is it fraud, is it incompetence, does it negate the results? Or is it good science being done by a human? An open website doesn't seem to me likely to be able to draw that out.

grumple 6y ago
> any such effort would be less trustworthy than the original publication. The idea that scrutiny would be less reliable than blind trust is absurd. The question in the OP, for example, could have been in the comments sections of these papers.
- throwaway2245 6y ago
  I understood that this was about scrutinising papers in academic journals - the academic journal's value is quite literally its trustworthiness. (The journal exists and is employed to do the scrutiny) A comment from any random person (in general) holds a lower level of trustworthiness.
  - grumple 6y ago
    This post - and many other conversations we’ve had on the subject on HN - are about the lack of integrity of academic journals. More broadly, this contributes to discussions about fraud in academia, the reproducibility crisis, and the pressure to publish. “Any random person” includes many researchers, including phd holders or just random people with time on their hands, but whose commentary could be judged on its own merits, not by some credentials or stamp of approval from journals that don’t even examine the data used by studies they publish. This does not mean that comments should go completely unmoderated. As far as I’ve seen, no journal does a thorough examination of data referenced by studies it publishes. Credentials, papers, citations, and studies do little to increase the levels of trustworthiness precisely because papers like these are not publicly scrutinized.

renewiltord 6y ago

Well, I suspect that comments section could be autogenerated by a bot that randomly generates comments that say "Correlation != causation" and "Sample size only 11". The bot needn't even parse out the true sample size. It can always use a constant number.