4 ms·

The review process is broken. Reviewing pre print papers isnt any more effective than reviewing printed papers. Review, and publication is a meaningless bar.

The review process is broken. Reviewing pre print papers isnt any more effective than reviewing printed papers. Review, and publication is a meaningless bar. Publish -> people find insight and try to pick it apart -> You either have flaws or you get reproduced... Only then should your paper be of any worth to be quoted or sighted from. The current system is glad-handing, intellectual protectionism and mastrubation. Academia has only itself to blame for this, and they are apparently unwilling to fix it.

bumby 3y ago
I think we need to find ways of giving status to reproducing studies. Maybe not as much as novelty, but definitely something greater than what it is currently.
- glial 3y ago
  IMO reproducing findings could/should be a mandatory part of PhD training.
ajmurmann 3y ago
"Publish -> people find insight and try to pick it apart -> You either have flaws or you get reproduced... Only then should your paper be of any worth to be quoted or sighted from." This is already how it's supposed to work. The review before publication is a fairly superficial check that just confirms that what you describe follows basic scientific practices. There is no validation of the actual research. A proper reproduction is what's supposed to come after publication. IMO the real problems are that a) there isn't much glamour and funding for reproducing other's studies and b) "science journalists", university PR departments and now in part people on social media are picking up research before people on the field looked at it or misrepresent it. Suddenly the audience is a lot of folks who never were the intended audience of the process.
- zer00eyz 3y ago
  There is a pretty simple way to change all of that. Academic standards: You are not longer allowed to site a non reproduced paper in yours. Citations matter as much as the print, put the hurdle there and all of a sudden things will change real quick.
  - freedomben 3y ago
    > You are not longer allowed to site a non reproduced paper in yours. I fully agree that in an ideal world, that would be the case. But some reproductions (especially now with machine learning) could cost millions of dollars and years to do. I don't think that's a reasonable or feasible thing to require.
    - BlueTemplar 3y ago
      Well, then, at least there should be pressure around having to mention this : a non-replicated study (by an independent group) is after all inherently suspect.
    - zer00eyz 3y ago
      >> could cost millions of dollars and years to do https://en.wikipedia.org/wiki/List_of_colleges_and_universities_in_the_United_States_by_endowment https://en.wikipedia.org/wiki/List_of_colleges_and_universit... Not only do they have the money they are charging the students more than ever.
      - freedomben 3y ago
        This is a very fair criticism and something that absolutely should be discussed, although I think it's a separate issue as papers can be written and published by anybody, regardless of backing from academic institutions.
  - ramblenode 3y ago
    > Academic standards: You are not longer allowed to site a non reproduced paper in yours. > Citations matter as much as the print, put the hurdle there and all of a sudden things will change real quick. The reproducibility crisis is just a symptom of the publish-or-perish culture that pushes academics to churning out bad research. Academia already over-emphasizes publishing positive results at the expense of studying important questions. Your solution would further incentivize low risk, low impact research that we have too much of. Aside from that, there are a lot of edge cases that would make this difficult. If I do five studies that are modifications of each other, and all show the same basic effect, but I publish it as one paper, does that count as being reproduced? What if a result has only ever been reproduced within a single research group? Does the Higgs Boson need to be confirmed at at a collider outside the LHC?

bumby 3y ago

I think we need to find ways of giving status to reproducing studies. Maybe not as much as novelty, but definitely something greater than what it is currently.

glial 3y ago
IMO reproducing findings could/should be a mandatory part of PhD training.

ajmurmann 3y ago

"Publish -> people find insight and try to pick it apart -> You either have flaws or you get reproduced... Only then should your paper be of any worth to be quoted or sighted from." This is already how it's supposed to work. The review before publication is a fairly superficial check that just confirms that what you describe follows basic scientific practices. There is no validation of the actual research. A proper reproduction is what's supposed to come after publication. IMO the real problems are that a) there isn't much glamour and funding for reproducing other's studies and b) "science journalists", university PR departments and now in part people on social media are picking up research before people on the field looked at it or misrepresent it. Suddenly the audience is a lot of folks who never were the intended audience of the process.

zer00eyz 3y ago
There is a pretty simple way to change all of that. Academic standards: You are not longer allowed to site a non reproduced paper in yours. Citations matter as much as the print, put the hurdle there and all of a sudden things will change real quick.
- freedomben 3y ago
  > You are not longer allowed to site a non reproduced paper in yours. I fully agree that in an ideal world, that would be the case. But some reproductions (especially now with machine learning) could cost millions of dollars and years to do. I don't think that's a reasonable or feasible thing to require.
  - BlueTemplar 3y ago
    Well, then, at least there should be pressure around having to mention this : a non-replicated study (by an independent group) is after all inherently suspect.
  - zer00eyz 3y ago
    >> could cost millions of dollars and years to do https://en.wikipedia.org/wiki/List_of_colleges_and_universities_in_the_United_States_by_endowment https://en.wikipedia.org/wiki/List_of_colleges_and_universit... Not only do they have the money they are charging the students more than ever.
    - freedomben 3y ago
      This is a very fair criticism and something that absolutely should be discussed, although I think it's a separate issue as papers can be written and published by anybody, regardless of backing from academic institutions.
- ramblenode 3y ago
  > Academic standards: You are not longer allowed to site a non reproduced paper in yours. > Citations matter as much as the print, put the hurdle there and all of a sudden things will change real quick. The reproducibility crisis is just a symptom of the publish-or-perish culture that pushes academics to churning out bad research. Academia already over-emphasizes publishing positive results at the expense of studying important questions. Your solution would further incentivize low risk, low impact research that we have too much of. Aside from that, there are a lot of edge cases that would make this difficult. If I do five studies that are modifications of each other, and all show the same basic effect, but I publish it as one paper, does that count as being reproduced? What if a result has only ever been reproduced within a single research group? Does the Higgs Boson need to be confirmed at at a collider outside the LHC?