5 ms·

I'm not sure how less gatekeeping mechanisms would be helpful ? Wouldn't that just lead to more crap being produced, and informal power structures and cliques

I'm not sure how less gatekeeping mechanisms would be helpful ? Wouldn't that just lead to more crap being produced, and informal power structures and cliques forming ? It sounds like research needs to be held accountable to some kind of standard for accuracy or quality.

dokein 4y ago
One issue is that the current gatekeepers (peer reviewers for journals, grant proposal scoring committees, promotion committees, etc.) are often the people most prominent in their field. On one hand this makes sense for obvious reasons (an expert is the most equipped to judge their field), but on the other hand things like the amyloid hypothesis get 'baked-in' because, well, it's pretty hard to ask those same individuals to highly rank a large grant proposal that goes against their own theory. So I think the answer is gatekeeping needs to be different -- not less.
- Beldin 4y ago
  Do note that the more the gate is kept, the stronger the incentive for cheating. Moreover, peer review is overrated. An experiment by the NIPS conference in 2014 found that ~60% of submissions are in the "gray middle" - not obviously great or obviously crap. In more detail: they split the PC in two, and had 10% of papers (166) reviewed by both halves. Each half had to accept ~37 papers. The halves disagreed on 21 accepted papers [1]. So yeah, if your submission is sciencey, it's the flip of a coin whether peer review will accept it. [1] https://hunch.net/?p=467864 https://hunch.net/?p=467864
slowmovintarget 4y ago
Agree. I don't think we want /r/science moderators determining what is accepted research. But there must be some way to democratize the process short of waiting for the perfect AGI to moderate publications.
naasking 4y ago
It seems plausible that informal power structures may be easier to disrupt with better data than formal power structures.

dokein 4y ago

One issue is that the current gatekeepers (peer reviewers for journals, grant proposal scoring committees, promotion committees, etc.) are often the people most prominent in their field. On one hand this makes sense for obvious reasons (an expert is the most equipped to judge their field), but on the other hand things like the amyloid hypothesis get 'baked-in' because, well, it's pretty hard to ask those same individuals to highly rank a large grant proposal that goes against their own theory. So I think the answer is gatekeeping needs to be different -- not less.

Beldin 4y ago
Do note that the more the gate is kept, the stronger the incentive for cheating. Moreover, peer review is overrated. An experiment by the NIPS conference in 2014 found that ~60% of submissions are in the "gray middle" - not obviously great or obviously crap. In more detail: they split the PC in two, and had 10% of papers (166) reviewed by both halves. Each half had to accept ~37 papers. The halves disagreed on 21 accepted papers [1]. So yeah, if your submission is sciencey, it's the flip of a coin whether peer review will accept it. [1] https://hunch.net/?p=467864 https://hunch.net/?p=467864

slowmovintarget 4y ago

Agree. I don't think we want /r/science moderators determining what is accepted research. But there must be some way to democratize the process short of waiting for the perfect AGI to moderate publications.

naasking 4y ago

It seems plausible that informal power structures may be easier to disrupt with better data than formal power structures.