6 ms·

Unfortunately, what you'll actually incentivize is spending as little effort as possible to get the money paid out.

by nitwit005 3y ago

matwood 3y ago
Opposed to now where it appears lots of science is peer reviewed with all the problems found later?
- nitwit005 3y ago
  Replacing a broken system, with a broken and also expensive system, does not sound like an improvement.
- andyferris 3y ago
  It might cost $100k - $1m (or more) to repeat the work and run replications. The $5k - $10k mentioned earlier would be enough to allow time reading and thinking and checking some stuff on pen-and-paper.
  - michaelt 3y ago
    > The $5k - $10k mentioned earlier would be enough to allow time reading and thinking and checking some stuff on pen-and-paper. The average postdoc in the US earns $32.81/hour, according to the results of some googling. Even taking overheads into account, $5k should cover more than a week's full time work.
    - sseagull 3y ago
      In what area of science would it take only a week or two to replicate? It might take several days to a week of literature review just to fully understand the problem. Then you might need equipment, chemicals, cultures, etc. Then depending on the area of science, doing the actual experiment could take several weeks (waiting for reactions, computer simulations, etc). Then possibly tricky analysis and statistics on top of that. Nowadays, science is deep
      - rhelz 3y ago
        Science is very deep; I’m sure that to replicate some studies it would cost as much as performing the original study did. But whether it’s $10k or $100k, we really should provide the funds to do it. Expensive? Yeah, but not as expensive funding grants for generations of psychology professors and getting nothing—or worse than nothing—in return. Psychology could fix its replication crisis tomorrow if as part of writing every grant, they also calculated what it would take for another group to replicate their experiments, and put that money in escrow to hire reviewers and replicators who had to sign off on any papers published.
ramblenode 3y ago
Is that different from any other job?
aashiq 3y ago
It depends! There should probably also be a process by which reviewers themselves get graded. Then paper writers can choose whether to splurge for fewer really amazing reviewers, or a larger quantity of mediocre reviewers. Also, readers will be able to see the quality of the reviewers that looked at a preprint.
- BlueTemplar 3y ago
  How do you have all three of anonymous authors, anonymous reviewers, and reviewer ratings ?
  - bumby 3y ago
    It might be possible to have a third-party manage the reviewer ratings. Although I suspect some fields are so small/niche that if someone wanted to associate some random ID with a real person, they could match writing styles etc.
    - BlueTemplar 3y ago
      How isn't that third party just reinventing journals ?
      - bumby 3y ago
        The product is different. Consumer Reports isn't reinventing cars; it provides a service by independently rating cars.
- bumby 3y ago
  Splurging for "amazing reviewers" could also be gamed to "splurge on those reviewers who are likely to rubber-stamp my submission to get paid" (not unlike some of the questionable open-access journals' current business models).
hiddencost 3y ago
I hear this sentiment a lot. There was a time when academia was intensely driven by culture. People did stuff because they cared about the community surviving. It is, in fact, possible to choose the "cooperate" triangle of the prisoner's dilemma, but it takes a lot of work, and is vulnerable to being wrecked by people who don't care / believe in the goals.
- matthewdgreen 3y ago
  If you keep increasing publication requirements in a Red Queen’s race, you both increase the amount of peer-review required and also reduce the time that qualified scientists have to do it. No matter how much people want to build a functioning system, they only have so many hours in the day.
  - rhelz 3y ago
    A valid concern, but what is driving this red-queen race is that you are competing with people who cheat. If you’ve got one really solid paper published, but your competitor fraudulently published 10, it’s real tempting to fraud a few papers yourself. But they multiply fraudulent papers because they can get away with it, and they can get away with it because nobody is really reviewing or replicating those results. I propose a fix: if a paper wouldn’t get published unless somebody has replicated the results—-there would be a lot less papers published, and the the expectation of how many papers a prof should publish a year would fall to reasonable levels. It’s the fraud which is killing us. Get rid of the fraud and a lot of the other problems with academia would start resolving themselves.
    - matthewdgreen 3y ago
      The idea that fraud is rampant and driving the explosion in publication rates is deeply tempting to folks who don’t work directly in a scientific field. But it’s a misconception that’s largely driven by non-scientific media and bloggers. In practice literal fraud does exist, but it’s relatively rare in most scientific fields. The explosion in publication rates is largely caused by a combination of (1) more people entering the field, and (2) “paper slicing” where people submit smaller contributions as part of each paper, in order to maximize publication metrics. As far as “no publication without replicating the results”. Think for a second about how that will work. How is anyone going to replicate a result if it’s not published? This is literally the reason that scientific publication exists in the first place: to provide a channel by which new results can be conveyed to other scientists so that they can learn from, and in some cases, replicate those results. So clearly you don’t want to prevent publication of non-replicated results, you just want to prevent people from getting some kind of academic brownie-point credit for those publications. Perhaps that’s workable, I don’t know. But you need to be a lot more knowledgeable and curious about the scientific publication system in order to have any hope of improving it. Taken literally, your “no publication without replication” proposal would inhibit scientific replication entirely.

Unfortunately, what you'll actually incentivize is spending as little effort as possible to get the money paid out.

matwood 3y ago

Opposed to now where it appears lots of science is peer reviewed with all the problems found later?

nitwit005 3y ago
Replacing a broken system, with a broken and also expensive system, does not sound like an improvement.
andyferris 3y ago
It might cost $100k - $1m (or more) to repeat the work and run replications. The $5k - $10k mentioned earlier would be enough to allow time reading and thinking and checking some stuff on pen-and-paper.
- michaelt 3y ago
  > The $5k - $10k mentioned earlier would be enough to allow time reading and thinking and checking some stuff on pen-and-paper. The average postdoc in the US earns $32.81/hour, according to the results of some googling. Even taking overheads into account, $5k should cover more than a week's full time work.
  - sseagull 3y ago
    In what area of science would it take only a week or two to replicate? It might take several days to a week of literature review just to fully understand the problem. Then you might need equipment, chemicals, cultures, etc. Then depending on the area of science, doing the actual experiment could take several weeks (waiting for reactions, computer simulations, etc). Then possibly tricky analysis and statistics on top of that. Nowadays, science is deep
    - rhelz 3y ago
      Science is very deep; I’m sure that to replicate some studies it would cost as much as performing the original study did. But whether it’s $10k or $100k, we really should provide the funds to do it. Expensive? Yeah, but not as expensive funding grants for generations of psychology professors and getting nothing—or worse than nothing—in return. Psychology could fix its replication crisis tomorrow if as part of writing every grant, they also calculated what it would take for another group to replicate their experiments, and put that money in escrow to hire reviewers and replicators who had to sign off on any papers published.

ramblenode 3y ago

Is that different from any other job?

aashiq 3y ago

It depends! There should probably also be a process by which reviewers themselves get graded. Then paper writers can choose whether to splurge for fewer really amazing reviewers, or a larger quantity of mediocre reviewers. Also, readers will be able to see the quality of the reviewers that looked at a preprint.

BlueTemplar 3y ago
How do you have all three of anonymous authors, anonymous reviewers, and reviewer ratings ?
- bumby 3y ago
  It might be possible to have a third-party manage the reviewer ratings. Although I suspect some fields are so small/niche that if someone wanted to associate some random ID with a real person, they could match writing styles etc.
  - BlueTemplar 3y ago
    How isn't that third party just reinventing journals ?
    - bumby 3y ago
      The product is different. Consumer Reports isn't reinventing cars; it provides a service by independently rating cars.
bumby 3y ago
Splurging for "amazing reviewers" could also be gamed to "splurge on those reviewers who are likely to rubber-stamp my submission to get paid" (not unlike some of the questionable open-access journals' current business models).

hiddencost 3y ago

I hear this sentiment a lot. There was a time when academia was intensely driven by culture. People did stuff because they cared about the community surviving. It is, in fact, possible to choose the "cooperate" triangle of the prisoner's dilemma, but it takes a lot of work, and is vulnerable to being wrecked by people who don't care / believe in the goals.

matthewdgreen 3y ago
If you keep increasing publication requirements in a Red Queen’s race, you both increase the amount of peer-review required and also reduce the time that qualified scientists have to do it. No matter how much people want to build a functioning system, they only have so many hours in the day.
- rhelz 3y ago
  A valid concern, but what is driving this red-queen race is that you are competing with people who cheat. If you’ve got one really solid paper published, but your competitor fraudulently published 10, it’s real tempting to fraud a few papers yourself. But they multiply fraudulent papers because they can get away with it, and they can get away with it because nobody is really reviewing or replicating those results. I propose a fix: if a paper wouldn’t get published unless somebody has replicated the results—-there would be a lot less papers published, and the the expectation of how many papers a prof should publish a year would fall to reasonable levels. It’s the fraud which is killing us. Get rid of the fraud and a lot of the other problems with academia would start resolving themselves.
  - matthewdgreen 3y ago
    The idea that fraud is rampant and driving the explosion in publication rates is deeply tempting to folks who don’t work directly in a scientific field. But it’s a misconception that’s largely driven by non-scientific media and bloggers. In practice literal fraud does exist, but it’s relatively rare in most scientific fields. The explosion in publication rates is largely caused by a combination of (1) more people entering the field, and (2) “paper slicing” where people submit smaller contributions as part of each paper, in order to maximize publication metrics. As far as “no publication without replicating the results”. Think for a second about how that will work. How is anyone going to replicate a result if it’s not published? This is literally the reason that scientific publication exists in the first place: to provide a channel by which new results can be conveyed to other scientists so that they can learn from, and in some cases, replicate those results. So clearly you don’t want to prevent publication of non-replicated results, you just want to prevent people from getting some kind of academic brownie-point credit for those publications. Perhaps that’s workable, I don’t know. But you need to be a lot more knowledgeable and curious about the scientific publication system in order to have any hope of improving it. Taken literally, your “no publication without replication” proposal would inhibit scientific replication entirely.