5 ms·

> If you believe replicability is central to science. I do believe that, but it doesn’t matter what anybody believes, replicatable experiments and results, whi

by rhelz 3y ago

> If you believe replicability is central to science. I do believe that, but it doesn’t matter what anybody believes, replicatable experiments and results, which peers can review and agree on, are the soul of science. Without that it’s not science, it’s just creative writing.

bumby 3y ago
Then don’t you think the first part of that process is explaining the methods and results of your experiment? That’s precisely what the current situation does. What’s lacking is the replication incentives.
- rhelz 3y ago
  I propose to supply those incentives—-pay for good reviewer and replicators. If you want something done, you gotta pay for it. We can’t just rely on volunteers.
  - bumby 3y ago
    I'm on board with that idea, as long as we can also provide guardrails against the perverse incentives of paying for them. E.g., we need to avoid frivolous reviews/replication as well as something evolving into a "pay for a good review" service.
    - rhelz 3y ago
      Well, if you are paying somebody to do something, you gain a lot of leverage: 1. You can negotiate a due date. No more waiting for years before the journal's reviewers actually review your paper. 2. You can negotiate a set of deliverables. You can specify that they can't just say "this sux"; they have to show the lines where the big hairy proof are wrong, or if its an algorithm, they have to actually code it up and run it themselves before they say it doesn't work. 3. You can more reliably attract good reviewers. If you aren't begging for people to volunteer, but you are paying good money, you can be a lot pickier about who you hire. I mean, I've been a consultant: what are the guardrails that I won't rip-off my clients? I don't want to ruin my reputation, I want repeat business, and I want to be able to charge a high hourly rate because I deliver a premium product. Same guardrails would apply to peer reviewers and to reproducers.
      - bumby 3y ago
        Sure, but I’m poking at the bad leverage you can also wield. 1) you can create undue schedule pressure that results in a cursory review that may not catch the more nuanced problems in your investigation. 2) you can be more belligerent about not sharing data. If they want to get, they won’t argue. 3) you can pay for reviewers who you know will give a positive review. Without guards against this, it’s almost a certainty that the glut of PhDs will result in some treating it like a scam de hustle where it’s more about the economics than the science. Some consultants are well known to play the game where they tell clients what they want to hear rather than what they need to hear. I don’t think consultancy is a good model for this.
        rhelz 3y ago
        #1 isn't an issue unique to paying peer reviewers. We've learned how to negotiate such hazards. #2 Seems like a team who wants their paper published would be super-helpful to the reviewers and replicators....why wouldn't they be maximally motivated to help them to by sharing data, techniques, etc...and writing their paper so that its easy for reviews and replicators to do their jobs. #3 The authors of a paper don't get to choose who their reviewers are!! > consultants...play the game And yet we have millions of clients hiring millions of consultants, and somehow they are able to make it work....yeah, all these issues can arise in other contexts, we know how to deal with them.
        bumby 3y ago
        You are right that #1 isn't unique. But I think you're wrong that we've got the issue easily solved because it's rooted in human psychology. Just look at the last few years of Boeing headlines and tell me you still think schedule pressure in a competitive environment is a solved problem. Your response to #2 assumes the researcher wants to create the most transparent and highest-quality paper. Because of perverse incentives, I don't think this is the case. Many times researchers just want a publication because that gets them the career status they're after. Good point on #3, but it still leaves the question about the tradeoff between quantity and quality. I can surely churn out many more reviews of questionable quality than I can a single, well-researched and thoughtful review. The quantity vs. quality tradeoff is really what is at the heart of that point. >And yet we have millions of clients hiring millions of consultants The existence of that market doesn't mean the market does what you're claiming. Many times, consultancy is a mechanism to pay for plausible deniability rather than a novel solution.

by rhelz 3y ago

bumby 3y ago

Then don’t you think the first part of that process is explaining the methods and results of your experiment? That’s precisely what the current situation does. What’s lacking is the replication incentives.

rhelz 3y ago
I propose to supply those incentives—-pay for good reviewer and replicators. If you want something done, you gotta pay for it. We can’t just rely on volunteers.
- bumby 3y ago
  I'm on board with that idea, as long as we can also provide guardrails against the perverse incentives of paying for them. E.g., we need to avoid frivolous reviews/replication as well as something evolving into a "pay for a good review" service.
  - rhelz 3y ago
    Well, if you are paying somebody to do something, you gain a lot of leverage: 1. You can negotiate a due date. No more waiting for years before the journal's reviewers actually review your paper. 2. You can negotiate a set of deliverables. You can specify that they can't just say "this sux"; they have to show the lines where the big hairy proof are wrong, or if its an algorithm, they have to actually code it up and run it themselves before they say it doesn't work. 3. You can more reliably attract good reviewers. If you aren't begging for people to volunteer, but you are paying good money, you can be a lot pickier about who you hire. I mean, I've been a consultant: what are the guardrails that I won't rip-off my clients? I don't want to ruin my reputation, I want repeat business, and I want to be able to charge a high hourly rate because I deliver a premium product. Same guardrails would apply to peer reviewers and to reproducers.
    - bumby 3y ago
      Sure, but I’m poking at the bad leverage you can also wield. 1) you can create undue schedule pressure that results in a cursory review that may not catch the more nuanced problems in your investigation. 2) you can be more belligerent about not sharing data. If they want to get, they won’t argue. 3) you can pay for reviewers who you know will give a positive review. Without guards against this, it’s almost a certainty that the glut of PhDs will result in some treating it like a scam de hustle where it’s more about the economics than the science. Some consultants are well known to play the game where they tell clients what they want to hear rather than what they need to hear. I don’t think consultancy is a good model for this.
      - rhelz 3y ago
        #1 isn't an issue unique to paying peer reviewers. We've learned how to negotiate such hazards. #2 Seems like a team who wants their paper published would be super-helpful to the reviewers and replicators....why wouldn't they be maximally motivated to help them to by sharing data, techniques, etc...and writing their paper so that its easy for reviews and replicators to do their jobs. #3 The authors of a paper don't get to choose who their reviewers are!! > consultants...play the game And yet we have millions of clients hiring millions of consultants, and somehow they are able to make it work....yeah, all these issues can arise in other contexts, we know how to deal with them.
        bumby 3y ago
        You are right that #1 isn't unique. But I think you're wrong that we've got the issue easily solved because it's rooted in human psychology. Just look at the last few years of Boeing headlines and tell me you still think schedule pressure in a competitive environment is a solved problem. Your response to #2 assumes the researcher wants to create the most transparent and highest-quality paper. Because of perverse incentives, I don't think this is the case. Many times researchers just want a publication because that gets them the career status they're after. Good point on #3, but it still leaves the question about the tradeoff between quantity and quality. I can surely churn out many more reviews of questionable quality than I can a single, well-researched and thoughtful review. The quantity vs. quality tradeoff is really what is at the heart of that point. >And yet we have millions of clients hiring millions of consultants The existence of that market doesn't mean the market does what you're claiming. Many times, consultancy is a mechanism to pay for plausible deniability rather than a novel solution.