5 ms·

Individual studies shouldn't be the primary focus. It shouldn't fall to every individual researcher to be fully accurate and honest. This is just not possible b

Individual studies shouldn't be the primary focus. It shouldn't fall to every individual researcher to be fully accurate and honest. This is just not possible because people are imperfect, and have all the wrong incentives in many cases. What we need is a system designed to deal with this better. My suggestion would be greater incentives to the publication of negative results, and incentives to reproduce research. Create a system that encourages reproduction as a second step to peer review. This should also increase quality. If your research can't be easily reproduced it's probably not doing a whole lot of good even if ultimately you got the right answers. Test driven research, if you will.

jonhohle 4y ago
Or even requirements for replication amongst groups with separate funding sources. If everyone needs replication for publishing, the incentive to replicate other’s work increases.
Tao3300 4y ago
> It shouldn't fall to every individual researcher to be fully... honest. Really? I don't know, "Don't be a fraud" might be a good Rule No. 1
- fennecfoxen 4y ago
  It’s more that a system should not allow a single fraud to shape its understanding and its policy. Assume fraud exists.
- naasking 4y ago
  Fraud is only a small part of the problem. No open data, poorly described methodology so reproductions sometimes aren't even possible, too little funding for reproductions, p-hacking, no incentives to publish negative results, journals incentivized to publish sensational results to increase their impact factor even though sensational results are likely false thus perpetuating misinformation. These are the main drivers.
FrenchAmerican 4y ago
Despite his dubious past, the billionaire John Arnold has focused time and money to amend scientific research. I've read maybe 2 years ago about that endeavor and followed him on Twitter ; I found it quite interesting and original. For example, he offers researchers to save their initial data (before any screening) so that other researchers can check which cases were thrown out of the study. He also funded reproduction tests. I haven't followed since, so I hope he kept on doing good work.
- pgwhalen 4y ago
  This is my first time learning about him and haven't been exposed to much more than the wikipedia article - what about his past is dubious?
Enginerrrd 4y ago
Peronsally, IMO: you should submit methodology--including the data analysis and statistical tests you'll perform, to a journal before the study has been conducted. Then, they say yes or no and if yes you perform the study and as long as it is performed to the approved specifications, you get published.
- latchkey 4y ago
  Wouldn't that just put the journals in control of what studies get performed (as well as what gets published)? Seems weird to give all that power to the journals.
  - marcosdumay 4y ago
    Well, they are already in control of what studies get published, what isn't much different. I do think they shouldn't control that, but it's a deeper change, for a different gain.
    - latchkey 4y ago
      It is a lot different because if the study isn't published in one journal, it could always be published in another. In this situation, the study wouldn't be done at all until a journal agreed to publish it.
      - hiptobecubic 4y ago
        You could submit your proposal to other journals as well though
        latchkey 4y ago
        If I was a journal, and I was promising to publish a study once it is completed, I'd ask for exclusivity rights. Which goes to show the whole issue with this idea.
bergenty 4y ago
In addition there needs to be tiers of publication that are purely objective. Maybe something along the lines of this paper was replicated by 3 different teams and the results hold up. Like a replication score or count- this would be desirable since then students could apply for funding to replicate papers.
endtime 4y ago
Prediction markets are an interesting alternative solution. If people can make money by correctly predicting the outcomes of experiments, and you have a way to prevent profit-driven bias impacting outcomes (e.g. independent experiment designers and performers, and the performers can't participate in the market but maybe get a cut of the transaction fees so that more "interesting" experiments get performed), then you get a pretty different research dynamic. (I am paraphrasing Eliezer Yuskowsky here, specifically a small part of his NSFW and frankly weird collaborative fiction Mad Investor Chaos.)
- marcosdumay 4y ago
  If prediction markets could help, we wouldn't need science.
  - endtime 4y ago
    Then how would you resolve the markets? The predictions would be about the outcomes of experiments...
kurthr 4y ago
The problem is that for any sufficiently complex or advanced research there are enough variables to contend with that there are 1000s of ways to get "no repeatable result" and only a few to actually replicate. Knowing which variables need to be controlled is part of the expertise, training, and lore. Knowing when those variables are getting out of control and how to fix them early (knowing which knob or often combination of knobs are required) is yet more expertise and experience. This is true in physics, chemistry, biology, and electronic design. Getting a null result really is often simply a failure to properly run the experiment... and as a scientist it would be very easy to follow any documented "recipe" and fail to get the proper result in hundreds of different ways. That does not make one a great scientist. You can't even tell me every way it could be screwed up. I can always come up with a new one! Programming is such a wonderfully controlled environment, it's digital, replicable, and _almost_ math-like. You can know the variables. You can trivially replicate a program, or dig down into the OS... but every physical experiment out there has much more common and worse problems than Spectre or Rowhammer waiting to destroy it. It's the cosmic rays. Your equipment/materials/specimens gets contaminated or reacts or ages. Your ultrapure water isn't. There's a vibration that changes everything. Your calibration sample changes. Cleaning staff used a new cleaner. The AC turned on. Build your experiment to be impervious to all of these and everything else I can't think of... or use the intelligence of an expert. Get there faster, cheaper, and find better ways of doing it along the way. You'ld be right to say that it should all be documented, but the problem is that you basically need to be an expert to even understand the variables and to those experts many variables are so obvious to control they wouldn't think of doing otherwise. To give concrete and extremely valuable examples in manufacturing: EUV systems for lithography, OLED deposition and patterning, Lithium battery electrode forming. You can have the very best and exquisitely designed equipment for manufacture and yet without experienced experts running the process (replicating the experiment day in and day out) it will go out of control and you will get garbage in a million ways. It's like amateurs playing chess against a master... where the master is the natural variation of the environment. High value semi manufacturing is a great example of how hard it is. The processes are designed to be as simple as possible to replicate. You replicate every day as many times as possible. You have huge amounts of money and resources to succeed and sometimes you still fail. Getting to 10% yield/replication is a big step... getting to 50% maybe harder... then 90%... and higher. Novel science is often harder and there's much less money and the time to replicate could be months/result rather than 1000s of testable results/min.

jonhohle 4y ago

Or even requirements for replication amongst groups with separate funding sources. If everyone needs replication for publishing, the incentive to replicate other’s work increases.

Tao3300 4y ago

> It shouldn't fall to every individual researcher to be fully... honest. Really? I don't know, "Don't be a fraud" might be a good Rule No. 1

fennecfoxen 4y ago
It’s more that a system should not allow a single fraud to shape its understanding and its policy. Assume fraud exists.
naasking 4y ago
Fraud is only a small part of the problem. No open data, poorly described methodology so reproductions sometimes aren't even possible, too little funding for reproductions, p-hacking, no incentives to publish negative results, journals incentivized to publish sensational results to increase their impact factor even though sensational results are likely false thus perpetuating misinformation. These are the main drivers.

FrenchAmerican 4y ago

Despite his dubious past, the billionaire John Arnold has focused time and money to amend scientific research. I've read maybe 2 years ago about that endeavor and followed him on Twitter ; I found it quite interesting and original. For example, he offers researchers to save their initial data (before any screening) so that other researchers can check which cases were thrown out of the study. He also funded reproduction tests. I haven't followed since, so I hope he kept on doing good work.

pgwhalen 4y ago
This is my first time learning about him and haven't been exposed to much more than the wikipedia article - what about his past is dubious?

Enginerrrd 4y ago

Peronsally, IMO: you should submit methodology--including the data analysis and statistical tests you'll perform, to a journal before the study has been conducted. Then, they say yes or no and if yes you perform the study and as long as it is performed to the approved specifications, you get published.

latchkey 4y ago
Wouldn't that just put the journals in control of what studies get performed (as well as what gets published)? Seems weird to give all that power to the journals.
- marcosdumay 4y ago
  Well, they are already in control of what studies get published, what isn't much different. I do think they shouldn't control that, but it's a deeper change, for a different gain.
  - latchkey 4y ago
    It is a lot different because if the study isn't published in one journal, it could always be published in another. In this situation, the study wouldn't be done at all until a journal agreed to publish it.
    - hiptobecubic 4y ago
      You could submit your proposal to other journals as well though
      - latchkey 4y ago
        If I was a journal, and I was promising to publish a study once it is completed, I'd ask for exclusivity rights. Which goes to show the whole issue with this idea.

bergenty 4y ago

In addition there needs to be tiers of publication that are purely objective. Maybe something along the lines of this paper was replicated by 3 different teams and the results hold up. Like a replication score or count- this would be desirable since then students could apply for funding to replicate papers.

endtime 4y ago

Prediction markets are an interesting alternative solution. If people can make money by correctly predicting the outcomes of experiments, and you have a way to prevent profit-driven bias impacting outcomes (e.g. independent experiment designers and performers, and the performers can't participate in the market but maybe get a cut of the transaction fees so that more "interesting" experiments get performed), then you get a pretty different research dynamic. (I am paraphrasing Eliezer Yuskowsky here, specifically a small part of his NSFW and frankly weird collaborative fiction Mad Investor Chaos.)

marcosdumay 4y ago
If prediction markets could help, we wouldn't need science.
- endtime 4y ago
  Then how would you resolve the markets? The predictions would be about the outcomes of experiments...

kurthr 4y ago

The problem is that for any sufficiently complex or advanced research there are enough variables to contend with that there are 1000s of ways to get "no repeatable result" and only a few to actually replicate. Knowing which variables need to be controlled is part of the expertise, training, and lore. Knowing when those variables are getting out of control and how to fix them early (knowing which knob or often combination of knobs are required) is yet more expertise and experience. This is true in physics, chemistry, biology, and electronic design. Getting a null result really is often simply a failure to properly run the experiment... and as a scientist it would be very easy to follow any documented "recipe" and fail to get the proper result in hundreds of different ways. That does not make one a great scientist. You can't even tell me every way it could be screwed up. I can always come up with a new one! Programming is such a wonderfully controlled environment, it's digital, replicable, and _almost_ math-like. You can know the variables. You can trivially replicate a program, or dig down into the OS... but every physical experiment out there has much more common and worse problems than Spectre or Rowhammer waiting to destroy it. It's the cosmic rays. Your equipment/materials/specimens gets contaminated or reacts or ages. Your ultrapure water isn't. There's a vibration that changes everything. Your calibration sample changes. Cleaning staff used a new cleaner. The AC turned on. Build your experiment to be impervious to all of these and everything else I can't think of... or use the intelligence of an expert. Get there faster, cheaper, and find better ways of doing it along the way. You'ld be right to say that it should all be documented, but the problem is that you basically need to be an expert to even understand the variables and to those experts many variables are so obvious to control they wouldn't think of doing otherwise. To give concrete and extremely valuable examples in manufacturing: EUV systems for lithography, OLED deposition and patterning, Lithium battery electrode forming. You can have the very best and exquisitely designed equipment for manufacture and yet without experienced experts running the process (replicating the experiment day in and day out) it will go out of control and you will get garbage in a million ways. It's like amateurs playing chess against a master... where the master is the natural variation of the environment. High value semi manufacturing is a great example of how hard it is. The processes are designed to be as simple as possible to replicate. You replicate every day as many times as possible. You have huge amounts of money and resources to succeed and sometimes you still fail. Getting to 10% yield/replication is a big step... getting to 50% maybe harder... then 90%... and higher. Novel science is often harder and there's much less money and the time to replicate could be months/result rather than 1000s of testable results/min.