4 ms·

Can you explain that a bit more? I'm fascinated but don't quite get the rationale behind having not even the authors of the paper know the results. What is thei

Can you explain that a bit more? I'm fascinated but don't quite get the rationale behind having not even the authors of the paper know the results. What is their process?

DavidSJ 11y ago
It means you have to develop your experimental method prior to seeing the data, so you can't try a dozen different ways of slicing that data up until you get the result you want.
- marcusgarvey 11y ago
  ...and now knowing this, I wish it were the approach for _all_ scientific research.
  - TeMPOraL 11y ago
    Yeah. This is what's absolutely missing in many scientific fields, and seems to be the reason for so many results turning out to be utter bullshit (hello, psychology). There's a strong push coming from research community to make this approach, which is known as 'study pre-registration', mandatory.
    - lucozade 11y ago
      Presumably pre-registration could also help with positive results bias. Assuming the registration was made public, one could mandate that something had to be produced from the study, if only a statement that nothing was found. Wouldn't solve the problem but would presumably aid meta-studies?
  - jessriedel 11y ago
    It's not really feasible for most research (although there are plenty of needed improvements along these lines). Big physics projects are unusual in that they are enormous undertakings of 100s or 1000s of PhDs, and few or no other experiments will take data that can cross-check the results. (For instance, no other machine will produce the conditions at the LHC for several decades at least, which is why they go to the fantastic expense of building two completely separate general purpose detectors.) Devoting ~5 full-time PhDs for the sake of super-duper methodological rigor is doable for LIGO, but not for smaller experiments.
  - amelius 11y ago
    Well, keep in mind that if hundreds of scientists take their chance on the same data using different methods, we basically have the same problem. One of them will be the lucky one that took the approach that shows the favorable results, whether the results are truly there or not.

Can you explain that a bit more? I'm fascinated but don't quite get the rationale behind having not even the authors of the paper know the results. What is their process?

DavidSJ 11y ago

It means you have to develop your experimental method prior to seeing the data, so you can't try a dozen different ways of slicing that data up until you get the result you want.

marcusgarvey 11y ago
...and now knowing this, I wish it were the approach for _all_ scientific research.
- TeMPOraL 11y ago
  Yeah. This is what's absolutely missing in many scientific fields, and seems to be the reason for so many results turning out to be utter bullshit (hello, psychology). There's a strong push coming from research community to make this approach, which is known as 'study pre-registration', mandatory.
  - lucozade 11y ago
    Presumably pre-registration could also help with positive results bias. Assuming the registration was made public, one could mandate that something had to be produced from the study, if only a statement that nothing was found. Wouldn't solve the problem but would presumably aid meta-studies?
- jessriedel 11y ago
  It's not really feasible for most research (although there are plenty of needed improvements along these lines). Big physics projects are unusual in that they are enormous undertakings of 100s or 1000s of PhDs, and few or no other experiments will take data that can cross-check the results. (For instance, no other machine will produce the conditions at the LHC for several decades at least, which is why they go to the fantastic expense of building two completely separate general purpose detectors.) Devoting ~5 full-time PhDs for the sake of super-duper methodological rigor is doable for LIGO, but not for smaller experiments.
- amelius 11y ago
  Well, keep in mind that if hundreds of scientists take their chance on the same data using different methods, we basically have the same problem. One of them will be the lucky one that took the approach that shows the favorable results, whether the results are truly there or not.