5 ms·

Really impressive effort to avoid the garden of forking paths. I wonder if (hope that) this sort of thing will become more standard in research.

exmadscientist 6y ago
It has been standard for a long time in particle physics. The experiments that I worked on were blinded in similar ways, though a bit less elaborately. On my would-be thesis experiment all the blinding turned out to be pointless... the signal was as bright as day. Always good when that happens!
Gatsky 6y ago
I think this is possible because of the exclusivity of the equipment. These scientists have the luxury of being the only ones able to conduct this experiment, and presumably some long run funding to do it. If there were 5 different teams competing, there would be less rigour I suspect, since other metrics (being first, number of papers, the usual academic swill) would come into play.
- mhh__ 6y ago
  I applied to do an internship working on air to air missiles for exactly this reason. Unluckily for them their online literacy says I'm too stupid to write simulation code them so their loss. But imagine the FPGA-grunt in those things!
dash2 6y ago
I've been working with an econometrician, what really impresses me is her insistence that we don't do any analyses until we've figured out what the right ones to do are. No exploring because it will taint your p values! It's a hard discipline to follow.
- PartiallyTyped 6y ago
  Makes sense really. From ML; the test data are an unbiased estimate of the performance of a model and you can use it get PAC (probably approximately correct) estimations of your performance. There are three (4) powers at play here, the confidence [1], the error bound [2], and the number of data points (and model complexity [3] but that is not relevant when we compare a single model against test data). Your performance is exponential in the number of samples you have (meaning more data gives you exponentially better bounds and more confidence), alternatively, for a fixed sample, you can have either high confidence, but weak bound, or small bound but low confidence. When we consider the training data as an estimate of the expected performance, we also take into consideration the class complexity. The moment you touch the test data you have tainted it because your brain can do a crude search on the space of solutions and reconcile them after the fact. This is why any preprocessing and cleaning is parameterized on the training set and then applied to the data, that also includes the class of models you can consider and is why you never select final models based on their performance on the test set; it becomes biased. [1] probability of getting a good model. [2] how much the empirical performance underestimates the expected performance. [3] the number of models you can select from a particular class of models.

exmadscientist 6y ago

It has been standard for a long time in particle physics. The experiments that I worked on were blinded in similar ways, though a bit less elaborately. On my would-be thesis experiment all the blinding turned out to be pointless... the signal was as bright as day. Always good when that happens!

Gatsky 6y ago

I think this is possible because of the exclusivity of the equipment. These scientists have the luxury of being the only ones able to conduct this experiment, and presumably some long run funding to do it. If there were 5 different teams competing, there would be less rigour I suspect, since other metrics (being first, number of papers, the usual academic swill) would come into play.

mhh__ 6y ago
I applied to do an internship working on air to air missiles for exactly this reason. Unluckily for them their online literacy says I'm too stupid to write simulation code them so their loss. But imagine the FPGA-grunt in those things!

dash2 6y ago

I've been working with an econometrician, what really impresses me is her insistence that we don't do any analyses until we've figured out what the right ones to do are. No exploring because it will taint your p values! It's a hard discipline to follow.

PartiallyTyped 6y ago
Makes sense really. From ML; the test data are an unbiased estimate of the performance of a model and you can use it get PAC (probably approximately correct) estimations of your performance. There are three (4) powers at play here, the confidence [1], the error bound [2], and the number of data points (and model complexity [3] but that is not relevant when we compare a single model against test data). Your performance is exponential in the number of samples you have (meaning more data gives you exponentially better bounds and more confidence), alternatively, for a fixed sample, you can have either high confidence, but weak bound, or small bound but low confidence. When we consider the training data as an estimate of the expected performance, we also take into consideration the class complexity. The moment you touch the test data you have tainted it because your brain can do a crude search on the space of solutions and reconcile them after the fact. This is why any preprocessing and cleaning is parameterized on the training set and then applied to the data, that also includes the class of models you can consider and is why you never select final models based on their performance on the test set; it becomes biased. [1] probability of getting a good model. [2] how much the empirical performance underestimates the expected performance. [3] the number of models you can select from a particular class of models.