4 ms·

What’s the difference between Derek Jeter and preregistration?

34 points by luu 4y ago

abeppu 4y ago
For those here that work on ML or data science teams in industry, do you use something analogous to preregistration? I think it's relatively common for practitioners to have an idea for some change to an existing model or system, be that a structural change, new feature, some preprocessing step, etc, develop it by iteratively making changes and running some pipeline, and stop when it looks like a measurable improvement has been achieved. Cut a PR, put in some graphs of the change in metrics of interest, merge, and start the process again. Any sequence of negative results encountered on the way are quietly presumed to be because something in the code or the setup wasn't right yet -- and often there were clear problems which get fixed along the way. But as a result, those negative outcomes often aren't kept or reexamined.
- timy2shoes 4y ago
  I think the equivalent is the test set. A test set is used to evaluate the performance of the model, after all the tinkering. As such, you need to take great care to make sure that you don't have train-test leakage (e.g. taking a random sample for stock market prediction when you need to do a time-based split), or that you don't use the test set more than once (e.g. choosing the best model based on the test set).
chirau 4y ago
What is preregistration?
aj7 4y ago
What is pre-registration? https://www.cos.io/initiatives/prereg https://www.cos.io/initiatives/prereg
andrewla 4y ago
Boy do I not agree with the substance of this article. I agree that yes, preregistration is not a panacea that will fix the reproducibility crisis or whatever. But the problem that it is solving is not to improve the quality of studies that get published. It only does that implicitly. What it does is allows us to factor in the unpublished studies. It gives us access to previously unpublishable null results. Would this have debunked the ESP research? No, that study would have gotten published because its results supported the expectations of the experimenters. BUT all the attempts to reproduce that would also have been pre-registered, and their lack of publishing (or the publishing of null results) would have effectively debunked the original.
- jrochkind1 4y ago
  The ESP research? Is a bigger story than some readers may realize. The "ESP guy" is J.B. Rhine. Not everyone knows/remembers that ESP was considered scientifically verified for a good couple 20th century decades, such that many people who considered themselves scientific thinkers felt they had to accept it's validity even though they had been suspicious. It was not just one study to be "debunked", it was literally a whole field (although granted one mostly controlled by Rhine, but that was not that unusual of scientific fields back then, not sure about now). Makes you think about the whole endeavor of science. On the other hand, it was eventually abandoned (not with a big debunking bang so much as just kind of forgotten). One thing it does is make me wonder what things today are accepted that won't be in the future. I'm not totally sure what the conclusions should be from the "ESP guy" thing. I think they include bigger questions about epistemology than simply "good research design" -- although on the other hand, research design was definitely a part of it, and pre-registration specifically actually specifically relevant to Rhine. One of the debunking claims of his critics is that he achieved his positive results by burying studies that didn't have the results he wanted (or even not being hygienic about what runs were part of what study), just keeping the ones with positive results. Pre-registration actually might have nipped him in the bud... (unless he was willing to be just plain fraudulent of course; there are few guards against that except discovering it; I think Rhine did not think he was being fraudulent, he was just conveniently sloppy in a way he didn't think mattered... to discover fraud you have to have a common understanding of what constitutes fraud, which pre-registration is part (just part) of developing I think)
  - moyix 4y ago
    I suspect the "ESP guy" is actually meant to refer to Daryl Bem? https://slate.com/health-and-science/2017/06/daryl-bem-proved-esp-is-real-showed-science-is-broken.html https://slate.com/health-and-science/2017/06/daryl-bem-prove... https://slatestarcodex.com/2014/04/28/the-control-group-is-out-of-control/ https://slatestarcodex.com/2014/04/28/the-control-group-is-o...
    - jrochkind1 4y ago
algoatecorn 4y ago
I know the article is just using Jeter as a foil, but here is a great video about how or why Jeter is often considered overrated and if it's actually true. https://youtu.be/4kiAHDbWeCw https://youtu.be/4kiAHDbWeCw
- altdataseller 4y ago
  Very very hard to argue that Jeter is overrated when he played in the most stressful, demanding market in all of baseball: New York (with an equally demanding boss in Steinbrenner). To be able to produce in the clutch as well as he did? I'm not sure there are many ppl who could do that.
  - msbarnett 4y ago
    Nevertheless it's very very easy to argue that Jeter is overrated if you stop evaluating with your eyes and dive into the metrics. He's got the lowest defensive WAR (-9.4) of any shortstop in the modern era – the objective outcomes of him playing in that position are quite literally worse than average, and worse than any other shortstop who played under Steinbrenner or in that media market (Didi Gregorius gets far less attention, but is a far better SS). People rate him highly based on watching his spectacular, dramatic diving catches – forgetting that those dives are the product of him being badly positioned on the hit, and that a correctly-positioned player would simply make an unremarkable routine catch on the same hit. Jeter's the baseball equivalent of the programmer who an Org is in love with for constantly spending nights and weekends fighting fires, failing to notice that it's the same person who's accidentally starting all of the fires to begin with.
    - theluketaylor 4y ago
      That’s all certainly true, and he should have moved to third for A-Rod, and yet he is still a hall of fame calibre player, which don’t exactly come around every day. Despite his -9.4 dWAR, his 96.3 oWAR still leaves him well beyond the 60 WAR HOF barrier. Jeter is such an odd player since he is simultaneously one of the best to ever play the game and massively overrated. I’m not sure there is another baseball player who fits that description.
selimthegrim 4y ago
The linked paper in the comments by Betancourt is here: https://betanalpha.github.io/assets/case_studies/principled_bayesian_workflow.html#23_Stay_Where_a_Kiss_Won't_Mislead_You https://betanalpha.github.io/assets/case_studies/principled_...
csours 4y ago
The products of science are a public good. The products of science are personal advancement in the field. The products of science have monetary value to the people who fund science. --- There are lots of reasons to "do science", but how does a particular person associated to a particular lab decide to do a particular study? Anyway, it's hard to take any study involving humans (health, diet, economics, psychology) seriously if it's not pre-registered. I think there should be funding specifically for pre-registered studies, but I'm not king of the world. Any journal in those fields should require preregistration, but it should also be funded.

34 points by luu 4y ago

For those here that work on ML or data science teams in industry, do you use something analogous to preregistration? I think it's relatively common for practitioners to have an idea for some change to an existing model or system, be that a structural change, new feature, some preprocessing step, etc, develop it by iteratively making changes and running some pipeline, and stop when it looks like a measurable improvement has been achieved. Cut a PR, put in some graphs of the change in metrics of interest, merge, and start the process again. Any sequence of negative results encountered on the way are quietly presumed to be because something in the code or the setup wasn't right yet -- and often there were clear problems which get fixed along the way. But as a result, those negative outcomes often aren't kept or reexamined.

timy2shoes 4y ago
I think the equivalent is the test set. A test set is used to evaluate the performance of the model, after all the tinkering. As such, you need to take great care to make sure that you don't have train-test leakage (e.g. taking a random sample for stock market prediction when you need to do a time-based split), or that you don't use the test set more than once (e.g. choosing the best model based on the test set).

chirau 4y ago

What is preregistration?

aj7 4y ago

What is pre-registration? https://www.cos.io/initiatives/prereg https://www.cos.io/initiatives/prereg

andrewla 4y ago

Boy do I not agree with the substance of this article. I agree that yes, preregistration is not a panacea that will fix the reproducibility crisis or whatever. But the problem that it is solving is not to improve the quality of studies that get published. It only does that implicitly. What it does is allows us to factor in the unpublished studies. It gives us access to previously unpublishable null results. Would this have debunked the ESP research? No, that study would have gotten published because its results supported the expectations of the experimenters. BUT all the attempts to reproduce that would also have been pre-registered, and their lack of publishing (or the publishing of null results) would have effectively debunked the original.

jrochkind1 4y ago
The ESP research? Is a bigger story than some readers may realize. The "ESP guy" is J.B. Rhine. Not everyone knows/remembers that ESP was considered scientifically verified for a good couple 20th century decades, such that many people who considered themselves scientific thinkers felt they had to accept it's validity even though they had been suspicious. It was not just one study to be "debunked", it was literally a whole field (although granted one mostly controlled by Rhine, but that was not that unusual of scientific fields back then, not sure about now). Makes you think about the whole endeavor of science. On the other hand, it was eventually abandoned (not with a big debunking bang so much as just kind of forgotten). One thing it does is make me wonder what things today are accepted that won't be in the future. I'm not totally sure what the conclusions should be from the "ESP guy" thing. I think they include bigger questions about epistemology than simply "good research design" -- although on the other hand, research design was definitely a part of it, and pre-registration specifically actually specifically relevant to Rhine. One of the debunking claims of his critics is that he achieved his positive results by burying studies that didn't have the results he wanted (or even not being hygienic about what runs were part of what study), just keeping the ones with positive results. Pre-registration actually might have nipped him in the bud... (unless he was willing to be just plain fraudulent of course; there are few guards against that except discovering it; I think Rhine did not think he was being fraudulent, he was just conveniently sloppy in a way he didn't think mattered... to discover fraud you have to have a common understanding of what constitutes fraud, which pre-registration is part (just part) of developing I think)
- moyix 4y ago
  I suspect the "ESP guy" is actually meant to refer to Daryl Bem? https://slate.com/health-and-science/2017/06/daryl-bem-proved-esp-is-real-showed-science-is-broken.html https://slate.com/health-and-science/2017/06/daryl-bem-prove... https://slatestarcodex.com/2014/04/28/the-control-group-is-out-of-control/ https://slatestarcodex.com/2014/04/28/the-control-group-is-o...
  - jrochkind1 4y ago

algoatecorn 4y ago

I know the article is just using Jeter as a foil, but here is a great video about how or why Jeter is often considered overrated and if it's actually true. https://youtu.be/4kiAHDbWeCw https://youtu.be/4kiAHDbWeCw

altdataseller 4y ago
Very very hard to argue that Jeter is overrated when he played in the most stressful, demanding market in all of baseball: New York (with an equally demanding boss in Steinbrenner). To be able to produce in the clutch as well as he did? I'm not sure there are many ppl who could do that.
- msbarnett 4y ago
  Nevertheless it's very very easy to argue that Jeter is overrated if you stop evaluating with your eyes and dive into the metrics. He's got the lowest defensive WAR (-9.4) of any shortstop in the modern era – the objective outcomes of him playing in that position are quite literally worse than average, and worse than any other shortstop who played under Steinbrenner or in that media market (Didi Gregorius gets far less attention, but is a far better SS). People rate him highly based on watching his spectacular, dramatic diving catches – forgetting that those dives are the product of him being badly positioned on the hit, and that a correctly-positioned player would simply make an unremarkable routine catch on the same hit. Jeter's the baseball equivalent of the programmer who an Org is in love with for constantly spending nights and weekends fighting fires, failing to notice that it's the same person who's accidentally starting all of the fires to begin with.
  - theluketaylor 4y ago
    That’s all certainly true, and he should have moved to third for A-Rod, and yet he is still a hall of fame calibre player, which don’t exactly come around every day. Despite his -9.4 dWAR, his 96.3 oWAR still leaves him well beyond the 60 WAR HOF barrier. Jeter is such an odd player since he is simultaneously one of the best to ever play the game and massively overrated. I’m not sure there is another baseball player who fits that description.

selimthegrim 4y ago

The linked paper in the comments by Betancourt is here: https://betanalpha.github.io/assets/case_studies/principled_bayesian_workflow.html#23_Stay_Where_a_Kiss_Won't_Mislead_You https://betanalpha.github.io/assets/case_studies/principled_...

csours 4y ago

The products of science are a public good. The products of science are personal advancement in the field. The products of science have monetary value to the people who fund science. --- There are lots of reasons to "do science", but how does a particular person associated to a particular lab decide to do a particular study? Anyway, it's hard to take any study involving humans (health, diet, economics, psychology) seriously if it's not pre-registered. I think there should be funding specifically for pre-registered studies, but I'm not king of the world. Any journal in those fields should require preregistration, but it should also be funded.