3 ms·
For those here that work on ML or data science teams in industry, do you use something analogous to preregistration? I think it's relatively common for practit
by abeppu 4y ago
For those here that work on ML or data science teams in industry, do you use something analogous to preregistration?
I think it's relatively common for practitioners to have an idea for some change to an existing model or system, be that a structural change, new feature, some preprocessing step, etc, develop it by iteratively making changes and running some pipeline, and stop when it looks like a measurable improvement has been achieved. Cut a PR, put in some graphs of the change in metrics of interest, merge, and start the process again. Any sequence of negative results encountered on the way are quietly presumed to be because something in the code or the setup wasn't right yet -- and often there were clear problems which get fixed along the way. But as a result, those negative outcomes often aren't kept or reexamined.
- timy2shoes 4y agoI think the equivalent is the test set. A test set is used to evaluate the performance of the model, after all the tinkering. As such, you need to take great care to make sure that you don't have train-test leakage (e.g. taking a random sample for stock market prediction when you need to do a time-based split), or that you don't use the test set more than once (e.g. choosing the best model based on the test set).