3 ms·
> So, you’re working on a new ML project. The first thing you do, model-wise – you implement a simple heuristic baseline and see where you stand. Second, you tr
by rmnclmnt 4y ago
> So, you’re working on a new ML project. The first thing you do, model-wise – you implement a simple heuristic baseline and see where you stand. Second, you try a simple ML solution and analyze how much it improves the baseline. One thing you can try to do after this stage, at least what I like to do, is to try to estimate what would be your upper bound in terms of predictive performance, and let an AutoML solution squeeze the most out of your data and preprocessing.
100% agreed! And yet, most data scientists completely jump over the first two steps described here because it is not "sexy enough" and not "valuable"
- sdsr 4y ago> estimate what would be your upper bound in terms of predictive performance Could you share how do you approach this estimation step of the predictive performance?
- aburla 4y agoBasically I assume a good AutoML solution, given time and enough compute, can come up with a very good model, much better than I can do in a few weeks, assuming no feature engineering. This way I can focus more on what could be missing and I don't really have to think "what if I used more estimators in that random forest" et al. There's no formal "estimation".
- aburla 4y agoI find having a simple baseline, and ideally a human baseline, extra important for a real world project. I had once the experience of working on a complicated project without any human baseline, even when requested. It was like walking through a dark forest without any lights. A lot of bruises.
- rmnclmnt 4y agoI like your "dark forest" analogy! With students I used to talk about "fog" or simply "doing things blindly", but the "dark forest" image makes everyone want to get out there ASAP!