2 ms·
In line with the other comments, let me give you a recent example from my work. We had a dataset with just about 30 records and about twice as many features, a
by matmatmatmat 4y ago
In line with the other comments, let me give you a recent example from my work.
We had a dataset with just about 30 records and about twice as many features, about 1:3 class balance between positive and negative. Not an impossible situation at all, but there was a big premium on performance, so we threw the book at it.
Lo and behold, SVM with some particular parameters comes out on top. Ship it, right?
Not so fast, what decision did the model learn? Turns out, the model was learning decision boundaries that did not make sense. It was overfitting because the model did not know about what kinds of decision boundaries were allowable.
In the end, someone looked at a 2D scatter plot, drew a line and called it a day. It took a fraction of the effort and time of running through a box of models. It'll save the company $MM and it's trivially explainable to engineers and execs.
AutoML with blind application would've completely botched this.
- timy2shoes 4y agoOr you have the classic case where the model trains on the wrong things (e.g. the classic husky/wolf and snow case). One example I heard was in heart attack risk prediction from x-rays, the model was focusing on the present of stents in the x-rays. Which is a trivial prediction because the doctor already knows the patient has a stent, so the model is not adding anything of value.