4 ms·

They've removed the part about ensembles and replaced it with a call for more "human understandable models." This at least addresses the premise of the article,

by tensor 10y ago

They've removed the part about ensembles and replaced it with a call for more "human understandable models." This at least addresses the premise of the article, though I'm not completely sold on it as a solution. > The problem here is knowing how well the model will work with radically (or even somewhat) different data than it was trained on. This is not the same as doing CV or hold out. This isn't a problem with validation techniques, but rather a problem with the data supplied to the algorithms. The established solution here is to make sure your test data actually resembles the data you intend the algorithms to work on. Regarding the new suggestion of using models that are easy for humans to understand, I suppose my question would be "how do you know that a model that a human feels good about actually generalizes any better?" In my experience things that seem to be right by intuition are often wrong. This is one of the advantages of machine learning approaches over rules based systems, the machine can learn things a human wouldn't think of. The other big advantage is being able to scale easily, perhaps harder if you are expecting a human to manually inspect every model. I'm definitely on board with their suggestion to continually monitor performance. There are some validation techniques that can help with that such as incremental validation.

nl 10y ago
The established solution here is to make sure your test data actually resembles the data you intend the algorithms to work on. That's an ideal solution, yes. But in practice - especially for deep neural networks that take weeks to train on pretty significant hardware - most people use pre-trained models, and attempt to retrain the last layer[1]. The other issue is of course the large amounts of supervised training data needed. [1] eg https://www.tensorflow.org/versions/r0.9/how_tos/image_retraining/index.html https://www.tensorflow.org/versions/r0.9/how_tos/image_retra...

by tensor 10y ago

nl 10y ago

The established solution here is to make sure your test data actually resembles the data you intend the algorithms to work on. That's an ideal solution, yes. But in practice - especially for deep neural networks that take weeks to train on pretty significant hardware - most people use pre-trained models, and attempt to retrain the last layer[1]. The other issue is of course the large amounts of supervised training data needed. [1] eg https://www.tensorflow.org/versions/r0.9/how_tos/image_retraining/index.html https://www.tensorflow.org/versions/r0.9/how_tos/image_retra...