4 ms·

That's true, but there's a lot of value in waking people up to the idea that ML works, even if what they're doing has worked for a long time. There are a lot o

That's true, but there's a lot of value in waking people up to the idea that ML works, even if what they're doing has worked for a long time. There are a lot of situations where before people would have assumed their best option is to carefully tweak a custom statistical model, whereas now they're just happy to throw a black box at it and see what happens. This is as much a cultural change as a technological change, and it's good that it is finally happening. That's what a "paradigm shift" is after all.

gubbrora 7y ago
I think something is lost when doing this. I'd bet the researcher who first builds a model and then reaches for ml will outperform the researcher who goes straight for ml. Building a custom model will help with feature selection. It will provide a baseline to compare the ml model to which can help debug problem points of the ml model. And finally it serves as a sanity check that you aren't leaving a lot of performance on the table.
mattkrause 7y ago
But why is "throwing a black box at it" good? The goal of research is usually to rip those boxes open to figure out what's inside and how it works. Moving away from that towards opaque predictions doesn't make a lot of sense to me, especially when the predictions aren't even that much better. Plus, a lot of this work seems weirdly disconnected from what the rest of the field knows to be (im)plausible. Obviously, black boxes can be useful tools. DeepLabCut is incredibly helpful and will save a lot of grad students a lot of tedium, and that probably wouldn't happen if it involved a lot of tuning. Predictions can also be very useful--frankly, we'll take anything we can get for most neuropsych conditions--but mechanisms and targets for intervention are so much more useful. I know there is some work on this, but it's drown out by the 0.99AUC!!1!! (in a small, cherrypicked group) stuff.
- hadsed 7y ago
  Black boxes are better than nothing. The ultimate black box is the universe, where experimental scientists can fiddle to try and understand. They give a great starting point to make progress, if that's what you want, and if not (like in some commerical applications) you have something that works (ish).

gubbrora 7y ago

I think something is lost when doing this. I'd bet the researcher who first builds a model and then reaches for ml will outperform the researcher who goes straight for ml. Building a custom model will help with feature selection. It will provide a baseline to compare the ml model to which can help debug problem points of the ml model. And finally it serves as a sanity check that you aren't leaving a lot of performance on the table.

mattkrause 7y ago

But why is "throwing a black box at it" good? The goal of research is usually to rip those boxes open to figure out what's inside and how it works. Moving away from that towards opaque predictions doesn't make a lot of sense to me, especially when the predictions aren't even that much better. Plus, a lot of this work seems weirdly disconnected from what the rest of the field knows to be (im)plausible. Obviously, black boxes can be useful tools. DeepLabCut is incredibly helpful and will save a lot of grad students a lot of tedium, and that probably wouldn't happen if it involved a lot of tuning. Predictions can also be very useful--frankly, we'll take anything we can get for most neuropsych conditions--but mechanisms and targets for intervention are so much more useful. I know there is some work on this, but it's drown out by the 0.99AUC!!1!! (in a small, cherrypicked group) stuff.

hadsed 7y ago
Black boxes are better than nothing. The ultimate black box is the universe, where experimental scientists can fiddle to try and understand. They give a great starting point to make progress, if that's what you want, and if not (like in some commerical applications) you have something that works (ish).