4 ms·
This readme lands to me like this: "People say deep learning killed stats, but that's not true; in fact, DL can be a huge mistake." Ok, I fully agree with thei
by tylerneylon 4y ago
This readme lands to me like this: "People say deep learning killed stats, but that's not true; in fact, DL can be a huge mistake."
Ok, I fully agree with their foundational premise: Start simple.
But: They've overstated their case a bit. Saying that deep learning will cost $11,000 and need 14 days on this data set is not reasonable. I believe you can find some code that will cost that much. The readme suggests that this is typical of deep learning, which is not true. DL models have enormous variety. You can train a useful, high-performance model on a laptop CPU in a seconds-to-minutes timeframe; examples include multilayer perceptrons for simple classification, a smaller-scale CNN, or a collaborative filtering model.
While I don't endorse all details of their argument, I do think the culture of applied ML/data science has shifted too far toward default-DL. The truth is that many problems faced by real companies can be solved with simple techniques or pre-trained models.
Another perspective: A DL model is a spacecraft (expensive, sophisticated, powerful). Simple models like logistic regression are bikes and cars (affordable, efficient, less powerful). Using heuristics is like walking. Often your goal is just a few blocks away, in which case it would be inefficient to use a spacecraft.
- sigmoid10 4y ago>They've overstated their case a bit. Saying that deep learning will cost $11,000 and need 14 days on this data set is not reasonable. After glancing at the paper they're criticising, I really wonder how they arrived at these insane figures. From what I saw, they were mostly using stuff like MLPs with a handful layers at O(100) neurons at most. Yeah, if you put a hundred million parameter transformer in there you will train forever (and waste tons of compute since that would be complete overkill), but not with simple perceptrons. I don't know the extent of the data, but given these architectures I very much doubt a practical model would take this long to train - even on a CPU - given that you could run a statistical ensemble in 5 minutes.