3 ms·
I'm curious, are there papers in other ML fields that could be considered breakthroughs comparable in impact to AlexNet? For NLP the recent ELMo and BERT paper
by rococode 8y ago
I'm curious, are there papers in other ML fields that could be considered breakthroughs comparable in impact to AlexNet?
For NLP the recent ELMo and BERT papers for word embeddings come to mind, although their scope is somewhat different than AlexNet.
- LeanderK 8y agoMaybe LeNet from LeCun et. al? It was before my time, but the architecture is still influential. The more important paper would probably be Phoneme Recognition Using Time-Delay Neural Networks, which introduces Time-Delay NNs, from which ConvNets arise naturally. Other architectures, like Sepp Hochreiter; Jürgen Schmidhuber (1997): "Long short-term memory", should also be mentioned. Not sure which paper would be the most important for SVMs. I don't know what to do about concepts that took a long time to develop, like backpropagation. Research is usually incremental.
- bkanber 8y agoSure, lots! We've made lots of progress just in the last decade or two. LSTM, recurrent, and convolutional nets. Slightly older (late 90's), but I think Random Forests were a pretty significant breakthrough. Another huge one is Paul Graham's own work using Naive Bayes to filter spam.
- p1esk 8y agoBoth convolutional and recurrent architectures were developed in late 80s. LSTM (improved RNN) was published in 1997.
- Khaki 8y agoMost research looks more incremental than people think. AlexNet is based on much work by Fukushima, Yann LeCun and others. Likewise, ELMO and BERT are based on 2015 work on pretrained language models: https://papers.nips.cc/paper/5949-semi-supervised-sequence-learning https://papers.nips.cc/paper/5949-semi-supervised-sequence-l...