4 ms·
Advancing NLP with Efficient Projection-Based Model Architectures
- bratao 6y agoThis is the exciting moment that I´m expecting. A transition to more efficient models that are near enough to SOTA
- datameta 6y agoAbsolutely. ML on Edge will be one of the defining technologies of this decade. Comms use many orders of magnitude more energy than chip operations and sensor reads on embedded devices. The key is to avoid sending all that data to ML models on the cloud and instead inference on the device itself.
- riku_iki 6y ago> that are near enough to SOTA Only on one task of text classification by 7 categories.
- palad1n 6y agoAre there any NLP techniques that are not neural net based?
- teruakohatu 6y agoLots and lots. NLP covers a very wide range of techniques. Even some 'fancy' techniques such as sentiment analysis can be done relatively simply without neural nets (eg VADER). On the low end tokenizing is a NLP technique eg. r"\w+"
- laingc 6y agoYes, but right now there is no technique not making use of Deep Learning that is remotely competitive. It really is that clear a distinction.
- palad1n 6y agoRight, that's really what I was asking. Thanks.
- SomewhatLikely 6y agoThis depends on the task. The non-neural techniques aren't making huge leaps in performance though the way neural methods are. There are tasks where traditional methods still excel or are at parity with neural methods, but the metrics haven't moved much in recent years so you don't see them in the news.
- teruakohatu 6y agoI am more excited about this in some ways than GPT-3. GPT-3 is amazing, but I can actually use a small high performing model.
- polymorph1sm 6y agoI recently found this paper[1] claiming near GPT-3 performance with only a fraction of parameters. They seems to simply reformulate the input sequence to change classification to a sequence generation task. Disclaimer, I am not affiliated to any of the authors [1] https://arxiv.org/pdf/2009.07118.pdf https://arxiv.org/pdf/2009.07118.pdf
- dave_sullivan 6y agoNew word vector scheme that learns segments of words rather than unique words... this cuts down on size of vocab and size of soft max at the end? No expensive attention mechanism? This cuts down on number of parameters needed quite a bit, especially when sequences get very long. I know what an RNN is but what is a QRNN? Will need to read this more closely. I thought the key research direction now was improving the efficiency of the self attention block but maybe not.
- bvm 6y agoQuasi: https://arxiv.org/abs/1611.01576 https://arxiv.org/abs/1611.01576