2 ms·
Darn, based on the title, I was hoping for an overview of recent research. Lots of people are having fun playing with GPT-3 or AI Dungeon, myself included, but
by skybrian 6y ago
Darn, based on the title, I was hoping for an overview of recent research.
Lots of people are having fun playing with GPT-3 or AI Dungeon, myself included, but it seems like there is other interesting research going on like the REALM paper [1], [2]. What should I be reading? Why aren't people talking about REALM more? I'm no expert, but it seems like keeping the knowledge base outside the language model has a lot going for it?
[1] https://ai.googleblog.com/2020/08/realm-integrating-retrieval-into.html https://ai.googleblog.com/2020/08/realm-integrating-retrieva...
[2] https://arxiv.org/abs/2002.08909 https://arxiv.org/abs/2002.08909
- nl 6y agoREALM is amazing, and Google's PEGASUS[1] taught itself to count between 2 and 5 (which is mindblowing). Basically the NLP groups from Google and FB[2][3] are always worth watching. [1] https://ai.googleblog.com/2020/06/pegasus-state-of-art-model-for.html https://ai.googleblog.com/2020/06/pegasus-state-of-art-model... [2] https://ai.facebook.com/blog/covost-v2-expanding-the-largest-most-diverse-multilingual-speech-to-text-translation-data-set/ https://ai.facebook.com/blog/covost-v2-expanding-the-largest... [3] https://ai.facebook.com/blog/introducing-a-new-large-scale-dynamic-data-set-to-push-the-limits-of-natural-language-processing/ https://ai.facebook.com/blog/introducing-a-new-large-scale-d...
- MiroF 6y agoI think there's a pretty large gap between what people in academia NLP/industry labs find interesting and what the bitcoing/VC/whatever people on twitter seem to find interesting. realm is definitely cool, and i agree that techniques like this are the way forward.