4 ms·
> zero ability to do language modeling If I am reading this correctly; then who invented/discovered attention networks ?
by neatze 3y ago
> zero ability to do language modeling
If I am reading this correctly; then who invented/discovered attention networks ?
- srslack 3y agoThe human being, who instructed the computer to use it to do the language modeling? What does attention have to do, exclusively, with language modeling?
- Jack000 3y agoHumans use language for communicating ideas, which is very different from language modeling. Here's some discussion on this topic: https://www.alignmentforum.org/posts/htrZrxduciZ5QaCjw/language-models-seem-to-be-much-better-than-humans-at-next https://www.alignmentforum.org/posts/htrZrxduciZ5QaCjw/langu... I would add that the above comparison is misleading, because humans have a massive advantage in that they have prior knowledge of what words mean. A more apples-to-apples comparison would have the human do next word prediction on a language they don't know. This would be akin to me giving you a few GBs of Chinese text, with no grounding or translation, then try to communicate with you in Chinese after you've read the whole thing.