4 ms·
No, all you need to do is memorize enough vocabulary and have enough chances to get lucky on occasion. You can have better odds if you learn the conditional pro
by counters 4y ago
No, all you need to do is memorize enough vocabulary and have enough chances to get lucky on occasion. You can have better odds if you learn the conditional probability of that vocabulary given surrounding words.
And this is all that GPT is doing.
- dilap 4y agoI don't think so at all. The number of possible sentences is far too great to ever hope to accomplish the task (at some threshold level of difficulty) simply by having a "conditional probability table". Nor would it ever be reasonable to store or generate such a table. To do it it, you need a form of compression: reasoning, intelligence, knowledge.
- pwinnski 4y agoRight, for a human, presumably it would be easier to go ahead and learn the language. But that begs the question. We already know LLMs tokenize words, so they have definitely generated and stored such a table. The marvelous step forward from previous word-based solutions is that they store not just single words, not just pairs of words, but larger sequences of words, even up to short sentence length. It makes the table pretty large, as you suggest, but that's exactly we've been seeing, and each generation of GPT is that much larger than the one before.