3 ms·
You could, in principle, use a hash table to implement what an LLM has 'learned' about token occurrences, but LLMs do not return the same response to a given pr
by mannykannot 4y ago
You could, in principle, use a hash table to implement what an LLM has 'learned' about token occurrences, but LLMs do not return the same response to a given prompt, while hash tables do. Consequently, Mjburgess' attempt to dismiss LLMs as mere hash tables is a flawed argument by analogy.
And even if it were a justifiable analogy for LLMs, it does not follow that it applies to Turing-equivalent devices in general. A hash table is unequivocally not a Turing-equivalent device, even though it can be implemented by one.
In fact, the more one argues that LLMs are ordinary technology, the greater the challenge they present to the notion that things like conversational language use are beyond such technologies. The most interesting thing about these models, in my opinion, lies in figuring out what their conceptual simplicity says about us.
- jameshart 4y agoThey return the same probability distribution.
- mannykannot 4y agoI extended my reply before seeing your response, and my response to your reply is largely contained within that. LLMs do not just return a probability distribution (and their prompts are not questions about probability distributions.) It would be a category error to conflate something that is part of how they work for the totality of what they do. When you say "GPT, for all its clever responses, could be replaced with that hashtable with no loss of fidelity", where does one get all the prompts that will be given to it, in advance? A hash table does not respond to any input for which it has not already been given a response to return.
- jameshart 4y ago> where does one get all the prompts that will be given to it, in advance? By exhaustive enumeration. I never said it was going to be quick
- mannykannot 4y agoFair point, but still, LLMs do not invariably give the same continuation to the same prompt. Update: it is a significant difference that you don't have to do that with an LLM.