3 ms·

> Hash tables don't think, hash tables model conversations... They don't think, and they don't get very far in modeling conversations, either. Even the current

> Hash tables don't think, hash tables model conversations... They don't think, and they don't get very far in modeling conversations, either. Even the current LLMs are strictly better at modeling conversations than any hash table.

jameshart 4y ago
Actually since GPT is a function that takes a few thousand tokens as input and produces a deterministic probability distribution for ‘next token’, you could in theory just enumerate all possible input sequences, and precache all the corresponding results in a lookup hashtable. That hashtable would be ridiculously larger than the weighted model that constitutes GPT itself. But it at least theoretically could exist. GPT, for all its clever responses, could be replaced with that hashtable with no loss of fidelity. So it is not ‘better’ than a hashtable. It is much better than a hashtable of equivalent size.
- mannykannot 4y ago
  You could, in principle, use a hash table to implement what an LLM has 'learned' about token occurrences, but LLMs do not return the same response to a given prompt, while hash tables do. Consequently, Mjburgess' attempt to dismiss LLMs as mere hash tables is a flawed argument by analogy. And even if it were a justifiable analogy for LLMs, it does not follow that it applies to Turing-equivalent devices in general. A hash table is unequivocally not a Turing-equivalent device, even though it can be implemented by one. In fact, the more one argues that LLMs are ordinary technology, the greater the challenge they present to the notion that things like conversational language use are beyond such technologies. The most interesting thing about these models, in my opinion, lies in figuring out what their conceptual simplicity says about us.
  - jameshart 4y ago
    They return the same probability distribution.
    - mannykannot 4y ago
      I extended my reply before seeing your response, and my response to your reply is largely contained within that. LLMs do not just return a probability distribution (and their prompts are not questions about probability distributions.) It would be a category error to conflate something that is part of how they work for the totality of what they do. When you say "GPT, for all its clever responses, could be replaced with that hashtable with no loss of fidelity", where does one get all the prompts that will be given to it, in advance? A hash table does not respond to any input for which it has not already been given a response to return.
      - jameshart 4y ago
        > where does one get all the prompts that will be given to it, in advance? By exhaustive enumeration. I never said it was going to be quick
        mannykannot 4y ago
        Fair point, but still, LLMs do not invariably give the same continuation to the same prompt. Update: it is a significant difference that you don't have to do that with an LLM.
- tacitusarc 4y ago
  I don’t think this is obviously true… We don’t know for certain the nature of reality, and cannot positively claim that a hash table would give the same results as a sufficiently advanced predictor, because it supposes the predictor is deterministic. Assuming the predictor is advanced enough to be considered intelligent, that might not be the case.

jameshart 4y ago

Actually since GPT is a function that takes a few thousand tokens as input and produces a deterministic probability distribution for ‘next token’, you could in theory just enumerate all possible input sequences, and precache all the corresponding results in a lookup hashtable. That hashtable would be ridiculously larger than the weighted model that constitutes GPT itself. But it at least theoretically could exist. GPT, for all its clever responses, could be replaced with that hashtable with no loss of fidelity. So it is not ‘better’ than a hashtable. It is much better than a hashtable of equivalent size.

mannykannot 4y ago
You could, in principle, use a hash table to implement what an LLM has 'learned' about token occurrences, but LLMs do not return the same response to a given prompt, while hash tables do. Consequently, Mjburgess' attempt to dismiss LLMs as mere hash tables is a flawed argument by analogy. And even if it were a justifiable analogy for LLMs, it does not follow that it applies to Turing-equivalent devices in general. A hash table is unequivocally not a Turing-equivalent device, even though it can be implemented by one. In fact, the more one argues that LLMs are ordinary technology, the greater the challenge they present to the notion that things like conversational language use are beyond such technologies. The most interesting thing about these models, in my opinion, lies in figuring out what their conceptual simplicity says about us.
- jameshart 4y ago
  They return the same probability distribution.
  - mannykannot 4y ago
    I extended my reply before seeing your response, and my response to your reply is largely contained within that. LLMs do not just return a probability distribution (and their prompts are not questions about probability distributions.) It would be a category error to conflate something that is part of how they work for the totality of what they do. When you say "GPT, for all its clever responses, could be replaced with that hashtable with no loss of fidelity", where does one get all the prompts that will be given to it, in advance? A hash table does not respond to any input for which it has not already been given a response to return.
    - jameshart 4y ago
      > where does one get all the prompts that will be given to it, in advance? By exhaustive enumeration. I never said it was going to be quick
      - mannykannot 4y ago
        Fair point, but still, LLMs do not invariably give the same continuation to the same prompt. Update: it is a significant difference that you don't have to do that with an LLM.
tacitusarc 4y ago
I don’t think this is obviously true… We don’t know for certain the nature of reality, and cannot positively claim that a hash table would give the same results as a sufficiently advanced predictor, because it supposes the predictor is deterministic. Assuming the predictor is advanced enough to be considered intelligent, that might not be the case.