3 ms·

The LLM has no idea what words mean, so its quite impossible for them to reason about things. They just statistically mash up patterns of words from massive in

• nl 4y ago
  > The LLM has no idea what words mean Well that isn't true for any sensible definition of "idea of what words mean". If you examine embeddings produced by LLMs you find similar words in similar locations in the multi-dimensional space. > They just statistically mash up patterns of words from massive input with no clue whatsoever what those patterns of words actually mean in the real world. This is simply not true. As multiple papers (eg [1]) have shown these models can do tasks that require chained reasoning and give results that can only be explained by this ability. There's even a hint of a possible explanation in that paper: > For certain tasks, there may be natural intuitions for why emergence requires a model larger than a particular threshold scale. For instance, if a multi-step reasoning task requires l steps of sequential computation, this might require a model with a depth of at least O(l) layers. Your understanding might have been correct for LSTMs trained before 2018 or so, and it was a reasonable model in the BERT era of early transformers. But it needs to be updated now with more recent results. [1] https://ai.googleblog.com/2022/11/characterizing-emergent-phenomena-in.html https://ai.googleblog.com/2022/11/characterizing-emergent-ph...
  • deleted 4y ago
    [deleted]
• nuancebydefault 4y ago
  Define "mean". Then show/prove how a computer algo like GPT does not have that ability.