3 ms·

This paper is interesting, but ultimately it's just restating that LLMs are statistical tools and not cognitive systems. The information-theoretic framing doesn

• Nevermark 1y ago
  > LLMs are statistical tools and not cognitive systems I have never understood broad statements that models are just (or mostly) statistical tools. Certainly statistics apply, minimizing mismatches results in mean (or similar measure) target predictions. But the architecture of a model is the difference between compressed statistics vs. forcing a model to translate information in a highly organized way reflecting the actual shape of the problem to get any accuracy at all. In both cases, statistics are relevant, but in the latter it's not a particularly insightful way to talk about what a model has learned. Statistical accuracy, prediction, etc. are basic problems to solve. The training criteria being optimized. But they don't limit the nature of solutions. They both leave problem difficulty, and solution sophistication unbounded.
• larodi 1y ago
  my take from the paper is that the authors overstate the following: LLMs cannot zoom in and out of certain details, while we humans subconsciously zoom in and out of context and even go through conceptual spaces in the very hermetic sense of it. LLMs do not. they work on the present context, which kickstarts them into expounding likeliness in the same contextual space.