4 ms·

Maybe there doing something we don’t really have a name for yet ? This is why it causes so much controversy. Not really generalizing, not memorizing, maybe app

Maybe there doing something we don’t really have a name for yet ? This is why it causes so much controversy. Not really generalizing, not memorizing, maybe approximating ?

dataflow 3y ago
"Vibing"?
- mirekrusin 3y ago
  No need to come up with any new name, we already have it: 1. autocompletion or 2. next token prediction and 3. (reverse) diffusion
ein0p 3y ago
There is a name for this already: associative memory. That’s how you catch a ball: you condition your memory of catching a ball with proprioceptive and visual input, much like a multimodal transformer. There’s no thinking involved - you wouldn’t be able to catch it if you had to think
- vidarh 3y ago
  An associative memory can model truth tables of logic circuits, and so the steps to go from associative memory to arbitrary computation pretty much requires adding state and a loop. And we do provide state and a loop. As such, while that does not prove that any given model is capable of reasoning, describing what goes on at any step as associative memory says nothing meaningful about the computational power of the wider system. It's very possible the way we train current models will turn out to place fundamental limits on the abilities the models will have, but that will not be because the models act as associative memories.
  - ein0p 3y ago
    I think the current models will be a fundamental part of larger, more comprehensive models, and they will handle the “easy” parts, much like we, humans, use our associative memory to be able to do things in realtime, “without thinking”. Yann LeCun wrote a paper on this which more people should read: https://openreview.net/pdf?id=BZ5a1r-kVsf https://openreview.net/pdf?id=BZ5a1r-kVsf. Moreover, Meta’s research is starting to move in that direction https://ai.meta.com/blog/v-jepa-yann-lecun-ai-model-video-joint-embedding-predictive-architecture/ https://ai.meta.com/blog/v-jepa-yann-lecun-ai-model-video-jo...
crotchfire 3y ago
> Not really generalizing, not memorizing, maybe approximating ? The term you are searching for is confabulating.
spuz 3y ago
There is certainly a lot of discussion around semantics to be had before we can agree on what exactly an LLM is capable of. We know that LLMs tend not to be good at math. Some people will say that the fact that an LLM cannot sum two numbers demonstrates they cannot generalise. Yet others would say the way LLMs calculate sums is generalisation because it mirrors our own process of addition which works mostly by memorisation and a lot of double checking. My perspective is that the "thinking" that LLMs do is a lot closer to the kind that humans do which is to say a lot of pattern matching but with no fundamentally precise logic underlying it. If LLMs are flawed in some way then humans are also flawed in the same way.
deleted 3y ago
[deleted]

Maybe there doing something we don’t really have a name for yet ? This is why it causes so much controversy. Not really generalizing, not memorizing, maybe approximating ?

dataflow 3y ago

"Vibing"?

mirekrusin 3y ago
No need to come up with any new name, we already have it: 1. autocompletion or 2. next token prediction and 3. (reverse) diffusion

ein0p 3y ago

There is a name for this already: associative memory. That’s how you catch a ball: you condition your memory of catching a ball with proprioceptive and visual input, much like a multimodal transformer. There’s no thinking involved - you wouldn’t be able to catch it if you had to think

vidarh 3y ago
An associative memory can model truth tables of logic circuits, and so the steps to go from associative memory to arbitrary computation pretty much requires adding state and a loop. And we do provide state and a loop. As such, while that does not prove that any given model is capable of reasoning, describing what goes on at any step as associative memory says nothing meaningful about the computational power of the wider system. It's very possible the way we train current models will turn out to place fundamental limits on the abilities the models will have, but that will not be because the models act as associative memories.
- ein0p 3y ago
  I think the current models will be a fundamental part of larger, more comprehensive models, and they will handle the “easy” parts, much like we, humans, use our associative memory to be able to do things in realtime, “without thinking”. Yann LeCun wrote a paper on this which more people should read: https://openreview.net/pdf?id=BZ5a1r-kVsf https://openreview.net/pdf?id=BZ5a1r-kVsf. Moreover, Meta’s research is starting to move in that direction https://ai.meta.com/blog/v-jepa-yann-lecun-ai-model-video-joint-embedding-predictive-architecture/ https://ai.meta.com/blog/v-jepa-yann-lecun-ai-model-video-jo...

crotchfire 3y ago

> Not really generalizing, not memorizing, maybe approximating ? The term you are searching for is confabulating.

spuz 3y ago

There is certainly a lot of discussion around semantics to be had before we can agree on what exactly an LLM is capable of. We know that LLMs tend not to be good at math. Some people will say that the fact that an LLM cannot sum two numbers demonstrates they cannot generalise. Yet others would say the way LLMs calculate sums is generalisation because it mirrors our own process of addition which works mostly by memorisation and a lot of double checking. My perspective is that the "thinking" that LLMs do is a lot closer to the kind that humans do which is to say a lot of pattern matching but with no fundamentally precise logic underlying it. If LLMs are flawed in some way then humans are also flawed in the same way.

deleted 3y ago

[deleted]