4 ms·
"LLMs just interpolate their training data" Cracks me up. What exactly do we think that human brains do?
by paulddraper 5mo ago
"LLMs just interpolate their training data"
Cracks me up.
What exactly do we think that human brains do?
- gpugreg 5mo agoMaybe the human brain also does other things besides interpolation?
- paulddraper 5mo agoThere is pre-training, and then empirical observations. Yes?
- omnimus 5mo agoThat has been the question since the beginning of humans. Maybe computers can help understand better because by now it's pretty clear brains aren't just LLMs.
- deleted 5mo ago[deleted]
- ActorNightly 5mo agoI love this comment because it so clearly highlights the difference between intelligence and reasoning. A lot of people across all fields seem to operate in a mode of information lookup as intelligence. They have the memory of solving particular problems, and when faced with a new problem, they basically do a "nearest search" in their brain to find the most similar problem, and apply the same principles to it. While that works for a large number of tasks this intelligence is not the same as reasoning. Reasoning is the ability to discover new information that you haven't seen before (i.e growing a new branch on the knowledge tree instead of interpolating). Think of it like filling a space on the floor of arbitrary shape with smaller arbitrary shapes, trying to fill as much space as possible. With interpolation, your smaller shapes are medium size, each with a non rectangular shape. You may have a large library of them, but in the end, there are just certain floor spaces that you won't be able to fill fully. Reasoning on the flip side is having access to very fine shape, and knowing the procedure of how to stack shapes depending on what shapes are next to it and whether you are on a boundary of the floor space or not. Using these rules, you can fill pretty much any floor space fully.
- deleted 5mo ago[deleted]
- charlie90 5mo agoI agree. Humans are given a body that lets them "discover" things on accident, test out ideas, i.e. randomness. As in, I would hazard a guess the discovery of the wheel wasn't "pure intelligence", it was humans accidentally viewing a rock roll down a hill and getting an idea. If we give AI a "body", it will become as creative as humans are.
- __s 5mo agoCreativity is hard. Pretty much needs a fuzzer process to generate new strings, mostly nonsense, & pick up when that nonsense happens to be correct
- oh_my_goodness 5mo agoWe don't know what human brains do.
- fragmede 5mo agoWe have some idea.
- deleted 5mo ago[deleted]
- slashdave 5mo agoYou have to define what you mean by "interpolate". The mechanisms that LLM use are not mysterious, and they are not the same as used by humans.
- drdeca 5mo agoIf you interpret “interpolate” in the literal sense, and apply it to the mechanisms behind LLMs, then the claim that they only interpolate, is straightforwardly false. Taking it instead as a metaphorical claim may be more valid, but in that case it doesn’t depend on our understanding of how LLMs work.
- slashdave 5mo agoLLMs are statistical models by construction, so depending on how liberal you want to be with terminology, "interpolate" is not so bad. Might make a statistician upset.
- baq 5mo agoThey’re also capable of performing arbitrary computation when ran in a loop - so they can be made to quite literally interpolate whatever. Philosophers are quite upset, too.
- slashdave 5mo agoDefine "arbitrary". Without RAG, they screw up basic algebra.
- drdeca 5mo agoBut people aren’t giving a (less literal) definition of what they mean by “interpolate” that relies on the internal mechanisms of these models, just a vague metaphor, which, as this vague metaphor, there’s nothing it uses about LLMs that makes the question “do LLMs just interpolate” less of a type error than “do people just interpolate”. And I don’t think it’s a good metaphor.