4 ms·

This perspective is a useful source of intuition against the “LLMs can’t have new ideas, they’re just next-token-predictors” style arguments. What if you shift

This perspective is a useful source of intuition against the “LLMs can’t have new ideas, they’re just next-token-predictors” style arguments. What if you shift your perspective to thinking of training as optimization over a vast parametrized family of compression algorithms? Well, it suddenly looks a lot more plausible that “new” “ideas” can emerge from that process!

throwaway_7274 2mo ago
Incidentally, the relationship is bidirectional. You can try it out just for fun. zstd is a pretty crappy language model :)
glial 2mo ago
> it suddenly looks a lot more plausible that “new” “ideas” can emerge from that process This is not intuitive to me. It seems like a "new idea" is something that (almost by definition) isn't in the training set. Can you elaborate a bit? Edit: but perhaps a good model could arise from training, which would be a good idea in the sense that parsimonious ideas are good scientific ideas.
- cyanydeez 2mo ago
  Once MP3s were invented, I had the idea for the Apple IPOD; but obviously I didn't have a giant manufacturing wing, the ability to make small hard drives, or anything else. I don't think Apple invented the ipod anymore than I invented it; LLMs likely would have also come to the same conclusion about an ipod like device. Original ideas either dont exist or have a functionally irrelevent definition in comparison with inputing tokens to LLMs to get novel ideas out.
- redhed 2mo ago
  How I see it, is if the human brain does lossy compression/prediction of the natural world that learns from its "training set" (sensory inputs) and we have been able to come up with new ideas, then it seems like AI would be able to as well.
- jbay808 2mo ago
  Imagine you're curve-fitting a bunch of data points on, say, the orbital motion of planets and asteroids. You get tons and tons of data on these orbital motions, and then put it into a huge black-box optimization algorithm that compresses the heck out of it. It compresess and compresses until it can't find a more compact representation, no matter how much more effort it applies. The output is a function, where you can provide an initial condition, and it gives you the predicted future orbital position at any requested time. Of course, one thing you get out of this is a great curve-fit for your existing data, which you can interpolate to find the position of any observed planet at any desired time. But could this function also succeed in predicting the orbital motion of objects that aren't in the dataset? If I spot a new comet, and put it into the compressed function, would I get an accurate prediction of its orbital motion, even though that object wasn't in the training data? The answer is "it depends, but probably yes". Newton's laws of orbital motion turn out to be simple compared to the size of the training data. So if the black-box compression has done a good job, it might output that function, or a close approximation of it. With a sufficient quantity of sufficiently accurate data, it might even improve on it; random errors can't be compressed, but where the deviations between observations and Newton's law turn out not to be random but rather the influence of an unobserved gravity source, or general relativity, the black-box algorithm will likely capture that as well. A lot of people seem to think of the training process as curve-fitting data (the "stochastic parrot" model), but I think of it more as "solving an inverse problem to approximate the unknown source that generated the training data". Machine learning has proven to be quite good at solving inverse problems, and this is just a very abstract one of them. (A forward-problem is something like solving for the electric fields from a set of charged particles; an inverse problem, https://en.wikipedia.org/wiki/Inverse_problem https://en.wikipedia.org/wiki/Inverse_problem, is one where you have data on the electric fields at various points and want to reconstruct the arrangement of charged particles that produced it. Or more generally, you have sampled data on the output of an unknown process, and want to reconstruct the process that produced the data). The inverse-problem-solving happens at the ML training step. The language model itself, that comes out of that, is solving the forward-problem: it has a generative-process baked in and now it's generating new data from it. But if the training process has done a good job of compression, it will certainly be able to generate valid new ideas that aren't in the training set, because the inverse model has solved for the underlying features of the real process that generated the training data, and those features can generate additional valid outputs that it wasn't trained on.
- throwaway_7274 2mo ago
  Wow, I didn’t even notice how badly I underexplained that! Yeah, thanks, your edit is what I had in mind. jbay808 explains it well, too.
- AnthonBerg 2mo ago
  An idea is a construction on existing concepts, a newer idea links more previously unlinked concepts.
- dboreham 2mo ago
  Therefore Einstein couldn't have had the idea of relativity because it wasn't in his training data. Joking aside, the reality is that a pre-Einstein in 1850 couldn't have had the idea of relativity because Maxwell wasn't in his training data. And a pre-Maxwell couldn't have had the idea of electromagnetism in 1820 because Faraday wasn't in his training data. What we call "new ideas" are actually just deduced from the available training data, which includes all the previous ideas and all experimental results. AI can do the exact same thing (I assert). Perhaps not today's AI, but some AI, probably not too far in the future.
nonameiguess 2mo ago
There's another element to this that I almost never see discussed. Ideas are not facts. Neither LLMs nor humans can generate new knowledge, as opposed to ideas, by thinking alone. Physical investigation and experimentation is necessary. The exception being pure mathematics since it exists solely in the realm of ideas. I'm willing to call that knowledge, but it's still a distinction, the old analytic/synthetic dichotomy of Kant.
- versteegen 2mo ago
  That conception of knowledge is interesting, but I think using the label 'knowledge' for it is very problematic, it's too far from common definitions. The fact that you have to carve out an exception for mathematics already shows there's a problem. Because if maths, shouldn't thought experiments also produce new knowledge? You're excluding special and general relativity. It seems to me that what the concept actually describes is "information about the world".
  - zahlman 2mo ago
    > The fact that you have to carve out an exception for mathematics already shows there's a problem. I don't see how that's an exceptional case. Mathematical theories may not correspond to physical objects, but they still have to be tested for consistency, logical validity etc. As for relativity, one could argue that Einstein's theories were indeed not "knowledge" until their predictions were confirmed (like, say, the gravitational lensing thing). The Michelson–Morley experiment preceded Einstein, but it had to be checked that the theory was consistent with that, and so on and so forth.
AnotherGoodName 2mo ago
Prediction is literally what's allowed computers to make amazingly creative chess and go moves that a human would never have thought of. More generally prediction allows you to path find towards a solution to reach some goal that no human might ever see. If someone states "LLMs can't have new ideas because they only predict" you have to laugh. Prediction literally enables new ideas as you use those predictions to path find to a goal.
- zahlman 2mo ago
  Being able to read out positions (minimax, alpha-beta pruning etc.) is not "prediction".
  - AnotherGoodName 2mo ago
    Tree pruning needs prediction to work and is merely part of the search of predicted spaces. The prediction is in the state evaluator that predicts how much closer to victory any position is. Chess and go absolutely have predictions being made by their value nets.

throwaway_7274 2mo ago

Incidentally, the relationship is bidirectional. You can try it out just for fun. zstd is a pretty crappy language model :)

glial 2mo ago

> it suddenly looks a lot more plausible that “new” “ideas” can emerge from that process This is not intuitive to me. It seems like a "new idea" is something that (almost by definition) isn't in the training set. Can you elaborate a bit? Edit: but perhaps a good model could arise from training, which would be a good idea in the sense that parsimonious ideas are good scientific ideas.

cyanydeez 2mo ago
Once MP3s were invented, I had the idea for the Apple IPOD; but obviously I didn't have a giant manufacturing wing, the ability to make small hard drives, or anything else. I don't think Apple invented the ipod anymore than I invented it; LLMs likely would have also come to the same conclusion about an ipod like device. Original ideas either dont exist or have a functionally irrelevent definition in comparison with inputing tokens to LLMs to get novel ideas out.
redhed 2mo ago
How I see it, is if the human brain does lossy compression/prediction of the natural world that learns from its "training set" (sensory inputs) and we have been able to come up with new ideas, then it seems like AI would be able to as well.
jbay808 2mo ago
Imagine you're curve-fitting a bunch of data points on, say, the orbital motion of planets and asteroids. You get tons and tons of data on these orbital motions, and then put it into a huge black-box optimization algorithm that compresses the heck out of it. It compresess and compresses until it can't find a more compact representation, no matter how much more effort it applies. The output is a function, where you can provide an initial condition, and it gives you the predicted future orbital position at any requested time. Of course, one thing you get out of this is a great curve-fit for your existing data, which you can interpolate to find the position of any observed planet at any desired time. But could this function also succeed in predicting the orbital motion of objects that aren't in the dataset? If I spot a new comet, and put it into the compressed function, would I get an accurate prediction of its orbital motion, even though that object wasn't in the training data? The answer is "it depends, but probably yes". Newton's laws of orbital motion turn out to be simple compared to the size of the training data. So if the black-box compression has done a good job, it might output that function, or a close approximation of it. With a sufficient quantity of sufficiently accurate data, it might even improve on it; random errors can't be compressed, but where the deviations between observations and Newton's law turn out not to be random but rather the influence of an unobserved gravity source, or general relativity, the black-box algorithm will likely capture that as well. A lot of people seem to think of the training process as curve-fitting data (the "stochastic parrot" model), but I think of it more as "solving an inverse problem to approximate the unknown source that generated the training data". Machine learning has proven to be quite good at solving inverse problems, and this is just a very abstract one of them. (A forward-problem is something like solving for the electric fields from a set of charged particles; an inverse problem, https://en.wikipedia.org/wiki/Inverse_problem https://en.wikipedia.org/wiki/Inverse_problem, is one where you have data on the electric fields at various points and want to reconstruct the arrangement of charged particles that produced it. Or more generally, you have sampled data on the output of an unknown process, and want to reconstruct the process that produced the data). The inverse-problem-solving happens at the ML training step. The language model itself, that comes out of that, is solving the forward-problem: it has a generative-process baked in and now it's generating new data from it. But if the training process has done a good job of compression, it will certainly be able to generate valid new ideas that aren't in the training set, because the inverse model has solved for the underlying features of the real process that generated the training data, and those features can generate additional valid outputs that it wasn't trained on.
throwaway_7274 2mo ago
Wow, I didn’t even notice how badly I underexplained that! Yeah, thanks, your edit is what I had in mind. jbay808 explains it well, too.
AnthonBerg 2mo ago
An idea is a construction on existing concepts, a newer idea links more previously unlinked concepts.
dboreham 2mo ago
Therefore Einstein couldn't have had the idea of relativity because it wasn't in his training data. Joking aside, the reality is that a pre-Einstein in 1850 couldn't have had the idea of relativity because Maxwell wasn't in his training data. And a pre-Maxwell couldn't have had the idea of electromagnetism in 1820 because Faraday wasn't in his training data. What we call "new ideas" are actually just deduced from the available training data, which includes all the previous ideas and all experimental results. AI can do the exact same thing (I assert). Perhaps not today's AI, but some AI, probably not too far in the future.

nonameiguess 2mo ago

There's another element to this that I almost never see discussed. Ideas are not facts. Neither LLMs nor humans can generate new knowledge, as opposed to ideas, by thinking alone. Physical investigation and experimentation is necessary. The exception being pure mathematics since it exists solely in the realm of ideas. I'm willing to call that knowledge, but it's still a distinction, the old analytic/synthetic dichotomy of Kant.

versteegen 2mo ago
That conception of knowledge is interesting, but I think using the label 'knowledge' for it is very problematic, it's too far from common definitions. The fact that you have to carve out an exception for mathematics already shows there's a problem. Because if maths, shouldn't thought experiments also produce new knowledge? You're excluding special and general relativity. It seems to me that what the concept actually describes is "information about the world".
- zahlman 2mo ago
  > The fact that you have to carve out an exception for mathematics already shows there's a problem. I don't see how that's an exceptional case. Mathematical theories may not correspond to physical objects, but they still have to be tested for consistency, logical validity etc. As for relativity, one could argue that Einstein's theories were indeed not "knowledge" until their predictions were confirmed (like, say, the gravitational lensing thing). The Michelson–Morley experiment preceded Einstein, but it had to be checked that the theory was consistent with that, and so on and so forth.

AnotherGoodName 2mo ago

Prediction is literally what's allowed computers to make amazingly creative chess and go moves that a human would never have thought of. More generally prediction allows you to path find towards a solution to reach some goal that no human might ever see. If someone states "LLMs can't have new ideas because they only predict" you have to laugh. Prediction literally enables new ideas as you use those predictions to path find to a goal.

zahlman 2mo ago
Being able to read out positions (minimax, alpha-beta pruning etc.) is not "prediction".
- AnotherGoodName 2mo ago
  Tree pruning needs prediction to work and is merely part of the search of predicted spaces. The prediction is in the state evaluator that predicts how much closer to victory any position is. Chess and go absolutely have predictions being made by their value nets.