4 ms·

I think the easiest way to get the gist of what he (and I) are arguing is a thought experiment. Take an LLM back to the dawn of mankind, and train it on all of

I think the easiest way to get the gist of what he (and I) are arguing is a thought experiment. Take an LLM back to the dawn of mankind, and train it on all of the collective knowledge of man. This would actually be impossible, because language hadn't yet been invented, so let's just imagine it could somehow read the collective mind of humanity. So cutting edge technology is poke 'em with the pointy side, science is nonexistent, and knowledge isn't that far behind. Now run this system through our LLM for as long as you want with as much power as you want. Where is it going to take you? Nowhere really, it's still just going to be constrained to its pool of data, recombining things in primitive ways and be essentially permanently stuck in the present, until somebody gives it some new data to train on it and mix and match. Yet somehow humanity, starting from that exact same basis, would soon (relatively speaking) put a man on the Moon, unlock the secrets of the atom, discover quantum mechanics, invent/discover mathematics, and much more. This is what is fundamentally meant by LLMs cannot create "new" knowledge. They absolutely can mix and match their pool of training statements in ways that can generate new statements. But if it's not an extremely simple 'remix' and we're in a domain where there are right and wrong answers, there's good chance it's just a nonsensical hallucination.

Zambyte 2y ago
You are comparing a species going from developing language to building spaceships on the order of hundreds of thousands of years, to a technology that was invented 7 years ago and only started to really be taken seriously 4 years ago. Working memory systems required for learning and iteratively developing ideas are in their infancy (on the scale of recent developments), but the recall technology (vector databases a la RAG) is quite well proven. I see no reason that a language model couldn't do iterative science in the same way humans have been with the resources available to it (APIs) using the composition of current technologies.
- somenameforme 2y ago
  The current technology for LLMs is still a rather complex guess the next word algorithm. But this leaves the software with no real room to ever move beyond its training, regardless of how much time or training you give it. You could give them literally infinite processing power and they would not suddenly start developing new meaningful knowledge - it would still be little more than simple recombinations of its training dataset, until somebody gave it something new to train upon.
- xracy 2y ago
  > I see no reason that a language model couldn't do iterative science in the same way humans have been with the resources available to it (APIs) using the composition of current technologies. I think this belies a common refrain from people in non-tech feeling like people in tech tend to oversimplify their fields and claim they can "fix any problem" without any actual specific knowledge in that field. This just feels like the next iteration of that.
  - Zambyte 2y ago
    This seems like a lot of words to just say "you are wrong". Can you explain why? What features of the scientific method cannot be achieved through the composition of existing technologies? Feel free to be specific and use technical terms.
    - xracy 2y ago
      I think you're misreading this. I'm not saying "you are wrong". I'm saying there's a common trap that a lot of people in tech (and therefore a lot of people on HN fall into), where they believe that they can solve any problem from what they know of basic tech problems. > What features of the scientific method cannot be achieved through the composition of existing technologies? ^This statement, sums up the above trap perfectly. "I know how to create a workflow, therefore, I know how to create a workflow of the scientific method, therefore I can replace a scientist with an AI robot." But okay, how do we know what quantitatively, is a significant result? R=0.05 is the most common cut-off for this, but like, that's a number we picked, and isn't even agreed on.
wongarsu 2y ago
LLMs can absolutely create synthesize new knowledge out of existing knowledge. They can't easily do so iteratively because we haven't quite figured out memory yet. Until we figure that out you won't have an LLM discover a new theory of quantum gravity. And even once we solve that, LLMs - just like human scientists - absolutely need new data from the outside world. Very few breakthroughs were achieved by just thinking about it long and hard, most were the the result of years of experimentation. Something LLMs simply can't do
- somenameforme 2y ago
  Recursive self improvement requires the ability to objectively select the "prime" of your own knowledge. From an LLM's perspective a hallucination and a correct answer are the same thing. It does not have any beliefs about what is true or false, because it has no concept of what it's even outputting. It's all just guess the next word. So even if the hallucination completely contradicts countless things it ostensibly "knows" to be true, it is unable to correct itself or realize that what it's outputting is unlikely to be correct.
  - jsnell 2y ago
    Why are humans capable of doing that, but it's categorically impossible for LLMs? That's a very high level capability. What's the primitive it is built on that humans have but machines can't have?
    - GreatLdisisp88 2y ago
      Quantum biomechanics. Human brain has built-in indeterministic pattern seeking. Computers are Turing based which 100% dependent on their programming and data inputs. All current LLMs operates on that very deterministic hardware. In order for LLMs to break thru existing glass ceiling, the hardware need to change.
  - wongarsu 2y ago
    Pause tokens [1] are directly dependent on the model recognizing whether the answer it arrived at is one it should "commit" to or whether it should abandon it and output a pause token instead. Similarly if you ask ChatGPT about the current president of Numbitistan it will tell you that it doesn't know about a county with that name, rather than just hallucinating an answer. So it can at least in this circumstance tell the difference between knowing something and not knowing something. 1: https://arxiv.org/abs/2310.02226 https://arxiv.org/abs/2310.02226
sitkack 2y ago
> Take an LLM back to the dawn of mankind, and train it on all of the collective knowledge of man. ... and that is the start of your argument. Could it also have gnomes and robots?
- xracy 2y ago
  That's not the start of an argument. It's the start of them explaining their point.
  - sitkack 2y ago
    Explaining a point is part of an argument. And the premise is unsound and illogical and if we had an LLM firewall their post would have been blocked until they cleaned up their reasoning.
tim333 2y ago
There's more to AI than LLMs though, like Alpha Fold figuring the structure of 200 million proteins for example. You can link those systems to your LLM so you can ask it what do you think of this protein and what experiment could I do on it and so on. There are many such avenues to explore. I don't see things getting stuck at a maxima for quite a while.

Zambyte 2y ago

You are comparing a species going from developing language to building spaceships on the order of hundreds of thousands of years, to a technology that was invented 7 years ago and only started to really be taken seriously 4 years ago. Working memory systems required for learning and iteratively developing ideas are in their infancy (on the scale of recent developments), but the recall technology (vector databases a la RAG) is quite well proven. I see no reason that a language model couldn't do iterative science in the same way humans have been with the resources available to it (APIs) using the composition of current technologies.

somenameforme 2y ago
The current technology for LLMs is still a rather complex guess the next word algorithm. But this leaves the software with no real room to ever move beyond its training, regardless of how much time or training you give it. You could give them literally infinite processing power and they would not suddenly start developing new meaningful knowledge - it would still be little more than simple recombinations of its training dataset, until somebody gave it something new to train upon.
xracy 2y ago
> I see no reason that a language model couldn't do iterative science in the same way humans have been with the resources available to it (APIs) using the composition of current technologies. I think this belies a common refrain from people in non-tech feeling like people in tech tend to oversimplify their fields and claim they can "fix any problem" without any actual specific knowledge in that field. This just feels like the next iteration of that.
- Zambyte 2y ago
  This seems like a lot of words to just say "you are wrong". Can you explain why? What features of the scientific method cannot be achieved through the composition of existing technologies? Feel free to be specific and use technical terms.
  - xracy 2y ago
    I think you're misreading this. I'm not saying "you are wrong". I'm saying there's a common trap that a lot of people in tech (and therefore a lot of people on HN fall into), where they believe that they can solve any problem from what they know of basic tech problems. > What features of the scientific method cannot be achieved through the composition of existing technologies? ^This statement, sums up the above trap perfectly. "I know how to create a workflow, therefore, I know how to create a workflow of the scientific method, therefore I can replace a scientist with an AI robot." But okay, how do we know what quantitatively, is a significant result? R=0.05 is the most common cut-off for this, but like, that's a number we picked, and isn't even agreed on.

wongarsu 2y ago

LLMs can absolutely create synthesize new knowledge out of existing knowledge. They can't easily do so iteratively because we haven't quite figured out memory yet. Until we figure that out you won't have an LLM discover a new theory of quantum gravity. And even once we solve that, LLMs - just like human scientists - absolutely need new data from the outside world. Very few breakthroughs were achieved by just thinking about it long and hard, most were the the result of years of experimentation. Something LLMs simply can't do

somenameforme 2y ago
Recursive self improvement requires the ability to objectively select the "prime" of your own knowledge. From an LLM's perspective a hallucination and a correct answer are the same thing. It does not have any beliefs about what is true or false, because it has no concept of what it's even outputting. It's all just guess the next word. So even if the hallucination completely contradicts countless things it ostensibly "knows" to be true, it is unable to correct itself or realize that what it's outputting is unlikely to be correct.
- jsnell 2y ago
  Why are humans capable of doing that, but it's categorically impossible for LLMs? That's a very high level capability. What's the primitive it is built on that humans have but machines can't have?
  - GreatLdisisp88 2y ago
    Quantum biomechanics. Human brain has built-in indeterministic pattern seeking. Computers are Turing based which 100% dependent on their programming and data inputs. All current LLMs operates on that very deterministic hardware. In order for LLMs to break thru existing glass ceiling, the hardware need to change.
- wongarsu 2y ago
  Pause tokens [1] are directly dependent on the model recognizing whether the answer it arrived at is one it should "commit" to or whether it should abandon it and output a pause token instead. Similarly if you ask ChatGPT about the current president of Numbitistan it will tell you that it doesn't know about a county with that name, rather than just hallucinating an answer. So it can at least in this circumstance tell the difference between knowing something and not knowing something. 1: https://arxiv.org/abs/2310.02226 https://arxiv.org/abs/2310.02226

sitkack 2y ago

> Take an LLM back to the dawn of mankind, and train it on all of the collective knowledge of man. ... and that is the start of your argument. Could it also have gnomes and robots?

xracy 2y ago
That's not the start of an argument. It's the start of them explaining their point.
- sitkack 2y ago
  Explaining a point is part of an argument. And the premise is unsound and illogical and if we had an LLM firewall their post would have been blocked until they cleaned up their reasoning.

tim333 2y ago

There's more to AI than LLMs though, like Alpha Fold figuring the structure of 200 million proteins for example. You can link those systems to your LLM so you can ask it what do you think of this protein and what experiment could I do on it and so on. There are many such avenues to explore. I don't see things getting stuck at a maxima for quite a while.