3 ms·

This is truly the biggest breakthrough from DeepSeek - that an LLM can teach itself to reason, no human feedback needed. That’s nuts and brings forward the ide

This is truly the biggest breakthrough from DeepSeek - that an LLM can teach itself to reason, no human feedback needed. That’s nuts and brings forward the idea that an AI is close to self improvement.

EGreg 2y ago
We already knew that kind of stuff from AlphaZero vs AlphaGo AlphaGo Zero is a version of DeepMind's Go software AlphaGo. AlphaGo's team published an article in Nature in October 2017 introducing AlphaGo Zero, a version created without using data from human games, and stronger than any previous version.[1] By playing games against itself, AlphaGo Zero: surpassed the strength of AlphaGo Lee in three days by winning 100 games to 0; reached the level of AlphaGo Master in 21 days; and exceeded all previous versions in 40 days.[2] Training artificial intelligence (AI) without datasets derived from human experts has significant implications for the development of AI with superhuman skills, as expert data is "often expensive, unreliable, or simply unavailable."[3] Demis Hassabis, the co-founder and CEO of DeepMind, said that AlphaGo Zero was so powerful because it was "no longer constrained by the limits of human knowledge".[4] Furthermore, AlphaGo Zero performed better than standard deep reinforcement learning models (such as Deep Q-Network implementations[5]) due to its integration of Monte Carlo tree search. David Silver, one of the first authors of DeepMind's papers published in Nature on AlphaGo, said that it is possible to have generalized AI algorithms by removing the need to learn from humans.[6] Google later developed AlphaZero, a generalized version of AlphaGo Zero that could play chess and Shōgi in addition to Go.[7] In December 2017, AlphaZero beat the 3-day version of AlphaGo Zero by winning 60 games to 40, and with 8 hours of training it outperformed AlphaGo Lee on an Elo scale. AlphaZero also defeated a top chess program (Stockfish) and a top Shōgi program (Elmo).[8][9] Source: https://en.wikipedia.org/wiki/AlphaGo_Zero https://en.wikipedia.org/wiki/AlphaGo_Zero
- aurareturn 2y ago
  Indeed. But DeepSeek is the first to release a working repro for LLMs as far as I know.
- simlevesque 2y ago
  Yeah but LLM and game AIs are very different. An AI can easily tell if it won the game but an LLM can't know by itself if what they said was useful.
thorum 2y ago
It’s great, but it only works for problems where there is exactly one correct solution and it’s possible to automatically verify the solution - like math and programming. So far these reasoning models have not shown much transfer learning of reasoning to other domains, and are often worse at non-math/code tasks than standard models.
- datameta 2y ago
  Not sure if this a trivial or naive thought - is that perhaps because in non-discrete ideas there is more granularity of information encoded compared to a numerical solution? Separatelt, but on a related note - do we need analog or quantum computing to "truly" scale?
  - cluckindan 2y ago
    To ”truly” scale we need a system directly doing calculation on some fundamental property of matter. Quantum computing is _close_ but suffers from a lack of interpretability: how would we know if our quantum simulation of the universe would include the quantum computer doing the simulation, as well as another recursive copy of the simulation, and another, and another…
- onlyrealcuzzo 2y ago
  There are infinite solutions to coding problems. You could have lots of comments and pass statements and unnecessary conditionals. I don't think it matters that there is only one correct answer. It matters that you can verify reliably if the answer is correct enough.
- JKCalhoun 2y ago
  So, we have a very good left-brain model.
- vintermann 2y ago
  We can probably make it work at more nebulous problems by using a big LLM to judge the quality of the answers as well. It should be easier to recognise e.g. a really good poetic translation than to make one, and as long as that's true it could benefit from internal monologue "reasoning" in those domains as well.
segasaturn 2y ago
This makes ScaleAI obsolete right?
ricksunny 2y ago
A physics simulator's rules will be derivative of known physics. If we ask it to push the boundaries of known physics, then we can't verify it without real-world experiment. Absent that then it's basically Chegg on steroids: "Tell me, based on what we believe about how the universe works, about X." is the implicit preface to every physics question. To the extent that someone can figure out a way out of this epistemic box, I'm interested.

EGreg 2y ago

We already knew that kind of stuff from AlphaZero vs AlphaGo AlphaGo Zero is a version of DeepMind's Go software AlphaGo. AlphaGo's team published an article in Nature in October 2017 introducing AlphaGo Zero, a version created without using data from human games, and stronger than any previous version.[1] By playing games against itself, AlphaGo Zero: surpassed the strength of AlphaGo Lee in three days by winning 100 games to 0; reached the level of AlphaGo Master in 21 days; and exceeded all previous versions in 40 days.[2] Training artificial intelligence (AI) without datasets derived from human experts has significant implications for the development of AI with superhuman skills, as expert data is "often expensive, unreliable, or simply unavailable."[3] Demis Hassabis, the co-founder and CEO of DeepMind, said that AlphaGo Zero was so powerful because it was "no longer constrained by the limits of human knowledge".[4] Furthermore, AlphaGo Zero performed better than standard deep reinforcement learning models (such as Deep Q-Network implementations[5]) due to its integration of Monte Carlo tree search. David Silver, one of the first authors of DeepMind's papers published in Nature on AlphaGo, said that it is possible to have generalized AI algorithms by removing the need to learn from humans.[6] Google later developed AlphaZero, a generalized version of AlphaGo Zero that could play chess and Shōgi in addition to Go.[7] In December 2017, AlphaZero beat the 3-day version of AlphaGo Zero by winning 60 games to 40, and with 8 hours of training it outperformed AlphaGo Lee on an Elo scale. AlphaZero also defeated a top chess program (Stockfish) and a top Shōgi program (Elmo).[8][9] Source: https://en.wikipedia.org/wiki/AlphaGo_Zero https://en.wikipedia.org/wiki/AlphaGo_Zero

aurareturn 2y ago
Indeed. But DeepSeek is the first to release a working repro for LLMs as far as I know.
simlevesque 2y ago
Yeah but LLM and game AIs are very different. An AI can easily tell if it won the game but an LLM can't know by itself if what they said was useful.

thorum 2y ago

It’s great, but it only works for problems where there is exactly one correct solution and it’s possible to automatically verify the solution - like math and programming. So far these reasoning models have not shown much transfer learning of reasoning to other domains, and are often worse at non-math/code tasks than standard models.

datameta 2y ago
Not sure if this a trivial or naive thought - is that perhaps because in non-discrete ideas there is more granularity of information encoded compared to a numerical solution? Separatelt, but on a related note - do we need analog or quantum computing to "truly" scale?
- cluckindan 2y ago
  To ”truly” scale we need a system directly doing calculation on some fundamental property of matter. Quantum computing is _close_ but suffers from a lack of interpretability: how would we know if our quantum simulation of the universe would include the quantum computer doing the simulation, as well as another recursive copy of the simulation, and another, and another…
onlyrealcuzzo 2y ago
There are infinite solutions to coding problems. You could have lots of comments and pass statements and unnecessary conditionals. I don't think it matters that there is only one correct answer. It matters that you can verify reliably if the answer is correct enough.
JKCalhoun 2y ago
So, we have a very good left-brain model.
vintermann 2y ago
We can probably make it work at more nebulous problems by using a big LLM to judge the quality of the answers as well. It should be easier to recognise e.g. a really good poetic translation than to make one, and as long as that's true it could benefit from internal monologue "reasoning" in those domains as well.

segasaturn 2y ago

This makes ScaleAI obsolete right?

ricksunny 2y ago

A physics simulator's rules will be derivative of known physics. If we ask it to push the boundaries of known physics, then we can't verify it without real-world experiment. Absent that then it's basically Chegg on steroids: "Tell me, based on what we believe about how the universe works, about X." is the implicit preface to every physics question. To the extent that someone can figure out a way out of this epistemic box, I'm interested.