4 ms·

This is not even remotely close and very silly. A ChainOfThought in a loop. TreeOfThoughts is a more sophisticated method, see - https://arxiv.org/pdf/2305.1

This is not even remotely close and very silly. A ChainOfThought in a loop. TreeOfThoughts is a more sophisticated method, see - https://arxiv.org/pdf/2305.10601 https://arxiv.org/pdf/2305.10601 The clue we all had with OpenAI for a long time that this was a search through a tree, they hired Noam Brown, and his past work all hinted towards that. Q, is obviously a search on a tree like A. So take something like CoT, build out a tree, search for the best solution across it. The search is the "system-2 reasoning"

boulos 2y ago
Reminder: you need to escape the * otherwise you end up with emphasis (italics here).
zaptrem 2y ago
Where in their blog post (which seemingly had complete examples of the model’s chain of thought) did they suggest they were using search or tree of thoughts?
- Joeri 2y ago
  Just a guess: The chain of thought would be the final path through the tree. Interactively showing the thought tokens would give the game away, which is why they don’t show that.
- blackbear_ 2y ago
  They mention reinforcement learning, so I guess they used some sort of Monte Carlo tree search (the same algorithm used for AlphaGo). In this case, the model would explore several chain of thoughts during training, but only output a single chain during inference (as the sibling comment suggests).
  - whimsicalism 2y ago
    as someone who works in this field, this comment is obviously uninformed even about old public research trends
    - ricardobeat 2y ago
      Care to elaborate? Your comment would be a lot more useful if it included a little why. Otherwise it’s just teasing readers and at the same time smearing the author without anything to back it up.
      - whimsicalism 2y ago
        reinforcement learning with ppo doesn’t involve mcts and has been the bread and butter of aligning LLMs since 2020. nothing about saying they use rl implies mcts
        janalsncm 2y ago
        > nothing about saying they use rl implies they use mcts We can say the same thing about RL implying PPO, however there’s pretty big hints, namely Noam Brown being involved. Many of the things Noam Brown has worked on involve RL in tree search contexts. He has also been consistently advocating the use of additional test-time compute to solve search problems. This is also consistent with the messaging regarding the reasoning tokens. There is likely some learned tree search algorithm, such as a learned policy/value function as in AlphaGo. It’s all speculation until we have an actual paper. So we can’t categorically say MCTS/learned tree search isn’t involved.
- whimsicalism 2y ago
  nowhere lol
dinobones 2y ago
OAI revealed on Twitter that there is no "system" at inference time, this is just a model. Did they maybe expand to a tree during training to learn more robust reasoning? Maybe. But it still comes down to a regular transformer model at inference time.
- pizza 2y ago
  Source?
  - nell 2y ago
    > I wouldn't call o1 a "system". It's a model, but unlike previous models, it's trained to generate a very long chain of thought before returning a final answer https://x.com/polynoamial/status/1834641202215297487 https://x.com/polynoamial/status/1834641202215297487
    - astrange 2y ago
      That answer seems to conflict with "in the future we'd like to give users more control over the thinking time". I've gotten mini to think harder by asking it to, but it didn't make a better answer. Though now I've run out of usage limits for both of them so can't try any more…
      - qeternity 2y ago
        I'm not convinced there isn't more going on behind the scenes but influencing test-time compute via prompt is a pretty universal capability.
        whimsicalism 2y ago
        not in a way that it is effectively used - in real life all of the papers using CoT compare against a weak baseline and the benefits level off extremely quickly. nobody except for recent deepmind research has shown test time scaling like o1
COAGULOPATH 2y ago
Came here hoping to find this. You will not unlock "o1-like" reasoning by making a model think step by step. This is an old trick that people were using on GPT3 in 2020. If it were that simple, it wouldn't have taken OpenAI so long to release it. Additionally, some of the prompt seems counterproductive: >Be aware of your limitations as an llm and what you can and cannot do. The LLM doesn't have a good idea of its limitations (any more than humans do). I expect this will create false refusals, as the model becomes overcautious.
- anshumankmr 2y ago
  >The LLM doesn't have a good idea of its limitations (any more than humans do). I expect this will create false refusals, as the model becomes overcautious. Can it not be trained to do so? From my anecdotal observations, the knowledge cutoff is one thing that LLMs are really well trained to know about. Those are limitations that LLMs are currently well trained to handle. Why can it not be trained to know that it is quite frequently bad at math, it may produce sometimes inaccurate code etc. For humans also, some people know some things are just not their cup of tea. Sure there are times people may have half baked knowledge about things but one can tell if they are good at XYZ things, and not so much at other things.
  - regularfry 2y ago
    Apart from anything else there will be a lot of text about the nature of LLMs and their inherent limitations in its training set. It might only need to be made salient the fact that it is one in order to produce the required effect.
  - fudged71 2y ago
    It's a chicken and egg situation. You don't know a model's capabilities until it is trained. When you then change the training with that learning, it will have modified capabilities.
- Meganet 2y ago
  You actually don't know that. A LLM has a huge amount of data ingested. It can create character profiles, audience, personas etc. Why wouldn't it have potentially even learned to 'understand' what 'being aware of your limitations' means? Right now for me 'change of reasoning' feels a little bit of quering the existing meta space through the reasoning process to adjust weights. Basically priming the model. I would also not just call it a 'trick'. This looks simple, weird or whatnot but i do believe that this is part of AI thinking process research. Its a good question though what did they train? New Architecture? More parameters? Is this training a mix of experiments they did? Some auto optimization mechanism?
PROMISE_237 2y ago
[dead]
cubefox 2y ago
It's interesting that DeepMind still publishes this stuff. OpenAI doesn't publish anything of that sort anymore. DeepMind is more research/publication focused, but this is a disadvantage in a competitive landscape where OpenAI and Anthropic can just apply the results of your paper without giving anything back to the research community.
- cabidaher 2y ago
  Anthropic publishes quite a lot too though.
  - cubefox 2y ago
    On safety, but no longer on capabilities.
- marricks 2y ago
  > but this is a disadvantage in a competitive landscape Or it's a unique advantage because this stuff doesn't happen without good researches who may want: 1) Their name in scientific papers 2) They might actually care about the openess of AI
  - cubefox 2y ago
    So far it seems to be a disadvantage as DeepMind has fallen behind OpenAI, despite their size, and to some extent even behind Anthropic.
    - marricks 2y ago
      They feel behind because they didn't have the smart guy with a new idea a few years back, and HE decided to work at a place which started as open. Playing catch up and trying to attract talent from the hot-new-thing OpenAI requires incentives beyond lots of money. I contend actually being open helps. I'm sure that's one reason Facebook has an open source model, scientists can care about ethics and could be attracted to openness.
thelastparadise 2y ago
Another serious advantage of a tree search is parallelism.

boulos 2y ago

Reminder: you need to escape the * otherwise you end up with emphasis (italics here).

zaptrem 2y ago

Where in their blog post (which seemingly had complete examples of the model’s chain of thought) did they suggest they were using search or tree of thoughts?

Joeri 2y ago
Just a guess: The chain of thought would be the final path through the tree. Interactively showing the thought tokens would give the game away, which is why they don’t show that.
blackbear_ 2y ago
They mention reinforcement learning, so I guess they used some sort of Monte Carlo tree search (the same algorithm used for AlphaGo). In this case, the model would explore several chain of thoughts during training, but only output a single chain during inference (as the sibling comment suggests).
- whimsicalism 2y ago
  as someone who works in this field, this comment is obviously uninformed even about old public research trends
  - ricardobeat 2y ago
    Care to elaborate? Your comment would be a lot more useful if it included a little why. Otherwise it’s just teasing readers and at the same time smearing the author without anything to back it up.
    - whimsicalism 2y ago
      reinforcement learning with ppo doesn’t involve mcts and has been the bread and butter of aligning LLMs since 2020. nothing about saying they use rl implies mcts
      - janalsncm 2y ago
        > nothing about saying they use rl implies they use mcts We can say the same thing about RL implying PPO, however there’s pretty big hints, namely Noam Brown being involved. Many of the things Noam Brown has worked on involve RL in tree search contexts. He has also been consistently advocating the use of additional test-time compute to solve search problems. This is also consistent with the messaging regarding the reasoning tokens. There is likely some learned tree search algorithm, such as a learned policy/value function as in AlphaGo. It’s all speculation until we have an actual paper. So we can’t categorically say MCTS/learned tree search isn’t involved.
whimsicalism 2y ago
nowhere lol

dinobones 2y ago

OAI revealed on Twitter that there is no "system" at inference time, this is just a model. Did they maybe expand to a tree during training to learn more robust reasoning? Maybe. But it still comes down to a regular transformer model at inference time.

pizza 2y ago
Source?
- nell 2y ago
  > I wouldn't call o1 a "system". It's a model, but unlike previous models, it's trained to generate a very long chain of thought before returning a final answer https://x.com/polynoamial/status/1834641202215297487 https://x.com/polynoamial/status/1834641202215297487
  - astrange 2y ago
    That answer seems to conflict with "in the future we'd like to give users more control over the thinking time". I've gotten mini to think harder by asking it to, but it didn't make a better answer. Though now I've run out of usage limits for both of them so can't try any more…
    - qeternity 2y ago
      I'm not convinced there isn't more going on behind the scenes but influencing test-time compute via prompt is a pretty universal capability.
      - whimsicalism 2y ago
        not in a way that it is effectively used - in real life all of the papers using CoT compare against a weak baseline and the benefits level off extremely quickly. nobody except for recent deepmind research has shown test time scaling like o1

COAGULOPATH 2y ago

Came here hoping to find this. You will not unlock "o1-like" reasoning by making a model think step by step. This is an old trick that people were using on GPT3 in 2020. If it were that simple, it wouldn't have taken OpenAI so long to release it. Additionally, some of the prompt seems counterproductive: >Be aware of your limitations as an llm and what you can and cannot do. The LLM doesn't have a good idea of its limitations (any more than humans do). I expect this will create false refusals, as the model becomes overcautious.

anshumankmr 2y ago
>The LLM doesn't have a good idea of its limitations (any more than humans do). I expect this will create false refusals, as the model becomes overcautious. Can it not be trained to do so? From my anecdotal observations, the knowledge cutoff is one thing that LLMs are really well trained to know about. Those are limitations that LLMs are currently well trained to handle. Why can it not be trained to know that it is quite frequently bad at math, it may produce sometimes inaccurate code etc. For humans also, some people know some things are just not their cup of tea. Sure there are times people may have half baked knowledge about things but one can tell if they are good at XYZ things, and not so much at other things.
- regularfry 2y ago
  Apart from anything else there will be a lot of text about the nature of LLMs and their inherent limitations in its training set. It might only need to be made salient the fact that it is one in order to produce the required effect.
- fudged71 2y ago
  It's a chicken and egg situation. You don't know a model's capabilities until it is trained. When you then change the training with that learning, it will have modified capabilities.
Meganet 2y ago
You actually don't know that. A LLM has a huge amount of data ingested. It can create character profiles, audience, personas etc. Why wouldn't it have potentially even learned to 'understand' what 'being aware of your limitations' means? Right now for me 'change of reasoning' feels a little bit of quering the existing meta space through the reasoning process to adjust weights. Basically priming the model. I would also not just call it a 'trick'. This looks simple, weird or whatnot but i do believe that this is part of AI thinking process research. Its a good question though what did they train? New Architecture? More parameters? Is this training a mix of experiments they did? Some auto optimization mechanism?

PROMISE_237 2y ago

[dead]

cubefox 2y ago

It's interesting that DeepMind still publishes this stuff. OpenAI doesn't publish anything of that sort anymore. DeepMind is more research/publication focused, but this is a disadvantage in a competitive landscape where OpenAI and Anthropic can just apply the results of your paper without giving anything back to the research community.

cabidaher 2y ago
Anthropic publishes quite a lot too though.
- cubefox 2y ago
  On safety, but no longer on capabilities.
marricks 2y ago
> but this is a disadvantage in a competitive landscape Or it's a unique advantage because this stuff doesn't happen without good researches who may want: 1) Their name in scientific papers 2) They might actually care about the openess of AI
- cubefox 2y ago
  So far it seems to be a disadvantage as DeepMind has fallen behind OpenAI, despite their size, and to some extent even behind Anthropic.
  - marricks 2y ago
    They feel behind because they didn't have the smart guy with a new idea a few years back, and HE decided to work at a place which started as open. Playing catch up and trying to attract talent from the hot-new-thing OpenAI requires incentives beyond lots of money. I contend actually being open helps. I'm sure that's one reason Facebook has an open source model, scientists can care about ethics and could be attracted to openness.

thelastparadise 2y ago

Another serious advantage of a tree search is parallelism.