4 ms·

Far as I can tell, all this demonstrates is there's tasks that CoT doesn't work for (yet)? If that? Or just that there's bad prompts? Really, the Twitter replie

Far as I can tell, all this demonstrates is there's tasks that CoT doesn't work for (yet)? If that? Or just that there's bad prompts? Really, the Twitter replies have already made these points so there's nothing to add. This just seems like research that's been written conclusion-first. > Ever since I came across the Chain of Though (CoT) for LLMs paper, I wondered how it can possibly make sense given that there is little reason to believe that LLMs can follow procedures and unroll them for the current problem at hand. (After all, if they can do that, they should be able to do verification and general reasoning too--and we know they suck at those). I don't follow. Why would procedure following imply general reasoning?

amenhotep 2y ago
It struck me as really weird that he seemed to assume implicitly that his Blocksworld problems would be something an LLM should be able to do and presented the failures as shocking; one look at them immediately gives me the impression that they are the kind of problem LLMs tend to be pathologically bad at. Probably he's better tuned in to the tone of the discourse than me but I feel like if you showed CoT helped with Blocksworld then that would be a huge and triumphant result, rather than something taken for granted that we must loudly disprove!
- YeGoblynQueenne 2y ago
  From the tweet: >> And yet the practitioners of CoT swear that any and every problem can be solved with LLM by giving it a bit of a CoT help. For example, see this arxiv paper: Generalized Planning in PDDL Domains with Pretrained Large Language Models https://arxiv.org/abs/2305.11014 https://arxiv.org/abs/2305.11014 Where the authors conclude: In this work, we showed that GPT-4 with CoT summarization and automated debugging is a surprisingly strong generalized planner in PDDL domains. The author of the tweet is an expert on planning and he's responding to that kind of thing.
  - FeepingCreature 2y ago
    I think these scenarios are compatible if we view LLMs as "fragile reasoners": they can occasionally reason, but it is an intermittent state that is easily disturbed. In such a world, we would expect to see that people who want LLMs to work can make them work with difficulty, and people who want or expect LLMs to fail can make them fail easily - or rather, maybe less adversarially phrased, one can generate examples of either outcome.
  - mistermann 2y ago
    >> And yet the practitioners of CoT swear that any and every problem... > Where the authors conclude: > ...is a surprisingly strong generalized planner Wtf is going on here?
YeGoblynQueenne 2y ago
>> Far as I can tell, all this demonstrates is there's tasks that CoT doesn't work for (yet)? What he says is that they can solve blocks world problems if you take them by the hand and show them how to do it, for every single problem you wan them to solve. Which is not very useful at all. He's arguing that this is what CoT achieves: it takes the LLM by the hand and introduces domain knowledge that the user has in every step of the way, so that it's not the LLM that's solving anything but the person using the LLM with CoT. Do read the linked paper, it goes over this in more detail. It's a bit unfortunate that Rao chooses to communicate through the very noisy medium of twitter, but that's the internet.
- FeepingCreature 2y ago
  I think LLMs and CoT are powerful, but I agree with this description of what they do. The important step is that they demonstrate that there are no architectural limits that keep the LLM from acting in these domains, "only" knowledge/planning ones. Once we have a big enough dataset of CoT prompts, the model "just" has to generalize CoT prompting, not CoT following. It decomposes the problem into two halves, and demonstrates that if one half (instruction generation) is provided, the other (instruction following) becomes very tractable.
  - YeGoblynQueenne 2y ago
    Thanks for clarifying. That's not the claim that Rao is tweeting about though. Rao as far as I can tell is responding to a stream of papers that appeared in the last couple of years claiming that LLMs can plan right now, whether using CoT, or not. See the paper I linked to my reply to amenhotep (!) below for an example. As to blocks world planning in particular, LLMs already have plenty of examples of block stacking problems - those are the standard motivational experiment in planning papers, like solving mazes is for Reinforcement Learning. Google returns 10 pages of results for "blocks world planning". If LLMs were capable of generalising as well as you expect they will one day, they should already be capable of solving block stacking problems without CoT and with no guidance.
    - 2y ago

amenhotep 2y ago

It struck me as really weird that he seemed to assume implicitly that his Blocksworld problems would be something an LLM should be able to do and presented the failures as shocking; one look at them immediately gives me the impression that they are the kind of problem LLMs tend to be pathologically bad at. Probably he's better tuned in to the tone of the discourse than me but I feel like if you showed CoT helped with Blocksworld then that would be a huge and triumphant result, rather than something taken for granted that we must loudly disprove!

YeGoblynQueenne 2y ago
From the tweet: >> And yet the practitioners of CoT swear that any and every problem can be solved with LLM by giving it a bit of a CoT help. For example, see this arxiv paper: Generalized Planning in PDDL Domains with Pretrained Large Language Models https://arxiv.org/abs/2305.11014 https://arxiv.org/abs/2305.11014 Where the authors conclude: In this work, we showed that GPT-4 with CoT summarization and automated debugging is a surprisingly strong generalized planner in PDDL domains. The author of the tweet is an expert on planning and he's responding to that kind of thing.
- FeepingCreature 2y ago
  I think these scenarios are compatible if we view LLMs as "fragile reasoners": they can occasionally reason, but it is an intermittent state that is easily disturbed. In such a world, we would expect to see that people who want LLMs to work can make them work with difficulty, and people who want or expect LLMs to fail can make them fail easily - or rather, maybe less adversarially phrased, one can generate examples of either outcome.
- mistermann 2y ago
  >> And yet the practitioners of CoT swear that any and every problem... > Where the authors conclude: > ...is a surprisingly strong generalized planner Wtf is going on here?

YeGoblynQueenne 2y ago

>> Far as I can tell, all this demonstrates is there's tasks that CoT doesn't work for (yet)? What he says is that they can solve blocks world problems if you take them by the hand and show them how to do it, for every single problem you wan them to solve. Which is not very useful at all. He's arguing that this is what CoT achieves: it takes the LLM by the hand and introduces domain knowledge that the user has in every step of the way, so that it's not the LLM that's solving anything but the person using the LLM with CoT. Do read the linked paper, it goes over this in more detail. It's a bit unfortunate that Rao chooses to communicate through the very noisy medium of twitter, but that's the internet.

FeepingCreature 2y ago
I think LLMs and CoT are powerful, but I agree with this description of what they do. The important step is that they demonstrate that there are no architectural limits that keep the LLM from acting in these domains, "only" knowledge/planning ones. Once we have a big enough dataset of CoT prompts, the model "just" has to generalize CoT prompting, not CoT following. It decomposes the problem into two halves, and demonstrates that if one half (instruction generation) is provided, the other (instruction following) becomes very tractable.
- YeGoblynQueenne 2y ago
  Thanks for clarifying. That's not the claim that Rao is tweeting about though. Rao as far as I can tell is responding to a stream of papers that appeared in the last couple of years claiming that LLMs can plan right now, whether using CoT, or not. See the paper I linked to my reply to amenhotep (!) below for an example. As to blocks world planning in particular, LLMs already have plenty of examples of block stacking problems - those are the standard motivational experiment in planning papers, like solving mazes is for Reinforcement Learning. Google returns 10 pages of results for "blocks world planning". If LLMs were capable of generalising as well as you expect they will one day, they should already be capable of solving block stacking problems without CoT and with no guidance.
  - 2y ago