3 ms·

It struck me as really weird that he seemed to assume implicitly that his Blocksworld problems would be something an LLM should be able to do and presented the

It struck me as really weird that he seemed to assume implicitly that his Blocksworld problems would be something an LLM should be able to do and presented the failures as shocking; one look at them immediately gives me the impression that they are the kind of problem LLMs tend to be pathologically bad at. Probably he's better tuned in to the tone of the discourse than me but I feel like if you showed CoT helped with Blocksworld then that would be a huge and triumphant result, rather than something taken for granted that we must loudly disprove!

YeGoblynQueenne 2y ago
From the tweet: >> And yet the practitioners of CoT swear that any and every problem can be solved with LLM by giving it a bit of a CoT help. For example, see this arxiv paper: Generalized Planning in PDDL Domains with Pretrained Large Language Models https://arxiv.org/abs/2305.11014 https://arxiv.org/abs/2305.11014 Where the authors conclude: In this work, we showed that GPT-4 with CoT summarization and automated debugging is a surprisingly strong generalized planner in PDDL domains. The author of the tweet is an expert on planning and he's responding to that kind of thing.
- FeepingCreature 2y ago
  I think these scenarios are compatible if we view LLMs as "fragile reasoners": they can occasionally reason, but it is an intermittent state that is easily disturbed. In such a world, we would expect to see that people who want LLMs to work can make them work with difficulty, and people who want or expect LLMs to fail can make them fail easily - or rather, maybe less adversarially phrased, one can generate examples of either outcome.
- mistermann 2y ago
  >> And yet the practitioners of CoT swear that any and every problem... > Where the authors conclude: > ...is a surprisingly strong generalized planner Wtf is going on here?

3 ms·

It struck me as really weird that he seemed to assume implicitly that his Blocksworld problems would be something an LLM should be able to do and presented the

by amenhotep 2y ago

YeGoblynQueenne 2y ago
From the tweet: >> And yet the practitioners of CoT swear that any and every problem can be solved with LLM by giving it a bit of a CoT help. For example, see this arxiv paper: Generalized Planning in PDDL Domains with Pretrained Large Language Models https://arxiv.org/abs/2305.11014 https://arxiv.org/abs/2305.11014 Where the authors conclude: In this work, we showed that GPT-4 with CoT summarization and automated debugging is a surprisingly strong generalized planner in PDDL domains. The author of the tweet is an expert on planning and he's responding to that kind of thing.
- FeepingCreature 2y ago
  I think these scenarios are compatible if we view LLMs as "fragile reasoners": they can occasionally reason, but it is an intermittent state that is easily disturbed. In such a world, we would expect to see that people who want LLMs to work can make them work with difficulty, and people who want or expect LLMs to fail can make them fail easily - or rather, maybe less adversarially phrased, one can generate examples of either outcome.
- mistermann 2y ago
  >> And yet the practitioners of CoT swear that any and every problem... > Where the authors conclude: > ...is a surprisingly strong generalized planner Wtf is going on here?