5 ms·

My experience interacting with chain-of-thought is that it should not be likened to the rigid chains of logic/math. Step-by-step reasoning by models isn’t magic

by meroes 3y ago

My experience interacting with chain-of-thought is that it should not be likened to the rigid chains of logic/math. Step-by-step reasoning by models isn’t magically imparting that much rigidity to their outputs. The strength of the chain is the strength of related contexts, which is to say much less than math/logic done by humans. We tell ourselves we are teaching AI to do step-by-step reasoning, but admittedly as someone who deals with models daily in this area and not programming them, I don’t see the tight necessary connections we teach in basic math because I see how much the model(s) fail in ways no human past a certain age could. It’s more of a search for related contexts, which is powerful, but again not how a human reasons logically. Humans can reason purely form the armchair, starting with very few concepts, and reach far far, ironclad conclusions. Models aren’t doing that. They are leapfrogging through context. Yes an argument can be that’s splitting hairs, but that’s because it’s hard to describe succinctly, not hard to see.

exe34 3y ago
I think a lot of what humans think of as "1. 2. Therefore 3." kind of reasoning isn't different from what the llm is doing, and not in fact any more clever than that. Plenty of people believe plenty of questionable things that they assume they have thought through but really haven't. They used the context to guess the next idea/word, often reaching the conclusions they started out with. When you talk about ironclad conclusions, I think what happens is that we come up with those confabulations intuitively, but then we subject them to intense checking - have we defined everything clearly enough, is that leap in reasoning justified, etc. So what I'd really like to see is a way to teach llms to take a vague English sentence and transform it into a form that can be run through a more formal reasoning engine. Often instead of asking an llm to tell you something like how many football fields could you fit inside England, you are better off telling it to write python code to do this, assume get_size_football_field() in m^2 and get_size_England() in m^2 is available.
- doctoboggan 3y ago
  > I think a lot of what humans think of as "1. 2. Therefore 3." kind of reasoning isn't different from what the llm is doing, and not in fact any more clever than that. Plenty of people believe plenty of questionable things that they assume they have thought through but really haven't. They used the context to guess the next idea/word, often reaching the conclusions they started out with. Agreed that many/most humans behave this way, but some do not. And those who do not are the ones advancing the boundaries of knowledge and it would be very nice if we could get our LLMs to behave in the same way.
  - exe34 3y ago
    That's what I mean though, people don't just come up with the right answer out of nowhere. They think through many possibilities generatively, based on "intuition", and most of what they come up with is rubbish - they find out by applying strict rules of reasoning and often checking against other known (or "probably true") ideas, and winnow it down to the ideas that do in fact advance the boundaries of knowledge. Often times it's not even the individual that throws out bad ideas - many times it'll be colleagues poking holes in his argument, removing further unsuitable generated candidates from the pool of possible answers. If you think clever people just sit in a corner and come up with revolutionary ideas, I think you're probably wrong. Even the ancient philosophers used to hang out with some wine and hear out their peers and poked holes in their arguments. They called it a symposium.
    - doctoboggan 3y ago
      Sorry yes I should have been more clear and I think I am agreeing with you. I was saying that most people just come up with a thought and retroactively apply "logic" to it so they feel like they've reasoned themselves there. A select few people rigorously apply logic and then follow that to whatever conclusion it leads to. We call these people scientists but honestly in my experience even many scientists can fall into the first camp.
      - exe34 3y ago
        Aha my bad!
    - meroes 3y ago
      The thing is those were much larger ideas/arguments that could be picked apart by sturdy logical targeting. My experience is narrow scope prompts (that still require chain-of-thought) that are much less lofty defeating models. No symposium ever entertained these prompts, because we all know the pigeon hole principle for very basic setups, for example. Humans a lot of the time do just come up with the right answer. We just don’t ask those questions much because we answer them ourselves a lot of the time. Though I only see one small angle with my work.
stavros 3y ago
I think that chain-of-thought for LLMs is just helping them enhance their "memory", as it puts their reasoning into the context and helps them refer to it more readily. That's just a guess, though.
- snorkel 3y ago
  That’s pretty much correct. An LLM is often used rather like a forecast model that can forecast the next word in a sequence of words. When it’s generating output it’s just continuously forecasting (predicting) the next word of output. Your prompt is just providing the model with input data to start forecasting from. The prior output itself also becomes part of the context to forecast from. The output of “think about it step-by-step” becomes part of its own context to continue forecasting from, hence guides its output. I know that “forecasting” is technically not the right term, but I’ve found it helpful to understand what it is LLM‘s are actually doing when generating output.
throwaway35777 3y ago
> Humans can reason purely form the armchair, starting with very few concepts, and reach far far, ironclad conclusions. Models aren’t doing that. Sure, but the structure of human reasoning is almost identical to chains of thought. We have an auditory loop and, faced with a complex problem we repeat the mantra "now that I know XYZ, then what..." until the a good next step pops into our head and we add that to the context. The transition function just is (currently) much better in humans. Edit: people who disagree with this, why?
- andoando 3y ago
  Chain of thought in itself is pretty simple. We had logical provers in the 50s. The difficulty imo is how "thought" is modeled. Pure logic is too rigorous, and pure statistics is too inconsistent.
  - oldsecondhand 3y ago
    Maybe we should teach Prolog/CLP and PDDL to LLMs. Unfortunately the training set would be too small. It would be cool to have logic based modeling jobs, even if the goal is just to feed the LLMs.
    - andoando 3y ago
      Considering GPT can do programming and logic to some level, I assume it has has training of that sort? It can seem to do logic even on some completely made up abstract notions. For example "Consider a jumajambi has 2 jimimis. Each jimijimi is a jomololo or a joobajooba. How many possible variations of jumajambi are there if there are 4 jumajambi?". People keep calling it "next next token predictors", but clearly there is something more going on and I would love for someone to give a simple explanation.
      - famouswaffles 3y ago
        >People keep calling it "next next token predictors", but clearly there is something more going on and I would love for someone to give a simple explanation. Next token prediction is the objective function. The model is asked to predict the next word yes but it's also allowed to compute the answer and more importantly, the entire training process is supposed to be the model learning and figuring out what sort of computations aid the prediction of the corpus it's trained on. If your corpus is language A followed by the translation in Language B then there's little choice but for the model to learn computations that translate as loss goes down. Is your corpus is chess moves then again, it's going to have to learn how to compute chess games to reduce loss. You can see this with toy models trained on toy problems. Example - a tiny transformer trained on addition examples - x + y = z learning an algorithm for addition. https://cprimozic.net/blog/reverse-engineering-a-small-neural-network/ https://cprimozic.net/blog/reverse-engineering-a-small-neura... "Pick the right word" is not a trivial exercise for the vast majority of text data. And again because people often make this mistake but a LLMs ultimate objective is NOT to produce "text that looks right" but "text that is right". Of course "right" as determined by the training corpus but basically anytime it picks a wrong word is opportunity for the model to learn and learn it does.
PheonixPharts 3y ago
Given that LLMs are basically doing Sequential Monte-carlo sampling in latent space, the "thought" part of chain-of-thought certainly seems more akin to the necessary warm up period whenever you do any kind of SMC sampling. Anyone whose done serious Bayesian stats work knows that the sampler needs to warm up for a bit to get start efficiently sampling. I suspect something similar is happening with chain-of-thought: the model needs to wander around a bit before it gets into the correct neighborhood for sampling the answer.
- leereeves 3y ago
  That's quite an interesting comparison. I like the description of both as Sequential Monte-carlo sampling from a desired distribution. But I think there are two crucial differences. First, in Bayesian sampling, the initial value and first samples are not sampled from the desired distribution. In a well trained LLM, the prompt is given and the first response is sampled from the desired distribution (of text that is likely to follow the prompt). Second, in Bayesian sampling, the fact that the samples aren't independent is an unwelcome but unsolvable problem. We want independent samples but can't generate them, so we settle for conditionally independent samples. In an LLM, we want each sample to be dependent on the preceding text, in particular the prompt. In summary: Bayesian sampling - poorly chosen "prompt" (the initial sample), future samples would ideally be independent of the prompt and each other. LLM sampling - carefully chosen prompt, future samples are ideally dependent on the prompt and on each other. And in conclusion: The warm up period helps a Bayesian sampler find values that are less dependent on the initial "prompt", which we definitely don't want in an LLM.
- hyperliner 3y ago
  [dead]
bigcat12345678 3y ago
》Humans can reason purely form the armchair, starting with very few concepts, and reach far far, ironclad conclusions. I myself have no such ability. I cannot reason beyond roughly 10 lines of golang code. That's evident in my numerous hobby puzzle solving sessions.

by meroes 3y ago

exe34 3y ago

I think a lot of what humans think of as "1. 2. Therefore 3." kind of reasoning isn't different from what the llm is doing, and not in fact any more clever than that. Plenty of people believe plenty of questionable things that they assume they have thought through but really haven't. They used the context to guess the next idea/word, often reaching the conclusions they started out with. When you talk about ironclad conclusions, I think what happens is that we come up with those confabulations intuitively, but then we subject them to intense checking - have we defined everything clearly enough, is that leap in reasoning justified, etc. So what I'd really like to see is a way to teach llms to take a vague English sentence and transform it into a form that can be run through a more formal reasoning engine. Often instead of asking an llm to tell you something like how many football fields could you fit inside England, you are better off telling it to write python code to do this, assume get_size_football_field() in m^2 and get_size_England() in m^2 is available.

doctoboggan 3y ago
> I think a lot of what humans think of as "1. 2. Therefore 3." kind of reasoning isn't different from what the llm is doing, and not in fact any more clever than that. Plenty of people believe plenty of questionable things that they assume they have thought through but really haven't. They used the context to guess the next idea/word, often reaching the conclusions they started out with. Agreed that many/most humans behave this way, but some do not. And those who do not are the ones advancing the boundaries of knowledge and it would be very nice if we could get our LLMs to behave in the same way.
- exe34 3y ago
  That's what I mean though, people don't just come up with the right answer out of nowhere. They think through many possibilities generatively, based on "intuition", and most of what they come up with is rubbish - they find out by applying strict rules of reasoning and often checking against other known (or "probably true") ideas, and winnow it down to the ideas that do in fact advance the boundaries of knowledge. Often times it's not even the individual that throws out bad ideas - many times it'll be colleagues poking holes in his argument, removing further unsuitable generated candidates from the pool of possible answers. If you think clever people just sit in a corner and come up with revolutionary ideas, I think you're probably wrong. Even the ancient philosophers used to hang out with some wine and hear out their peers and poked holes in their arguments. They called it a symposium.
  - doctoboggan 3y ago
    Sorry yes I should have been more clear and I think I am agreeing with you. I was saying that most people just come up with a thought and retroactively apply "logic" to it so they feel like they've reasoned themselves there. A select few people rigorously apply logic and then follow that to whatever conclusion it leads to. We call these people scientists but honestly in my experience even many scientists can fall into the first camp.
    - exe34 3y ago
      Aha my bad!
  - meroes 3y ago
    The thing is those were much larger ideas/arguments that could be picked apart by sturdy logical targeting. My experience is narrow scope prompts (that still require chain-of-thought) that are much less lofty defeating models. No symposium ever entertained these prompts, because we all know the pigeon hole principle for very basic setups, for example. Humans a lot of the time do just come up with the right answer. We just don’t ask those questions much because we answer them ourselves a lot of the time. Though I only see one small angle with my work.

stavros 3y ago

I think that chain-of-thought for LLMs is just helping them enhance their "memory", as it puts their reasoning into the context and helps them refer to it more readily. That's just a guess, though.

snorkel 3y ago
That’s pretty much correct. An LLM is often used rather like a forecast model that can forecast the next word in a sequence of words. When it’s generating output it’s just continuously forecasting (predicting) the next word of output. Your prompt is just providing the model with input data to start forecasting from. The prior output itself also becomes part of the context to forecast from. The output of “think about it step-by-step” becomes part of its own context to continue forecasting from, hence guides its output. I know that “forecasting” is technically not the right term, but I’ve found it helpful to understand what it is LLM‘s are actually doing when generating output.

throwaway35777 3y ago

> Humans can reason purely form the armchair, starting with very few concepts, and reach far far, ironclad conclusions. Models aren’t doing that. Sure, but the structure of human reasoning is almost identical to chains of thought. We have an auditory loop and, faced with a complex problem we repeat the mantra "now that I know XYZ, then what..." until the a good next step pops into our head and we add that to the context. The transition function just is (currently) much better in humans. Edit: people who disagree with this, why?

andoando 3y ago
Chain of thought in itself is pretty simple. We had logical provers in the 50s. The difficulty imo is how "thought" is modeled. Pure logic is too rigorous, and pure statistics is too inconsistent.
- oldsecondhand 3y ago
  Maybe we should teach Prolog/CLP and PDDL to LLMs. Unfortunately the training set would be too small. It would be cool to have logic based modeling jobs, even if the goal is just to feed the LLMs.
  - andoando 3y ago
    Considering GPT can do programming and logic to some level, I assume it has has training of that sort? It can seem to do logic even on some completely made up abstract notions. For example "Consider a jumajambi has 2 jimimis. Each jimijimi is a jomololo or a joobajooba. How many possible variations of jumajambi are there if there are 4 jumajambi?". People keep calling it "next next token predictors", but clearly there is something more going on and I would love for someone to give a simple explanation.
    - famouswaffles 3y ago
      >People keep calling it "next next token predictors", but clearly there is something more going on and I would love for someone to give a simple explanation. Next token prediction is the objective function. The model is asked to predict the next word yes but it's also allowed to compute the answer and more importantly, the entire training process is supposed to be the model learning and figuring out what sort of computations aid the prediction of the corpus it's trained on. If your corpus is language A followed by the translation in Language B then there's little choice but for the model to learn computations that translate as loss goes down. Is your corpus is chess moves then again, it's going to have to learn how to compute chess games to reduce loss. You can see this with toy models trained on toy problems. Example - a tiny transformer trained on addition examples - x + y = z learning an algorithm for addition. https://cprimozic.net/blog/reverse-engineering-a-small-neural-network/ https://cprimozic.net/blog/reverse-engineering-a-small-neura... "Pick the right word" is not a trivial exercise for the vast majority of text data. And again because people often make this mistake but a LLMs ultimate objective is NOT to produce "text that looks right" but "text that is right". Of course "right" as determined by the training corpus but basically anytime it picks a wrong word is opportunity for the model to learn and learn it does.

PheonixPharts 3y ago

Given that LLMs are basically doing Sequential Monte-carlo sampling in latent space, the "thought" part of chain-of-thought certainly seems more akin to the necessary warm up period whenever you do any kind of SMC sampling. Anyone whose done serious Bayesian stats work knows that the sampler needs to warm up for a bit to get start efficiently sampling. I suspect something similar is happening with chain-of-thought: the model needs to wander around a bit before it gets into the correct neighborhood for sampling the answer.

leereeves 3y ago
That's quite an interesting comparison. I like the description of both as Sequential Monte-carlo sampling from a desired distribution. But I think there are two crucial differences. First, in Bayesian sampling, the initial value and first samples are not sampled from the desired distribution. In a well trained LLM, the prompt is given and the first response is sampled from the desired distribution (of text that is likely to follow the prompt). Second, in Bayesian sampling, the fact that the samples aren't independent is an unwelcome but unsolvable problem. We want independent samples but can't generate them, so we settle for conditionally independent samples. In an LLM, we want each sample to be dependent on the preceding text, in particular the prompt. In summary: Bayesian sampling - poorly chosen "prompt" (the initial sample), future samples would ideally be independent of the prompt and each other. LLM sampling - carefully chosen prompt, future samples are ideally dependent on the prompt and on each other. And in conclusion: The warm up period helps a Bayesian sampler find values that are less dependent on the initial "prompt", which we definitely don't want in an LLM.
hyperliner 3y ago
[dead]

bigcat12345678 3y ago

》Humans can reason purely form the armchair, starting with very few concepts, and reach far far, ironclad conclusions. I myself have no such ability. I cannot reason beyond roughly 10 lines of golang code. That's evident in my numerous hobby puzzle solving sessions.