5 ms·

Which is part of the problem. LLMs don't like doing things step by step. You can force an LLM to go step by step with an external tool that continually re-prom

by phire 3y ago

Which is part of the problem. LLMs don't like doing things step by step. You can force an LLM to go step by step with an external tool that continually re-prompts with each step, but the challenge requires a single prompt. When you try to do prompt engineering to make it go step by step before telling you the answer, things go off the rails after a handful of steps.

vidarh 3y ago
Humans don't like doing things step by step without incentives and prodding either. To me, this is an odd thing to try to get LLMs to do because we know it's immensely hard to ensure reliable repetitive rule application to the point we spend many years conditioning humans on tasks where we care about the ability to stick with it, and then still built entire social structures on encouraging ongoing compliance, and still regularly fail to achieve it. Why would we expect LLMs that are still fairly primitive to do this well without mechanisms regulating it?
- HarHarVeryFunny 3y ago
  1) We'd generally expect a computer to be better at mindless repetitive rule application than us. This is what they excel at. 2) While these are just LLMs (specifically, pre-trained transformers), they do have a place on the spectrum of intelligence, and some insiders are claiming "scale is all you need" to reach human-level AGI. It's useful to step back, look at simple examples like this, and realize how far they are from being generally intelligent, as well as where our intuition for what they might be good at (repetitive tasks) is wrong due to their specific architecture.
  - vidarh 3y ago
    The problem is that 1) is a meaningless expectation when you're explicitly running a program that is giving probabilistic outputs. To 2), my point is exactly that if anything talking about the struggle to get LLMs to do mindless repetitive rule application is not even close to being an argument against LLMs ability to scale to AGI. At most they're an argument for why LLMs will need at least some scaffolding around them. (That does not mean they are certain to be otherwise architecturally enough) It is useful to step back and look at examples like this, but concluding from it that they say anything about how far they are from being generally intelligent is not. If it does get more people to drop the bizarrely unreasonable intuition that you should expect them to be good at repetitive tasks, that'd be helpful though.
    - HarHarVeryFunny 3y ago
      Right - it is a meaningless expectation (I used the word intuition), although I'm not sure that probabilistic outputs is the issue here. It's really the fact that without short-term memory it's difficult to keep track of what you are doing (flipping back and forth between rules and problem being worked on, figure out the rule pair prefix in the string, etc). A combination of step-by-step output (pseudo memory) and hoping the model will attend to that is best you can do, but it's not enough in many cases. Best measure of how far these models are from AGI, is what is architecturally missing (I'd say more than just scaffolding), although that also depends on how you define both AGI and intelligence itself. Still, I think examples like this are indicative of where we are right now - the model can neither do, nor be taught to do, the task, nor realize that it has failed (and then learn from the mistake). There is a lot missing.
      - vidarh 3y ago
        I say scaffolding because for any task an LLM can either do reliably enough or correct reliably enough, or where correctness is easily verified, all you need is a loop of multiple runs, test and feedback. So for a whole lot of tasks it is just down to scaffolding. For AGI it's not. But for a task like running a Turing machine it is and so it downplays their abilities to measure them by what they can be "convinced" to do in a single prompt.
        HarHarVeryFunny 3y ago
        An external loop really isn't any different to "think step by step" - you're giving the model a chance to see it's own prior output and carry on from that. Also, even without "think step by step", or an external loop, the generation process is already providing looping allowing the model to see it's entire output-so-far as input to generate each next word! The model itself just takes an input sequence and predicts one next word each time it is run; it's the generative process that then takes each generated word, appends it to the input and runs the model again to generate the next-next word, etc, only stopping when the model predicts the end-of-sequence token. So, these models already have a chance to catch their own errors, if they are capable, without any additional looping beyond what is already occurring. However, I've never seen an LLM do this ("watermelons are blue, err, I mean green") since whatever caused it to predict "blue" the first time it saw "watermelons are" will likely cause it to predict "blue" on subsequent times (when it's seeing it's own output) too, so it won't realize there's been an error and predict a correction. Even if a person points out a mistake, these models only have limited in-context learning to be able to use that to correct their built-in (pre-trained weights) behavior. Sometimes it works, sometimes it doesn't. I tried multiple times providing feedback and suggestions to Claude on how to fix it's mistakes on this string replacement task, and was never successful. Some types of external scaffolding (i.e. modified generative process) might be helpful, such as "tree of thoughts" which would let the scaffolding pick the best output the model is capable of, but of course sometimes your best is still not good enough!

by phire 3y ago

vidarh 3y ago

Humans don't like doing things step by step without incentives and prodding either. To me, this is an odd thing to try to get LLMs to do because we know it's immensely hard to ensure reliable repetitive rule application to the point we spend many years conditioning humans on tasks where we care about the ability to stick with it, and then still built entire social structures on encouraging ongoing compliance, and still regularly fail to achieve it. Why would we expect LLMs that are still fairly primitive to do this well without mechanisms regulating it?

HarHarVeryFunny 3y ago
1) We'd generally expect a computer to be better at mindless repetitive rule application than us. This is what they excel at. 2) While these are just LLMs (specifically, pre-trained transformers), they do have a place on the spectrum of intelligence, and some insiders are claiming "scale is all you need" to reach human-level AGI. It's useful to step back, look at simple examples like this, and realize how far they are from being generally intelligent, as well as where our intuition for what they might be good at (repetitive tasks) is wrong due to their specific architecture.
- vidarh 3y ago
  The problem is that 1) is a meaningless expectation when you're explicitly running a program that is giving probabilistic outputs. To 2), my point is exactly that if anything talking about the struggle to get LLMs to do mindless repetitive rule application is not even close to being an argument against LLMs ability to scale to AGI. At most they're an argument for why LLMs will need at least some scaffolding around them. (That does not mean they are certain to be otherwise architecturally enough) It is useful to step back and look at examples like this, but concluding from it that they say anything about how far they are from being generally intelligent is not. If it does get more people to drop the bizarrely unreasonable intuition that you should expect them to be good at repetitive tasks, that'd be helpful though.
  - HarHarVeryFunny 3y ago
    Right - it is a meaningless expectation (I used the word intuition), although I'm not sure that probabilistic outputs is the issue here. It's really the fact that without short-term memory it's difficult to keep track of what you are doing (flipping back and forth between rules and problem being worked on, figure out the rule pair prefix in the string, etc). A combination of step-by-step output (pseudo memory) and hoping the model will attend to that is best you can do, but it's not enough in many cases. Best measure of how far these models are from AGI, is what is architecturally missing (I'd say more than just scaffolding), although that also depends on how you define both AGI and intelligence itself. Still, I think examples like this are indicative of where we are right now - the model can neither do, nor be taught to do, the task, nor realize that it has failed (and then learn from the mistake). There is a lot missing.
    - vidarh 3y ago
      I say scaffolding because for any task an LLM can either do reliably enough or correct reliably enough, or where correctness is easily verified, all you need is a loop of multiple runs, test and feedback. So for a whole lot of tasks it is just down to scaffolding. For AGI it's not. But for a task like running a Turing machine it is and so it downplays their abilities to measure them by what they can be "convinced" to do in a single prompt.
      - HarHarVeryFunny 3y ago
        An external loop really isn't any different to "think step by step" - you're giving the model a chance to see it's own prior output and carry on from that. Also, even without "think step by step", or an external loop, the generation process is already providing looping allowing the model to see it's entire output-so-far as input to generate each next word! The model itself just takes an input sequence and predicts one next word each time it is run; it's the generative process that then takes each generated word, appends it to the input and runs the model again to generate the next-next word, etc, only stopping when the model predicts the end-of-sequence token. So, these models already have a chance to catch their own errors, if they are capable, without any additional looping beyond what is already occurring. However, I've never seen an LLM do this ("watermelons are blue, err, I mean green") since whatever caused it to predict "blue" the first time it saw "watermelons are" will likely cause it to predict "blue" on subsequent times (when it's seeing it's own output) too, so it won't realize there's been an error and predict a correction. Even if a person points out a mistake, these models only have limited in-context learning to be able to use that to correct their built-in (pre-trained weights) behavior. Sometimes it works, sometimes it doesn't. I tried multiple times providing feedback and suggestions to Claude on how to fix it's mistakes on this string replacement task, and was never successful. Some types of external scaffolding (i.e. modified generative process) might be helpful, such as "tree of thoughts" which would let the scaffolding pick the best output the model is capable of, but of course sometimes your best is still not good enough!