3 ms·

Right - it is a meaningless expectation (I used the word intuition), although I'm not sure that probabilistic outputs is the issue here. It's really the fact th

Right - it is a meaningless expectation (I used the word intuition), although I'm not sure that probabilistic outputs is the issue here. It's really the fact that without short-term memory it's difficult to keep track of what you are doing (flipping back and forth between rules and problem being worked on, figure out the rule pair prefix in the string, etc). A combination of step-by-step output (pseudo memory) and hoping the model will attend to that is best you can do, but it's not enough in many cases. Best measure of how far these models are from AGI, is what is architecturally missing (I'd say more than just scaffolding), although that also depends on how you define both AGI and intelligence itself. Still, I think examples like this are indicative of where we are right now - the model can neither do, nor be taught to do, the task, nor realize that it has failed (and then learn from the mistake). There is a lot missing.

vidarh 3y ago
I say scaffolding because for any task an LLM can either do reliably enough or correct reliably enough, or where correctness is easily verified, all you need is a loop of multiple runs, test and feedback. So for a whole lot of tasks it is just down to scaffolding. For AGI it's not. But for a task like running a Turing machine it is and so it downplays their abilities to measure them by what they can be "convinced" to do in a single prompt.
- HarHarVeryFunny 3y ago
  An external loop really isn't any different to "think step by step" - you're giving the model a chance to see it's own prior output and carry on from that. Also, even without "think step by step", or an external loop, the generation process is already providing looping allowing the model to see it's entire output-so-far as input to generate each next word! The model itself just takes an input sequence and predicts one next word each time it is run; it's the generative process that then takes each generated word, appends it to the input and runs the model again to generate the next-next word, etc, only stopping when the model predicts the end-of-sequence token. So, these models already have a chance to catch their own errors, if they are capable, without any additional looping beyond what is already occurring. However, I've never seen an LLM do this ("watermelons are blue, err, I mean green") since whatever caused it to predict "blue" the first time it saw "watermelons are" will likely cause it to predict "blue" on subsequent times (when it's seeing it's own output) too, so it won't realize there's been an error and predict a correction. Even if a person points out a mistake, these models only have limited in-context learning to be able to use that to correct their built-in (pre-trained weights) behavior. Sometimes it works, sometimes it doesn't. I tried multiple times providing feedback and suggestions to Claude on how to fix it's mistakes on this string replacement task, and was never successful. Some types of external scaffolding (i.e. modified generative process) might be helpful, such as "tree of thoughts" which would let the scaffolding pick the best output the model is capable of, but of course sometimes your best is still not good enough!
  - vidarh 3y ago
    > An external loop really isn't any different to "think step by step" - you're giving the model a chance to see it's own prior output and carry on from that. It is very different, because you control the context fed in at the start of each cycle. Put another way, it's fairly easy to get most LLMs to follow instructions one step. With a loop, that makes it fairly easy to get them to follow however many steps you like. But getting them to follow multiple steps by asking them to "think step by step" tends to fall apart after quite few steps because suddenly your instructions are a small, distant part of the context. Test it. The difference is dramatic.
    - HarHarVeryFunny 3y ago
      Interesting - it also suggests a "think step by step" approach where you tell the model to itself repeat (echo) the instructions before each step.
      - vidarh 3y ago
        I think that'd be a worthwhile experiment, but I suspect it might not always be enough (but absolutely worth testing). You can "coerce" the model more strongly when you explicitly control everything, though. Adding a loop also allows you to put it in a "straitjacket" in many ways - e.g. if a local model you can select output tokens to force the response to match a given grammar to give it less room to go off the rails (see llama.cpp/llamafile that supports this), you can repeat and adjust temperature until responses match requirements or manipulating the answer so far and have it complete it. Or combine with an evaluation step to force a "does this answer comply with restrictions X, Y, Z: ..." stage, and so on. Of course these are not suitable to get you AGI, but when you have tasks where testing compliance or ranking quality of output are "cheap"/"easy" but coming up with answers is hard, they provide ways to get LLMs to work even in cases where the simple approach of individual, complex prompts are too unreliable. (the tradeoff of course being that if you have to spend too much time tuning these kinds of tricks to a specific problem you need to weigh up if they're worth it vs. solving the problem without an LLM) Of course they are workarounds, and I'm sure there will be architectural advances that will hopefully make these relatively crude methods less important, but in the meantime, they do help.

vidarh 3y ago

I say scaffolding because for any task an LLM can either do reliably enough or correct reliably enough, or where correctness is easily verified, all you need is a loop of multiple runs, test and feedback. So for a whole lot of tasks it is just down to scaffolding. For AGI it's not. But for a task like running a Turing machine it is and so it downplays their abilities to measure them by what they can be "convinced" to do in a single prompt.

HarHarVeryFunny 3y ago
An external loop really isn't any different to "think step by step" - you're giving the model a chance to see it's own prior output and carry on from that. Also, even without "think step by step", or an external loop, the generation process is already providing looping allowing the model to see it's entire output-so-far as input to generate each next word! The model itself just takes an input sequence and predicts one next word each time it is run; it's the generative process that then takes each generated word, appends it to the input and runs the model again to generate the next-next word, etc, only stopping when the model predicts the end-of-sequence token. So, these models already have a chance to catch their own errors, if they are capable, without any additional looping beyond what is already occurring. However, I've never seen an LLM do this ("watermelons are blue, err, I mean green") since whatever caused it to predict "blue" the first time it saw "watermelons are" will likely cause it to predict "blue" on subsequent times (when it's seeing it's own output) too, so it won't realize there's been an error and predict a correction. Even if a person points out a mistake, these models only have limited in-context learning to be able to use that to correct their built-in (pre-trained weights) behavior. Sometimes it works, sometimes it doesn't. I tried multiple times providing feedback and suggestions to Claude on how to fix it's mistakes on this string replacement task, and was never successful. Some types of external scaffolding (i.e. modified generative process) might be helpful, such as "tree of thoughts" which would let the scaffolding pick the best output the model is capable of, but of course sometimes your best is still not good enough!
- vidarh 3y ago
  > An external loop really isn't any different to "think step by step" - you're giving the model a chance to see it's own prior output and carry on from that. It is very different, because you control the context fed in at the start of each cycle. Put another way, it's fairly easy to get most LLMs to follow instructions one step. With a loop, that makes it fairly easy to get them to follow however many steps you like. But getting them to follow multiple steps by asking them to "think step by step" tends to fall apart after quite few steps because suddenly your instructions are a small, distant part of the context. Test it. The difference is dramatic.
  - HarHarVeryFunny 3y ago
    Interesting - it also suggests a "think step by step" approach where you tell the model to itself repeat (echo) the instructions before each step.
    - vidarh 3y ago
      I think that'd be a worthwhile experiment, but I suspect it might not always be enough (but absolutely worth testing). You can "coerce" the model more strongly when you explicitly control everything, though. Adding a loop also allows you to put it in a "straitjacket" in many ways - e.g. if a local model you can select output tokens to force the response to match a given grammar to give it less room to go off the rails (see llama.cpp/llamafile that supports this), you can repeat and adjust temperature until responses match requirements or manipulating the answer so far and have it complete it. Or combine with an evaluation step to force a "does this answer comply with restrictions X, Y, Z: ..." stage, and so on. Of course these are not suitable to get you AGI, but when you have tasks where testing compliance or ranking quality of output are "cheap"/"easy" but coming up with answers is hard, they provide ways to get LLMs to work even in cases where the simple approach of individual, complex prompts are too unreliable. (the tradeoff of course being that if you have to spend too much time tuning these kinds of tricks to a specific problem you need to weigh up if they're worth it vs. solving the problem without an LLM) Of course they are workarounds, and I'm sure there will be architectural advances that will hopefully make these relatively crude methods less important, but in the meantime, they do help.