4 ms·

How about this as a nice reductive example of the things that current LLMs struggle/fail at: https://x.com/VictorTaelin/status/1776677635491344744 https://x.co

How about this as a nice reductive example of the things that current LLMs struggle/fail at: https://x.com/VictorTaelin/status/1776677635491344744 https://x.com/VictorTaelin/status/1776677635491344744 This guy is offering a $10K prize if anyone can prompt an LLM to reliably (by itself - no tool use) repeatedly apply some simple letter-pair substitution rules to strings of up to 12 characters. The rules to be applied (to strings composed of the letters A, B, X & Y) are: 1) AX -> delete 2) AY -> YA 3) BX -> BY 4) BY -> delete In his Tweet he uses A# #A B# #B rather than A X B Y, but has clarified it makes no difference. Try it - it's remarkably hard. You should be able to get the LLM to do a few short ones, but try getting it to be reliable or do longer ones. So, Turing? Not going to happen!

viraptor 3y ago
As long as you can convince LLM to do it step by step, reproducing the whole thing again, it gets much simpler. It changes the idea of an unlimited number of steps into one replacement at a time. Then make sure to split the list so that the letters don't get merged into larger tokens. (Same issue as numbers and math)
- sfn42 3y ago
  Then go win the prize?
- phire 3y ago
  Which is part of the problem. LLMs don't like doing things step by step. You can force an LLM to go step by step with an external tool that continually re-prompts with each step, but the challenge requires a single prompt. When you try to do prompt engineering to make it go step by step before telling you the answer, things go off the rails after a handful of steps.
  - vidarh 3y ago
    Humans don't like doing things step by step without incentives and prodding either. To me, this is an odd thing to try to get LLMs to do because we know it's immensely hard to ensure reliable repetitive rule application to the point we spend many years conditioning humans on tasks where we care about the ability to stick with it, and then still built entire social structures on encouraging ongoing compliance, and still regularly fail to achieve it. Why would we expect LLMs that are still fairly primitive to do this well without mechanisms regulating it?
    - HarHarVeryFunny 3y ago
      1) We'd generally expect a computer to be better at mindless repetitive rule application than us. This is what they excel at. 2) While these are just LLMs (specifically, pre-trained transformers), they do have a place on the spectrum of intelligence, and some insiders are claiming "scale is all you need" to reach human-level AGI. It's useful to step back, look at simple examples like this, and realize how far they are from being generally intelligent, as well as where our intuition for what they might be good at (repetitive tasks) is wrong due to their specific architecture.
      - vidarh 3y ago
        The problem is that 1) is a meaningless expectation when you're explicitly running a program that is giving probabilistic outputs. To 2), my point is exactly that if anything talking about the struggle to get LLMs to do mindless repetitive rule application is not even close to being an argument against LLMs ability to scale to AGI. At most they're an argument for why LLMs will need at least some scaffolding around them. (That does not mean they are certain to be otherwise architecturally enough) It is useful to step back and look at examples like this, but concluding from it that they say anything about how far they are from being generally intelligent is not. If it does get more people to drop the bizarrely unreasonable intuition that you should expect them to be good at repetitive tasks, that'd be helpful though.
        HarHarVeryFunny 3y ago
        Right - it is a meaningless expectation (I used the word intuition), although I'm not sure that probabilistic outputs is the issue here. It's really the fact that without short-term memory it's difficult to keep track of what you are doing (flipping back and forth between rules and problem being worked on, figure out the rule pair prefix in the string, etc). A combination of step-by-step output (pseudo memory) and hoping the model will attend to that is best you can do, but it's not enough in many cases. Best measure of how far these models are from AGI, is what is architecturally missing (I'd say more than just scaffolding), although that also depends on how you define both AGI and intelligence itself. Still, I think examples like this are indicative of where we are right now - the model can neither do, nor be taught to do, the task, nor realize that it has failed (and then learn from the mistake). There is a lot missing.
- HarHarVeryFunny 3y ago
  Try it! It's simple enough to instruct it to work step by step, showing it's work (as a type of memory - same way we'd use pencil and paper), so that each string transformation breaks down into a sequence of single replacement steps that it shows (input, rule applied, output). Tokenization is not the issue. Just give it the string as a space separated list of letters such as "A B B A X Y". The problem is getting it to work reliably, and for longer strings. The best I could do with Claude Haiku was to get it to apply three replacement rules successfully, then it just made a stupid mistake. LLMs are just not built for tasks like this. A lack of short-term memory is one obvious issue, but seems almost impossible to mitigate with step-by-step instruction.
- phkahler 3y ago
  So go claim the $10K prize.
  - viraptor 3y ago
    This has been effectively solved by the time of my comment. The verification was done now and the prize is claimed.
    - HarHarVeryFunny 3y ago
      Yes, although there is now a secondary competition, for 25% of the prize money (donated by the winner), to see who can reduce the prompt size/cost the most. The winning prompt will be published on wednesday. Interestingly the top solutions all used Claude-3 Opus (via the API, with temperature set to zero). The winning prompt apparently cost $1+ to run, which given the $15/M input token cost of Claude-3 Opus means the prompt was around 100K tokens in size(!!), which in turn implies heavy use of examples/in-context learning.
brookst 3y ago
Isn’t this just exploiting the loss of letter-level resolution from tokenization?
- HarHarVeryFunny 3y ago
  No - makes no difference if you give the input string, and replacement rules, as sequences of space separated letters. FWIW, despite tokenized input, these models all seem capable of giving you the sequence of letters in a word if you ask.
- geor9e 3y ago
  He addresses this in the tweet comments (it's the most common comment) and says it holds true even with byte (letter) level tokenization.

viraptor 3y ago

As long as you can convince LLM to do it step by step, reproducing the whole thing again, it gets much simpler. It changes the idea of an unlimited number of steps into one replacement at a time. Then make sure to split the list so that the letters don't get merged into larger tokens. (Same issue as numbers and math)

sfn42 3y ago
Then go win the prize?
phire 3y ago
Which is part of the problem. LLMs don't like doing things step by step. You can force an LLM to go step by step with an external tool that continually re-prompts with each step, but the challenge requires a single prompt. When you try to do prompt engineering to make it go step by step before telling you the answer, things go off the rails after a handful of steps.
- vidarh 3y ago
  Humans don't like doing things step by step without incentives and prodding either. To me, this is an odd thing to try to get LLMs to do because we know it's immensely hard to ensure reliable repetitive rule application to the point we spend many years conditioning humans on tasks where we care about the ability to stick with it, and then still built entire social structures on encouraging ongoing compliance, and still regularly fail to achieve it. Why would we expect LLMs that are still fairly primitive to do this well without mechanisms regulating it?
  - HarHarVeryFunny 3y ago
    1) We'd generally expect a computer to be better at mindless repetitive rule application than us. This is what they excel at. 2) While these are just LLMs (specifically, pre-trained transformers), they do have a place on the spectrum of intelligence, and some insiders are claiming "scale is all you need" to reach human-level AGI. It's useful to step back, look at simple examples like this, and realize how far they are from being generally intelligent, as well as where our intuition for what they might be good at (repetitive tasks) is wrong due to their specific architecture.
    - vidarh 3y ago
      The problem is that 1) is a meaningless expectation when you're explicitly running a program that is giving probabilistic outputs. To 2), my point is exactly that if anything talking about the struggle to get LLMs to do mindless repetitive rule application is not even close to being an argument against LLMs ability to scale to AGI. At most they're an argument for why LLMs will need at least some scaffolding around them. (That does not mean they are certain to be otherwise architecturally enough) It is useful to step back and look at examples like this, but concluding from it that they say anything about how far they are from being generally intelligent is not. If it does get more people to drop the bizarrely unreasonable intuition that you should expect them to be good at repetitive tasks, that'd be helpful though.
      - HarHarVeryFunny 3y ago
        Right - it is a meaningless expectation (I used the word intuition), although I'm not sure that probabilistic outputs is the issue here. It's really the fact that without short-term memory it's difficult to keep track of what you are doing (flipping back and forth between rules and problem being worked on, figure out the rule pair prefix in the string, etc). A combination of step-by-step output (pseudo memory) and hoping the model will attend to that is best you can do, but it's not enough in many cases. Best measure of how far these models are from AGI, is what is architecturally missing (I'd say more than just scaffolding), although that also depends on how you define both AGI and intelligence itself. Still, I think examples like this are indicative of where we are right now - the model can neither do, nor be taught to do, the task, nor realize that it has failed (and then learn from the mistake). There is a lot missing.
HarHarVeryFunny 3y ago
Try it! It's simple enough to instruct it to work step by step, showing it's work (as a type of memory - same way we'd use pencil and paper), so that each string transformation breaks down into a sequence of single replacement steps that it shows (input, rule applied, output). Tokenization is not the issue. Just give it the string as a space separated list of letters such as "A B B A X Y". The problem is getting it to work reliably, and for longer strings. The best I could do with Claude Haiku was to get it to apply three replacement rules successfully, then it just made a stupid mistake. LLMs are just not built for tasks like this. A lack of short-term memory is one obvious issue, but seems almost impossible to mitigate with step-by-step instruction.
phkahler 3y ago
So go claim the $10K prize.
- viraptor 3y ago
  This has been effectively solved by the time of my comment. The verification was done now and the prize is claimed.
  - HarHarVeryFunny 3y ago
    Yes, although there is now a secondary competition, for 25% of the prize money (donated by the winner), to see who can reduce the prompt size/cost the most. The winning prompt will be published on wednesday. Interestingly the top solutions all used Claude-3 Opus (via the API, with temperature set to zero). The winning prompt apparently cost $1+ to run, which given the $15/M input token cost of Claude-3 Opus means the prompt was around 100K tokens in size(!!), which in turn implies heavy use of examples/in-context learning.

brookst 3y ago

Isn’t this just exploiting the loss of letter-level resolution from tokenization?

HarHarVeryFunny 3y ago
No - makes no difference if you give the input string, and replacement rules, as sequences of space separated letters. FWIW, despite tokenized input, these models all seem capable of giving you the sequence of letters in a word if you ask.
geor9e 3y ago
He addresses this in the tweet comments (it's the most common comment) and says it holds true even with byte (letter) level tokenization.