6 ms·

I posted the same 'Zero-Shot Chain-of-Thought and Image' to ChatGPT-4o and it made the same error. I then followed up with 'Your math is good but you derived i

by jcims 2y ago

I posted the same 'Zero-Shot Chain-of-Thought and Image' to ChatGPT-4o and it made the same error. I then followed up with 'Your math is good but you derived incorrect data from the image. Can you take another look and see if you can tell where the error is?'. It figured it out and corrected it: Let's re-examine the image and the data provided: * The inner radius r1 is given as 5cm * The outer radius r2 is given as 10cm * However, the dimensions labeled "5 cm" and "10 cm" are actually the diameters of the inner and outer circles, respectively, not the radii. Then recomputed and got the right answer. I asked it if it could surmise why it got the wrong answer and it said, among a number of things, that math problems commonly operate in radii instead of diameter. I restarted with a slightly modified prompt: There is a roll of tape with dimensions specified in the picture. The tape is 100 meters long when unrolled. How thick is the tape? Examine the image carefully and ensure that you fully understand how it is labeled. Make no assumptions. Then when calculating, take a deep breath and work on this problem step-by-step. It got it the first try, and I'm not interested enough to try it a bunch of times to see if that's statistically significant :)

sabrina_ramonov 2y ago
confirmed worked for me first try EDIT: out of 3 times, got it correct 2/3
CooCooCaCha 2y ago
This speaks to a deeper issue that LLMs don’t just have statistically-based knowledge, they also have statistically-based reasoning. This means their reasoning process isn’t necessarily based on logic, but what is statistically most probable. As you’ve experienced, their reasoning breaks down in less-common scenarios even if it should be easy to use logic to get the answer.
- 12907835202 2y ago
  Does anyone know how far off we are having logical AI? Math seems like low hanging fruit in that regard. But logic as it's used in philosophy feels like it might be a whole different and more difficult beast to tackle. I wonder if LLM's will just get better to the point of being indistinguishable from logic rather than actually achieving logical reasoning. Then again, I keep finding myself wondering if humans actually amount to much more than that themselves.
  - ryanianian 2y ago
    (Not an AI researcher, just someone who likes complexity analysis.) Discrete reasoning is NP-Complete. You can get very close with the stats-based approaches of LLMs and whatnot, but your minima/maxima may always turn out to be local rather than global.
    - slushy-chivalry 2y ago
      maybe theorem proving could help? ask gpt4o to produce a proof in coq and see if it checks out...or split it into multiple agents -- one produces the proof of the closed formula for the tape roll thickness, and another one verifies it
      - ryanianian 2y ago
        Sure, but those are heuristics and feedback loops. They are not guaranteed to give you a solution. An LLM can never be a SAT solver unless it's an LLM with a SAT solver bolted on.
        slushy-chivalry 2y ago
        I don't disagree -- there is a place for specialized tool, and LLM wouldn't be my first pick if somebody asked me to add two large numbers. There is nothing wrong with LLM + SAT solver -- especially if for an end-user it feels like they have 1 tool that solves their problem (even if under the hood it's 500 specialized tools governed by LLM). My point about producing a proof was more about exploratory analysis -- sometimes reading (even incorrect) proofs can give you an idea for an interesting solution. Moreover, LLM can (potentially) spit out a bunch of possibly solutions and have another tool prune and verify and rank the most promising ones. Also, the problem described in the blog is not a decision problem, so I'm not sure if it should be viewed through the lenses of computational complexity.
      - jamilton 2y ago
        I had the thought recently that theorem provers could be a neat source of synthetic data. Make an LLM generate a proof, run it to evaluate it and label it as valid/invalid, fine-tune the LLM on the results. In theory it should then more consistently create valid proofs.
- rthnbgrredf 2y ago
  Statistically-based reasoning also applies to humans. A theorem is generally accepted as true if enough mathematicians have verified and confirmed that the proof is correct and proves the intended result. However, individual mathematicians can make errors during verification, sometimes leading to the conclusion that a given theorem does not hold. Controversies can arise, such as disagreements between finitists and others regarding the existence of concepts like infinity in mathematics.
- ericmcer 2y ago
  That plays out for all the examples, except for the one where its answer was way off it and it corrected itself and attempted again. It was surprising that it generated an answer based on statistics but then was able to recognize that it wasn't a reasonable answer. I wonder how they are achieving that.
aidenn0 2y ago
I mean I could see my kid making this exact mistake on a word problem, so I suppose we've achieved "human like" reasoning at the expense of actually getting the answer we want?
- nebster 2y ago
  I tried to work out the problem myself first (using only the text) and accidentally used the diameter as the radius just like ChatGPT! Granted I haven't really tackled any maths problems for many years though.
hatenberg 2y ago
Chain of thought is nothing more than limiting the probability space enough that the model can provide the most likely answer. It's too much damn work to be useful.
yatz 2y ago
Once you correct the LLM, it will continue to provide the corrected answer until some time later, when it will again make the same mistake. At least, this has been my experience. If you are using LLM to pull answers programmatically and rely on their accuracy, here is what worked for the structured or numeric answers, such as numbers, JSON, etc. 1) Send the same prompt twice, including "Can you double check?" in the second prompt to force GPT to verify the answer. 2) If both answers are the same, you got the correct answer. 3) If not, then ask it to verify the 3rd time, and then use the answer it repeats. Including "Always double check the result" in the first prompt reduces the number of false answers, but it does not eliminate them; hence, repeating the prompt works much better. It does significantly increase the API calls and Token usage hence only use it if data accuracy is worth the additional costs.
- groby_b 2y ago
  > Once you correct the LLM, it will continue to provide the corrected answer until some time later, That is only true if you stay within the same chat. It is not true across chats. Context caching is something that a lot of folks would really really like to see. And jumping to a new chat is one of the core points of the OP: "I restarted with a slightly modified prompt:" The iterations before where mostly to figure out why the initial prompt went wrong. And AFAICT there's a good insight in the modified prompt - "Make no assumptions". Probably also "ensure you fully understand how it's labelled". And no, asking repeatedly doesn't necessarily give different answers, not even with "can you double check". There are quite a few examples where LLMs are consistently and proudly wrong. Don't use LLMs if 100% accuracy matters.
  - sharemywin 2y ago
    so what would you use instead?
    - groby_b 2y ago
      Depends - what's your allowable error rate? What are you solving for?
  - 2y ago
Auracle 2y ago
That’s funny. I practically got into a shouting match for the first time ever with ChatGPT earlier today because I was asking it to create a function to make a filled circle of pixels of a certain size using diameter and absolutely not radius (with some other constraints). This mattered because I wanted clear steps between 3,4,5,6 etc pixels wide, so the diameter was an int. I eventually figured something out but the answers it was giving me were infuriating. At some point instead of a radius it put “int halfSize = diameter / 2”.
scottmcdot 2y ago
In this context, what does the author mean by 'Zero-Shot'? From what I read on Wikipedia [1], it's about the model performing a task without specific training data for that task. But all of the experiments in the post seem to involve the model working without additional training data. [1] https://en.wikipedia.org/wiki/Zero-shot_learning https://en.wikipedia.org/wiki/Zero-shot_learning

by jcims 2y ago

sabrina_ramonov 2y ago

confirmed worked for me first try EDIT: out of 3 times, got it correct 2/3

CooCooCaCha 2y ago

This speaks to a deeper issue that LLMs don’t just have statistically-based knowledge, they also have statistically-based reasoning. This means their reasoning process isn’t necessarily based on logic, but what is statistically most probable. As you’ve experienced, their reasoning breaks down in less-common scenarios even if it should be easy to use logic to get the answer.

12907835202 2y ago
Does anyone know how far off we are having logical AI? Math seems like low hanging fruit in that regard. But logic as it's used in philosophy feels like it might be a whole different and more difficult beast to tackle. I wonder if LLM's will just get better to the point of being indistinguishable from logic rather than actually achieving logical reasoning. Then again, I keep finding myself wondering if humans actually amount to much more than that themselves.
- ryanianian 2y ago
  (Not an AI researcher, just someone who likes complexity analysis.) Discrete reasoning is NP-Complete. You can get very close with the stats-based approaches of LLMs and whatnot, but your minima/maxima may always turn out to be local rather than global.
  - slushy-chivalry 2y ago
    maybe theorem proving could help? ask gpt4o to produce a proof in coq and see if it checks out...or split it into multiple agents -- one produces the proof of the closed formula for the tape roll thickness, and another one verifies it
    - ryanianian 2y ago
      Sure, but those are heuristics and feedback loops. They are not guaranteed to give you a solution. An LLM can never be a SAT solver unless it's an LLM with a SAT solver bolted on.
      - slushy-chivalry 2y ago
        I don't disagree -- there is a place for specialized tool, and LLM wouldn't be my first pick if somebody asked me to add two large numbers. There is nothing wrong with LLM + SAT solver -- especially if for an end-user it feels like they have 1 tool that solves their problem (even if under the hood it's 500 specialized tools governed by LLM). My point about producing a proof was more about exploratory analysis -- sometimes reading (even incorrect) proofs can give you an idea for an interesting solution. Moreover, LLM can (potentially) spit out a bunch of possibly solutions and have another tool prune and verify and rank the most promising ones. Also, the problem described in the blog is not a decision problem, so I'm not sure if it should be viewed through the lenses of computational complexity.
    - jamilton 2y ago
      I had the thought recently that theorem provers could be a neat source of synthetic data. Make an LLM generate a proof, run it to evaluate it and label it as valid/invalid, fine-tune the LLM on the results. In theory it should then more consistently create valid proofs.
rthnbgrredf 2y ago
Statistically-based reasoning also applies to humans. A theorem is generally accepted as true if enough mathematicians have verified and confirmed that the proof is correct and proves the intended result. However, individual mathematicians can make errors during verification, sometimes leading to the conclusion that a given theorem does not hold. Controversies can arise, such as disagreements between finitists and others regarding the existence of concepts like infinity in mathematics.
ericmcer 2y ago
That plays out for all the examples, except for the one where its answer was way off it and it corrected itself and attempted again. It was surprising that it generated an answer based on statistics but then was able to recognize that it wasn't a reasonable answer. I wonder how they are achieving that.

aidenn0 2y ago

I mean I could see my kid making this exact mistake on a word problem, so I suppose we've achieved "human like" reasoning at the expense of actually getting the answer we want?

nebster 2y ago
I tried to work out the problem myself first (using only the text) and accidentally used the diameter as the radius just like ChatGPT! Granted I haven't really tackled any maths problems for many years though.

hatenberg 2y ago

Chain of thought is nothing more than limiting the probability space enough that the model can provide the most likely answer. It's too much damn work to be useful.

yatz 2y ago

Once you correct the LLM, it will continue to provide the corrected answer until some time later, when it will again make the same mistake. At least, this has been my experience. If you are using LLM to pull answers programmatically and rely on their accuracy, here is what worked for the structured or numeric answers, such as numbers, JSON, etc. 1) Send the same prompt twice, including "Can you double check?" in the second prompt to force GPT to verify the answer. 2) If both answers are the same, you got the correct answer. 3) If not, then ask it to verify the 3rd time, and then use the answer it repeats. Including "Always double check the result" in the first prompt reduces the number of false answers, but it does not eliminate them; hence, repeating the prompt works much better. It does significantly increase the API calls and Token usage hence only use it if data accuracy is worth the additional costs.

groby_b 2y ago
> Once you correct the LLM, it will continue to provide the corrected answer until some time later, That is only true if you stay within the same chat. It is not true across chats. Context caching is something that a lot of folks would really really like to see. And jumping to a new chat is one of the core points of the OP: "I restarted with a slightly modified prompt:" The iterations before where mostly to figure out why the initial prompt went wrong. And AFAICT there's a good insight in the modified prompt - "Make no assumptions". Probably also "ensure you fully understand how it's labelled". And no, asking repeatedly doesn't necessarily give different answers, not even with "can you double check". There are quite a few examples where LLMs are consistently and proudly wrong. Don't use LLMs if 100% accuracy matters.
- sharemywin 2y ago
  so what would you use instead?
  - groby_b 2y ago
    Depends - what's your allowable error rate? What are you solving for?
- 2y ago

Auracle 2y ago

That’s funny. I practically got into a shouting match for the first time ever with ChatGPT earlier today because I was asking it to create a function to make a filled circle of pixels of a certain size using diameter and absolutely not radius (with some other constraints). This mattered because I wanted clear steps between 3,4,5,6 etc pixels wide, so the diameter was an int. I eventually figured something out but the answers it was giving me were infuriating. At some point instead of a radius it put “int halfSize = diameter / 2”.

scottmcdot 2y ago

In this context, what does the author mean by 'Zero-Shot'? From what I read on Wikipedia [1], it's about the model performing a task without specific training data for that task. But all of the experiments in the post seem to involve the model working without additional training data. [1] https://en.wikipedia.org/wiki/Zero-shot_learning https://en.wikipedia.org/wiki/Zero-shot_learning