4 ms·

>LLMs using code to answer questions is nothing new, it's why the "how many Rs in strawberry" question doesn't trip them up anymore, because they can write a fe

>LLMs using code to answer questions is nothing new, it's why the "how many Rs in strawberry" question doesn't trip them up anymore, because they can write a few lines of Python to answer it, run that, and return the answer. False. It has nothing to do with tool use but just reasoning.

danpalmer 8mo ago
It's so easy to google this and find that they all do exactly this. Gemini: https://ai.google.dev/gemini-api/docs/code-execution https://ai.google.dev/gemini-api/docs/code-execution ChatGPT: https://help.openai.com/en/articles/8437071-data-analysis-with-chatgpt https://help.openai.com/en/articles/8437071-data-analysis-wi... Claude: https://claude.com/blog/analysis-tool https://claude.com/blog/analysis-tool Reasoning only gets you so far, even humans write code or use spreadsheets, calculators, etc, to get their answers to problems.
- simianwords 8mo ago
  you have just linked the fact that they have code executions but not proved that it is needed for strawberry problem. there are multiple ways to disprove this 1. GPT o1 was released and it never supported the tools and it easily solved the strawberry problem - it was named strawberry internally 2. you can run GPT 5.2-thinking in the API right now and deny access to any tools, it will still work 3. you can run deepseek locally without tools and run it, it will still work Overall this idea that LLM's cant reason and need tools to do that is misleading and false and easily disproven.
  - danpalmer 8mo ago
    Oh right you're very focused on specifically the strawberry problem. I just gave that as a throwaway example. It's a solution but not necessarily the solution for something that simple. My point was much more general, that code execution is a key part of these models ability to perform maths, analysis, and provide precise answers. It's not the only way, but a key way that's very efficient compared to more inference for CoT.
    - simianwords 8mo ago
      I agree that tool usage dramatically improves the utility of LLM's. But it is absolutely not needed for the strawberry problem. It can perform complicated arithmatic without tools - multiplying multiple 20 digit numbers, division and so on (to an extent).
FrustratedMonky 8mo ago
What is reasoning? I also can not multiply large numbers without a paper and pencil, and following an algorithm learned in school. That is the same as an LLM running some python, is the same as me following instructions to perform multiplication.

danpalmer 8mo ago

It's so easy to google this and find that they all do exactly this. Gemini: https://ai.google.dev/gemini-api/docs/code-execution https://ai.google.dev/gemini-api/docs/code-execution ChatGPT: https://help.openai.com/en/articles/8437071-data-analysis-with-chatgpt https://help.openai.com/en/articles/8437071-data-analysis-wi... Claude: https://claude.com/blog/analysis-tool https://claude.com/blog/analysis-tool Reasoning only gets you so far, even humans write code or use spreadsheets, calculators, etc, to get their answers to problems.

simianwords 8mo ago
you have just linked the fact that they have code executions but not proved that it is needed for strawberry problem. there are multiple ways to disprove this 1. GPT o1 was released and it never supported the tools and it easily solved the strawberry problem - it was named strawberry internally 2. you can run GPT 5.2-thinking in the API right now and deny access to any tools, it will still work 3. you can run deepseek locally without tools and run it, it will still work Overall this idea that LLM's cant reason and need tools to do that is misleading and false and easily disproven.
- danpalmer 8mo ago
  Oh right you're very focused on specifically the strawberry problem. I just gave that as a throwaway example. It's a solution but not necessarily the solution for something that simple. My point was much more general, that code execution is a key part of these models ability to perform maths, analysis, and provide precise answers. It's not the only way, but a key way that's very efficient compared to more inference for CoT.
  - simianwords 8mo ago
    I agree that tool usage dramatically improves the utility of LLM's. But it is absolutely not needed for the strawberry problem. It can perform complicated arithmatic without tools - multiplying multiple 20 digit numbers, division and so on (to an extent).

FrustratedMonky 8mo ago

What is reasoning? I also can not multiply large numbers without a paper and pencil, and following an algorithm learned in school. That is the same as an LLM running some python, is the same as me following instructions to perform multiplication.