3 ms·
I don't think finding the worst model and asking it to do things is the best way to go. GPT-5 pro without tools can easily solve your question and much harder
by simianwords 9mo ago
I don't think finding the worst model and asking it to do things is the best way to go.
GPT-5 pro without tools can easily solve your question and much harder ones.
Rather: does there exist a model that can perform these calculations reliably is a better way to falsify this claim.
Else we can always find the worst model with 1B parameters to falsify any claim made on LLM's.
- AlexeyBrin 9mo agoHow do you know GPT-5 does not call a Python interpreter remotely on OpenAI servers when you ask it to do arithmetic ? Your prompt goes to their servers, you have no way to know what happens there. The only way to be sure a model calls no tool is to run it locally and control the network.
- simianwords 9mo agoIt’s documented that pro version in api can’t run interpreter. Do you not believe the documentation?
- floam 9mo agoThey don’t give the customer access to it, for probably an obvious reason.
- rybosworld 9mo ago> GPT-5 pro without tools can easily solve your question and much harder ones. How are you able to use GPT-5 with tools turned off? Do you mean external tools (like searching the web)? My understanding is that GPT models always have access to python, and it isn't something you can turn off.
- simianwords 9mo agoWhat if we use the use the api? You can explicitly disable tool class. Is that enough?
- chickenimprint 9mo agoI chose Mistral because it's a large SOTA model that allows me to disable tool calling. If you try using ChatGPT, you'll see it goes into reasoning mode (It flashes the text "Working") and takes a while when asked to do arithmetic. Chatbots that don't forbid the user from viewing the reasoning chain of thought, such as Deepseek or Mistral, reveal that these chatbots work exactly as I described: ==== Thought for 1s Thinking Okay, the user wants to multiply two large numbers: 63,157,997,633 and 6,311,490,009. Since these numbers are quite large, I should use the code interpreter to handle this calculation accurately. Let's write a simple piece of code to compute this product. Code output generated python # Calculating the product of the two numbers number1 = 63157997633 number2 = 6311490009 product = number1 * number2 product ResultCopy 398621071049125148697 Thinking Great, the code interpreter has given us the product of the two numbers. Now, let's present the result to the user. ==== Falsifying my claim is simple: Find a model that verifiably does basic arithmetic correctly without relying on any external tools.
- throw310822 9mo agoCan you do basic arithmetic correctly without relying on external tools?
- bdangubic 9mo agoI can, since roughly first grade. but I cost significantly North of $200/month
- throw310822 9mo agoThen please multiply 13584638263947303 by 259472845392638 without using any tools (that is, in your head). Get back to me when you're done.
- bdangubic 9mo agowithout “tools” easy, I have pen and paper and first grade math :)