5 ms·

I see this: > "tool_choice": "auto" > "parallel_tool_calls": true Can you remake the API call explicitly asking it to not perform any tool calls?

simonw 1y ago
I'm doing that here. It only makes tool calls if you give it a JSON list of tools it can call. Those are its default settings whether or not there are tools configured. You can set tool_choice to the name of a specific tool in order to force it to use that tool. I added my comment here to show an example of an API call with Python enabled: https://news.ycombinator.com/item?id=45686779 https://news.ycombinator.com/item?id=45686779 Update: Looks like you can add "tool_choice": "none" to prevent even tools you have configured from being called. https://platform.openai.com/docs/api-reference/responses/create#responses-create-tool_choice https://platform.openai.com/docs/api-reference/responses/cre...
- alganet 1y ago
  There are three possible generic values for `tool_choice`: none, auto and required. Can you remake the call explicitly using the value `none`? Maybe it's not using Python, but it's using something else. I think it's a good test. If you're right, then the response shouldn't change. Update: `auto` is ambiguous. It doesn't say whether is picking from your selection of tools or the pool of all available tools. Explicit is better than implicit. I think you should do the call with `none`, it can't hurt and it can prove me wrong.
  - simonw 1y ago
    I just ran it like this and got the same correct result: curl https://api.openai.com/v1/responses \ -H "Content-Type: application/json" \ -H "Authorization: Bearer $OPENAI_API_KEY" \ -d '{ "model": "o1", "input": "Multiply 87654321 × 98765432", "reasoning": { "effort": "medium", "summary": "detailed" }, "tool_choice": "none" }' Result: https://gist.github.com/simonw/52888b6546dcfc6a9dcc75bcf171bc7e https://gist.github.com/simonw/52888b6546dcfc6a9dcc75bcf171b... I promise you it is not using anything else. It is performing long multiplication entirely through model reasoning. (I suggest getting your own OpenAI API key so you can try these things yourself.)
    - simonw 1y ago
      And if you're still not convinced the definitive evidence is to run a reasoning model on your own machine. OpenAI's gpt-oss-20b is a 12GB download for LM Studio from https://lmstudio.ai/models/openai/gpt-oss-20b https://lmstudio.ai/models/openai/gpt-oss-20b It turns out it's powerful enough to solve this. Here's the thinking trace: We need product. Compute. Use approximate? Let's compute exactly. 87,654,321 * 98,765,432. We can use multiplication: Let's do 87,654,321 * 98,765,432. Compute using decomposition: 98,765,432 = 100,000,000 -1,234,568. So product = 87,654,321100,000,000 - 87,654,3211,234,568. First term: 8,765,432,100,000,000? Wait multiply: 87,654,321 *100,000,000 = 8,765,432,100,000,000. Now compute second term: 87,654,321 *1,234,568. Compute 1,234,568*87,654,321. Use maybe multiply 87,654,321 1,234,568 = 87,654,321(1,200,000+34,568). First: 1,200,000 = 87,654,32112100,000. 87,654,32112=1,051,851,852? Wait compute: 87,654,321*10=876,543,210; *2=175,308,642; sum =1,051,851,852. Then times100,000 =>105,185,185,200,000. Second: 34,568. Compute 87,654,32134,568 = 87,654,321*(30,000+4,568). 30,000: 87,654,3213=262,962,963; times10,000 =>2,629,629,630,000. 4,568: compute 87,654,3214,568 = *4000 + *500 + *60 + *8. 4000: 87,654,3214=350,617,284; times1000 =>350,617,284,000. *500: *5=438,271,605; times100 =>43,827,160,500. *60: *6=525,925,926; times10 =>5,259,259,260. *8: 700,834,568? Wait *8=700,834,568. Sum these: 350,617,284,000 +43,827,160,500 =394,444,444,500 +5,259,259,260 =399,703,703,760 +700,834,568 =400,404,538,328. Now add *30,000 part: 2,629,629,630,000 +400,404,538,328 =3,030,034,168,328. Now add to first part: 105,185,185,200,000 +3,030,034,168,328 =108,215,219,368,328. Thus product second term =108,215,219,368,328. Now subtract from first term: 8,765,432,100,000,000 -108,215,219,368,328 = 8,657,216,880,631,672. So answer: 8,657,216,880,631,672. And a screenshot: https://gist.github.com/simonw/a8929c0df5f2049816528715554208f0 https://gist.github.com/simonw/a8929c0df5f204981652871555420...
      - photonthug 1y ago
        Thanks for doing this. OpenAI is not in fact open, so referencing their claims as obviously true on anything else is just a non-starter. Counterpoint though, it's been a while since I've run this kind of experiment locally, so I started one too. For reasoning I only have qwen3:latest and I won't clutter the thread with the output, but it's complete junk. To summarize, with large numbers it goes nuts trying to find a trick or shortcut. After I cut off dead-ends in several trials, it always eventually considers long form addition, then ultimately rejects it as "tedious" and starts looking for "patterns". Wait, let me use the standard multiplication algorithm step by step, oh that's a lot of steps, break it down into parts. Let me think. Over ~45 minutes of thinking (I'm on CPU), but it basically cannot follow one strategy long enough to complete the work even if landed on a sensible approach. For multiplying two-digit numbers, it does better. Starts using the "manual way", messes up certain steps, then gets the right answer for sub-problems anyway because obviously those are memoized somewhere. But at least once, it got the correct answer with the correct approach. I think this raises the question, if you were to double the size of your input numbers and let the more powerful local model answer, could it still perform the process? Does that stop working for any reason at some point before the context window overflows?

simonw 1y ago

I'm doing that here. It only makes tool calls if you give it a JSON list of tools it can call. Those are its default settings whether or not there are tools configured. You can set tool_choice to the name of a specific tool in order to force it to use that tool. I added my comment here to show an example of an API call with Python enabled: https://news.ycombinator.com/item?id=45686779 https://news.ycombinator.com/item?id=45686779 Update: Looks like you can add "tool_choice": "none" to prevent even tools you have configured from being called. https://platform.openai.com/docs/api-reference/responses/create#responses-create-tool_choice https://platform.openai.com/docs/api-reference/responses/cre...

alganet 1y ago
There are three possible generic values for `tool_choice`: none, auto and required. Can you remake the call explicitly using the value `none`? Maybe it's not using Python, but it's using something else. I think it's a good test. If you're right, then the response shouldn't change. Update: `auto` is ambiguous. It doesn't say whether is picking from your selection of tools or the pool of all available tools. Explicit is better than implicit. I think you should do the call with `none`, it can't hurt and it can prove me wrong.
- simonw 1y ago
  I just ran it like this and got the same correct result: curl https://api.openai.com/v1/responses \ -H "Content-Type: application/json" \ -H "Authorization: Bearer $OPENAI_API_KEY" \ -d '{ "model": "o1", "input": "Multiply 87654321 × 98765432", "reasoning": { "effort": "medium", "summary": "detailed" }, "tool_choice": "none" }' Result: https://gist.github.com/simonw/52888b6546dcfc6a9dcc75bcf171bc7e https://gist.github.com/simonw/52888b6546dcfc6a9dcc75bcf171b... I promise you it is not using anything else. It is performing long multiplication entirely through model reasoning. (I suggest getting your own OpenAI API key so you can try these things yourself.)
  - simonw 1y ago
    And if you're still not convinced the definitive evidence is to run a reasoning model on your own machine. OpenAI's gpt-oss-20b is a 12GB download for LM Studio from https://lmstudio.ai/models/openai/gpt-oss-20b https://lmstudio.ai/models/openai/gpt-oss-20b It turns out it's powerful enough to solve this. Here's the thinking trace: We need product. Compute. Use approximate? Let's compute exactly. 87,654,321 * 98,765,432. We can use multiplication: Let's do 87,654,321 * 98,765,432. Compute using decomposition: 98,765,432 = 100,000,000 -1,234,568. So product = 87,654,321100,000,000 - 87,654,3211,234,568. First term: 8,765,432,100,000,000? Wait multiply: 87,654,321 *100,000,000 = 8,765,432,100,000,000. Now compute second term: 87,654,321 *1,234,568. Compute 1,234,568*87,654,321. Use maybe multiply 87,654,321 1,234,568 = 87,654,321(1,200,000+34,568). First: 1,200,000 = 87,654,32112100,000. 87,654,32112=1,051,851,852? Wait compute: 87,654,321*10=876,543,210; *2=175,308,642; sum =1,051,851,852. Then times100,000 =>105,185,185,200,000. Second: 34,568. Compute 87,654,32134,568 = 87,654,321*(30,000+4,568). 30,000: 87,654,3213=262,962,963; times10,000 =>2,629,629,630,000. 4,568: compute 87,654,3214,568 = *4000 + *500 + *60 + *8. 4000: 87,654,3214=350,617,284; times1000 =>350,617,284,000. *500: *5=438,271,605; times100 =>43,827,160,500. *60: *6=525,925,926; times10 =>5,259,259,260. *8: 700,834,568? Wait *8=700,834,568. Sum these: 350,617,284,000 +43,827,160,500 =394,444,444,500 +5,259,259,260 =399,703,703,760 +700,834,568 =400,404,538,328. Now add *30,000 part: 2,629,629,630,000 +400,404,538,328 =3,030,034,168,328. Now add to first part: 105,185,185,200,000 +3,030,034,168,328 =108,215,219,368,328. Thus product second term =108,215,219,368,328. Now subtract from first term: 8,765,432,100,000,000 -108,215,219,368,328 = 8,657,216,880,631,672. So answer: 8,657,216,880,631,672. And a screenshot: https://gist.github.com/simonw/a8929c0df5f2049816528715554208f0 https://gist.github.com/simonw/a8929c0df5f204981652871555420...
    - photonthug 1y ago
      Thanks for doing this. OpenAI is not in fact open, so referencing their claims as obviously true on anything else is just a non-starter. Counterpoint though, it's been a while since I've run this kind of experiment locally, so I started one too. For reasoning I only have qwen3:latest and I won't clutter the thread with the output, but it's complete junk. To summarize, with large numbers it goes nuts trying to find a trick or shortcut. After I cut off dead-ends in several trials, it always eventually considers long form addition, then ultimately rejects it as "tedious" and starts looking for "patterns". Wait, let me use the standard multiplication algorithm step by step, oh that's a lot of steps, break it down into parts. Let me think. Over ~45 minutes of thinking (I'm on CPU), but it basically cannot follow one strategy long enough to complete the work even if landed on a sensible approach. For multiplying two-digit numbers, it does better. Starts using the "manual way", messes up certain steps, then gets the right answer for sub-problems anyway because obviously those are memoized somewhere. But at least once, it got the correct answer with the correct approach. I think this raises the question, if you were to double the size of your input numbers and let the more powerful local model answer, could it still perform the process? Does that stop working for any reason at some point before the context window overflows?