5 ms·

There are three possible generic values for `tool_choice`: none, auto and required. Can you remake the call explicitly using the value `none`? Maybe it's not

There are three possible generic values for `tool_choice`: none, auto and required. Can you remake the call explicitly using the value `none`? Maybe it's not using Python, but it's using something else. I think it's a good test. If you're right, then the response shouldn't change. Update: `auto` is ambiguous. It doesn't say whether is picking from your selection of tools or the pool of all available tools. Explicit is better than implicit. I think you should do the call with `none`, it can't hurt and it can prove me wrong.

simonw 1y ago
I just ran it like this and got the same correct result: curl https://api.openai.com/v1/responses \ -H "Content-Type: application/json" \ -H "Authorization: Bearer $OPENAI_API_KEY" \ -d '{ "model": "o1", "input": "Multiply 87654321 × 98765432", "reasoning": { "effort": "medium", "summary": "detailed" }, "tool_choice": "none" }' Result: https://gist.github.com/simonw/52888b6546dcfc6a9dcc75bcf171bc7e https://gist.github.com/simonw/52888b6546dcfc6a9dcc75bcf171b... I promise you it is not using anything else. It is performing long multiplication entirely through model reasoning. (I suggest getting your own OpenAI API key so you can try these things yourself.)
- simonw 1y ago
  And if you're still not convinced the definitive evidence is to run a reasoning model on your own machine. OpenAI's gpt-oss-20b is a 12GB download for LM Studio from https://lmstudio.ai/models/openai/gpt-oss-20b https://lmstudio.ai/models/openai/gpt-oss-20b It turns out it's powerful enough to solve this. Here's the thinking trace: We need product. Compute. Use approximate? Let's compute exactly. 87,654,321 * 98,765,432. We can use multiplication: Let's do 87,654,321 * 98,765,432. Compute using decomposition: 98,765,432 = 100,000,000 -1,234,568. So product = 87,654,321100,000,000 - 87,654,3211,234,568. First term: 8,765,432,100,000,000? Wait multiply: 87,654,321 *100,000,000 = 8,765,432,100,000,000. Now compute second term: 87,654,321 *1,234,568. Compute 1,234,568*87,654,321. Use maybe multiply 87,654,321 1,234,568 = 87,654,321(1,200,000+34,568). First: 1,200,000 = 87,654,32112100,000. 87,654,32112=1,051,851,852? Wait compute: 87,654,321*10=876,543,210; *2=175,308,642; sum =1,051,851,852. Then times100,000 =>105,185,185,200,000. Second: 34,568. Compute 87,654,32134,568 = 87,654,321*(30,000+4,568). 30,000: 87,654,3213=262,962,963; times10,000 =>2,629,629,630,000. 4,568: compute 87,654,3214,568 = *4000 + *500 + *60 + *8. 4000: 87,654,3214=350,617,284; times1000 =>350,617,284,000. *500: *5=438,271,605; times100 =>43,827,160,500. *60: *6=525,925,926; times10 =>5,259,259,260. *8: 700,834,568? Wait *8=700,834,568. Sum these: 350,617,284,000 +43,827,160,500 =394,444,444,500 +5,259,259,260 =399,703,703,760 +700,834,568 =400,404,538,328. Now add *30,000 part: 2,629,629,630,000 +400,404,538,328 =3,030,034,168,328. Now add to first part: 105,185,185,200,000 +3,030,034,168,328 =108,215,219,368,328. Thus product second term =108,215,219,368,328. Now subtract from first term: 8,765,432,100,000,000 -108,215,219,368,328 = 8,657,216,880,631,672. So answer: 8,657,216,880,631,672. And a screenshot: https://gist.github.com/simonw/a8929c0df5f2049816528715554208f0 https://gist.github.com/simonw/a8929c0df5f204981652871555420...
  - photonthug 1y ago
    Thanks for doing this. OpenAI is not in fact open, so referencing their claims as obviously true on anything else is just a non-starter. Counterpoint though, it's been a while since I've run this kind of experiment locally, so I started one too. For reasoning I only have qwen3:latest and I won't clutter the thread with the output, but it's complete junk. To summarize, with large numbers it goes nuts trying to find a trick or shortcut. After I cut off dead-ends in several trials, it always eventually considers long form addition, then ultimately rejects it as "tedious" and starts looking for "patterns". Wait, let me use the standard multiplication algorithm step by step, oh that's a lot of steps, break it down into parts. Let me think. Over ~45 minutes of thinking (I'm on CPU), but it basically cannot follow one strategy long enough to complete the work even if landed on a sensible approach. For multiplying two-digit numbers, it does better. Starts using the "manual way", messes up certain steps, then gets the right answer for sub-problems anyway because obviously those are memoized somewhere. But at least once, it got the correct answer with the correct approach. I think this raises the question, if you were to double the size of your input numbers and let the more powerful local model answer, could it still perform the process? Does that stop working for any reason at some point before the context window overflows?
  - rahimnathwani 1y ago
    % ollama run gpt-oss:20b >>> Calculate 87654321 * 98765432 Thinking... We need to compute product of 87654321 and 98765432. Straight multiplication. Let's compute. 87654321 * 98765432. We can attempt to compute using big integer multiplication. Maybe use decomposition. Let a=87654321, b=98765432. We can try to compute with Python-like mental? Let's try. Use method: b=98765432 = 100000000 - 1234568? Wait 100,000,000 - 1,234,568 = 98,765,432? Let's check: 100,000,000 - 1,234,568 = 98,765,432 correct. So product = a*(100,000,000 - 1,234,568) = a*100,000,000 - a*1,234,568. Compute a*100,000,000 = 8,765,432,100,000,000? Wait multiply 87,654,321 by 100,000,000 gives 8,765,432,100,000,000? Let's check: 87,654,321 * 100,000,000 = 8,765,432,100,000,000? Actually 87,654,321 * 100,000,000 = 8,765,432,100,000,000? Multiply by 10^8: shift 8 digits: 87,654,321 -> 8,765,432,100,000,000? Wait 87,654,321 times 10^8 = 8,765,432,100,000,000? Let's do digits: 87,654,321 * 100,000,000 = 8,765,432,100,000,000? Write 87,654,321 followed by 8 zeros: 8,765,432,100,000,000. yes. Now compute a*1,234,568. a = 87,654,321. Multiply by 1,234,568. Use decomposition: 1,234,568 = 1,200,000 + 34,568? Actually 1,234,568 = 1,000,000 + 234,568? Let's use 1,234,568 = 1,234,568. We'll do multiply. Option: use long multiplication: 87,654,321 * 1,234,568. We can also use a*1,234,568 = a*1,000,000 + a*234,568. Compute a*1,000,000 = 87,654,321,000,000. Now a*234,568. Compute a*234,568 = a*234,000 + a*568. Compute a*234,000 = a*234 * 1000. First a*234: a*200 + a*34. a*200 = 17,530,864,200. a*34 = a*30 + a*4. a*30 = 2,629,629,630. a*4 = 350,617,284. Sum a*34 = 2,629,629,630 + 350,617,284 = 2,980,246,914. Add a*200: 17,530,864,200 + 2,980,246,914 = 20,511,111,114. That is a*234. Then a*234,000 = 20,511,111,114,000. Now a*568 = a*500 + a*60 + a*8. a*500 = 43,827,160,500. a*60 = a*6*10. a*6 = 525,925,926; times10 => 5,259,259,260. a*8 = 701,234,568. Sum: 43,827,160,500 + 5,259,259,260 = 49,086,419,760. + 701,234,568 = 49,787,654,328. Now a*234,568 = a*234,000 + a*568 = 20,511,111,114,000 + 49,787,654,328 = 20,560,898,768,328. Now a*1,234,568 = a*1,000,000 + a*234,568 = 87,654,321,000,000 + 20,560,898,768,328 = 108,215,219,768,328. Now product: a*100,000,000 - a*1,234,568 = 8,765,432,100,000,000 - 108,215,219,768,328 = ? Compute: 8,765,432,100,000,000 -108,215,219,768,328 =8,657,216,880,231,672? Let's subtract carefully. 8,765,432,100,000,000 - 108,215,219,768,328 = 8,657,216,880,231,672. Check: subtract 108,215,219,768,328 from 8,765,432,100,000,000. 8,765,432,100,000,000 - 108,215,219,768,328 -------------------------------- = 8,657,216,880,231,672. Yes. Thus product is 8,657,216,880,231,672. Let's double-check via alternative method to ensure no mistakes. We can use big-int mental? Let's cross-check using another decomposition: Use b=98,765,432 = 99,000,000 - 234,568? Actually 99,000,000 - 234,568 = 98,765,432. So product = a*99,000,000 - a*234,568. Compute a*99,000,000 = a*99 * 1,000,000. a*99 = a*100 - a = 8,765,432,100 - 87,654,321 = 8,677,777,779? Wait compute: a=87,654,321. a*100 = 8,765,432,100. Subtract a gives 8,677,777,779. Yes. Then a*99,000,000 = 8,677,777,779 * 1,000,000 = 8,677,777,779,000,000. Now subtract a*234,568. We previously computed a*234,568 = 20,560,898,768,328. Thus product = 8,677,777,779,000,000 - 20,560,898,768,328 = ? Compute: 8,677,777,779,000,000 - 20,560,898,768,328 = 8,657,216,880,231,672. Same result. Good. Thus answer is 8,657,216,880,231,672. Provide in numeric form. Let's output. ...done thinking. The product is \[ 87654321 \times 98765432 = 8\,657\,216\,880\,231\,672 \]
- alganet 1y ago
  Looks legit. I can see this call now has a lot more tokens for the reasoning steps. Maybe that's normal variance though. (I don't have a particular interest in proving or disproving LLM things, so there's no incentive for me to get a key). There was an ambiguous point in the "proof", I just highlighted it.
  - simonw 1y ago
    If you want to write about LLMs I really strongly recommend getting an API key for the major vendors! It's really useful being able to run quick experiments like this one if you want to develop a deeper understanding of what they can and cannot do. You can also get an account with something like https://openrouter.ai/ https://openrouter.ai/ which gives you one key to use with multiple different backends. Or use GitHub Models which gives you free albeit limited access to a bunch at once. https://github.com/marketplace/models https://github.com/marketplace/models
    - alganet 1y ago
      I want to write about thinking crictically, specially but not limited to a software development context. Lots of people don't have resources to invest in LLMs (either self-hosted or not). They rely on what other people say. And people get caught in the hype all the time. As it turns out, lots of hype nowadays is around LLMs, so that's where I'll go. I was skeptic about LK99. Didn't had the resources to independently verify it. It doesn't mean I don't believe in superconductors or that I should have no say in it. Some of that hype will be justified, some will not. And that's exactly what I expect from this kind of technology.
      - simonw 1y ago
        At this point most of the top tier LLMs are available for free across most of the world. If people aren't experimenting with LLMs it's not due to financial cost, it's due to either time constraints (figuring this stuff out does take a bunch of work) or because they find the field either uninteresting or downright scary (or both).

simonw 1y ago

I just ran it like this and got the same correct result: curl https://api.openai.com/v1/responses \ -H "Content-Type: application/json" \ -H "Authorization: Bearer $OPENAI_API_KEY" \ -d '{ "model": "o1", "input": "Multiply 87654321 × 98765432", "reasoning": { "effort": "medium", "summary": "detailed" }, "tool_choice": "none" }' Result: https://gist.github.com/simonw/52888b6546dcfc6a9dcc75bcf171bc7e https://gist.github.com/simonw/52888b6546dcfc6a9dcc75bcf171b... I promise you it is not using anything else. It is performing long multiplication entirely through model reasoning. (I suggest getting your own OpenAI API key so you can try these things yourself.)

simonw 1y ago
And if you're still not convinced the definitive evidence is to run a reasoning model on your own machine. OpenAI's gpt-oss-20b is a 12GB download for LM Studio from https://lmstudio.ai/models/openai/gpt-oss-20b https://lmstudio.ai/models/openai/gpt-oss-20b It turns out it's powerful enough to solve this. Here's the thinking trace: We need product. Compute. Use approximate? Let's compute exactly. 87,654,321 * 98,765,432. We can use multiplication: Let's do 87,654,321 * 98,765,432. Compute using decomposition: 98,765,432 = 100,000,000 -1,234,568. So product = 87,654,321100,000,000 - 87,654,3211,234,568. First term: 8,765,432,100,000,000? Wait multiply: 87,654,321 *100,000,000 = 8,765,432,100,000,000. Now compute second term: 87,654,321 *1,234,568. Compute 1,234,568*87,654,321. Use maybe multiply 87,654,321 1,234,568 = 87,654,321(1,200,000+34,568). First: 1,200,000 = 87,654,32112100,000. 87,654,32112=1,051,851,852? Wait compute: 87,654,321*10=876,543,210; *2=175,308,642; sum =1,051,851,852. Then times100,000 =>105,185,185,200,000. Second: 34,568. Compute 87,654,32134,568 = 87,654,321*(30,000+4,568). 30,000: 87,654,3213=262,962,963; times10,000 =>2,629,629,630,000. 4,568: compute 87,654,3214,568 = *4000 + *500 + *60 + *8. 4000: 87,654,3214=350,617,284; times1000 =>350,617,284,000. *500: *5=438,271,605; times100 =>43,827,160,500. *60: *6=525,925,926; times10 =>5,259,259,260. *8: 700,834,568? Wait *8=700,834,568. Sum these: 350,617,284,000 +43,827,160,500 =394,444,444,500 +5,259,259,260 =399,703,703,760 +700,834,568 =400,404,538,328. Now add *30,000 part: 2,629,629,630,000 +400,404,538,328 =3,030,034,168,328. Now add to first part: 105,185,185,200,000 +3,030,034,168,328 =108,215,219,368,328. Thus product second term =108,215,219,368,328. Now subtract from first term: 8,765,432,100,000,000 -108,215,219,368,328 = 8,657,216,880,631,672. So answer: 8,657,216,880,631,672. And a screenshot: https://gist.github.com/simonw/a8929c0df5f2049816528715554208f0 https://gist.github.com/simonw/a8929c0df5f204981652871555420...
- photonthug 1y ago
  Thanks for doing this. OpenAI is not in fact open, so referencing their claims as obviously true on anything else is just a non-starter. Counterpoint though, it's been a while since I've run this kind of experiment locally, so I started one too. For reasoning I only have qwen3:latest and I won't clutter the thread with the output, but it's complete junk. To summarize, with large numbers it goes nuts trying to find a trick or shortcut. After I cut off dead-ends in several trials, it always eventually considers long form addition, then ultimately rejects it as "tedious" and starts looking for "patterns". Wait, let me use the standard multiplication algorithm step by step, oh that's a lot of steps, break it down into parts. Let me think. Over ~45 minutes of thinking (I'm on CPU), but it basically cannot follow one strategy long enough to complete the work even if landed on a sensible approach. For multiplying two-digit numbers, it does better. Starts using the "manual way", messes up certain steps, then gets the right answer for sub-problems anyway because obviously those are memoized somewhere. But at least once, it got the correct answer with the correct approach. I think this raises the question, if you were to double the size of your input numbers and let the more powerful local model answer, could it still perform the process? Does that stop working for any reason at some point before the context window overflows?
- rahimnathwani 1y ago
  % ollama run gpt-oss:20b >>> Calculate 87654321 * 98765432 Thinking... We need to compute product of 87654321 and 98765432. Straight multiplication. Let's compute. 87654321 * 98765432. We can attempt to compute using big integer multiplication. Maybe use decomposition. Let a=87654321, b=98765432. We can try to compute with Python-like mental? Let's try. Use method: b=98765432 = 100000000 - 1234568? Wait 100,000,000 - 1,234,568 = 98,765,432? Let's check: 100,000,000 - 1,234,568 = 98,765,432 correct. So product = a*(100,000,000 - 1,234,568) = a*100,000,000 - a*1,234,568. Compute a*100,000,000 = 8,765,432,100,000,000? Wait multiply 87,654,321 by 100,000,000 gives 8,765,432,100,000,000? Let's check: 87,654,321 * 100,000,000 = 8,765,432,100,000,000? Actually 87,654,321 * 100,000,000 = 8,765,432,100,000,000? Multiply by 10^8: shift 8 digits: 87,654,321 -> 8,765,432,100,000,000? Wait 87,654,321 times 10^8 = 8,765,432,100,000,000? Let's do digits: 87,654,321 * 100,000,000 = 8,765,432,100,000,000? Write 87,654,321 followed by 8 zeros: 8,765,432,100,000,000. yes. Now compute a*1,234,568. a = 87,654,321. Multiply by 1,234,568. Use decomposition: 1,234,568 = 1,200,000 + 34,568? Actually 1,234,568 = 1,000,000 + 234,568? Let's use 1,234,568 = 1,234,568. We'll do multiply. Option: use long multiplication: 87,654,321 * 1,234,568. We can also use a*1,234,568 = a*1,000,000 + a*234,568. Compute a*1,000,000 = 87,654,321,000,000. Now a*234,568. Compute a*234,568 = a*234,000 + a*568. Compute a*234,000 = a*234 * 1000. First a*234: a*200 + a*34. a*200 = 17,530,864,200. a*34 = a*30 + a*4. a*30 = 2,629,629,630. a*4 = 350,617,284. Sum a*34 = 2,629,629,630 + 350,617,284 = 2,980,246,914. Add a*200: 17,530,864,200 + 2,980,246,914 = 20,511,111,114. That is a*234. Then a*234,000 = 20,511,111,114,000. Now a*568 = a*500 + a*60 + a*8. a*500 = 43,827,160,500. a*60 = a*6*10. a*6 = 525,925,926; times10 => 5,259,259,260. a*8 = 701,234,568. Sum: 43,827,160,500 + 5,259,259,260 = 49,086,419,760. + 701,234,568 = 49,787,654,328. Now a*234,568 = a*234,000 + a*568 = 20,511,111,114,000 + 49,787,654,328 = 20,560,898,768,328. Now a*1,234,568 = a*1,000,000 + a*234,568 = 87,654,321,000,000 + 20,560,898,768,328 = 108,215,219,768,328. Now product: a*100,000,000 - a*1,234,568 = 8,765,432,100,000,000 - 108,215,219,768,328 = ? Compute: 8,765,432,100,000,000 -108,215,219,768,328 =8,657,216,880,231,672? Let's subtract carefully. 8,765,432,100,000,000 - 108,215,219,768,328 = 8,657,216,880,231,672. Check: subtract 108,215,219,768,328 from 8,765,432,100,000,000. 8,765,432,100,000,000 - 108,215,219,768,328 -------------------------------- = 8,657,216,880,231,672. Yes. Thus product is 8,657,216,880,231,672. Let's double-check via alternative method to ensure no mistakes. We can use big-int mental? Let's cross-check using another decomposition: Use b=98,765,432 = 99,000,000 - 234,568? Actually 99,000,000 - 234,568 = 98,765,432. So product = a*99,000,000 - a*234,568. Compute a*99,000,000 = a*99 * 1,000,000. a*99 = a*100 - a = 8,765,432,100 - 87,654,321 = 8,677,777,779? Wait compute: a=87,654,321. a*100 = 8,765,432,100. Subtract a gives 8,677,777,779. Yes. Then a*99,000,000 = 8,677,777,779 * 1,000,000 = 8,677,777,779,000,000. Now subtract a*234,568. We previously computed a*234,568 = 20,560,898,768,328. Thus product = 8,677,777,779,000,000 - 20,560,898,768,328 = ? Compute: 8,677,777,779,000,000 - 20,560,898,768,328 = 8,657,216,880,231,672. Same result. Good. Thus answer is 8,657,216,880,231,672. Provide in numeric form. Let's output. ...done thinking. The product is \[ 87654321 \times 98765432 = 8\,657\,216\,880\,231\,672 \]
alganet 1y ago
Looks legit. I can see this call now has a lot more tokens for the reasoning steps. Maybe that's normal variance though. (I don't have a particular interest in proving or disproving LLM things, so there's no incentive for me to get a key). There was an ambiguous point in the "proof", I just highlighted it.
- simonw 1y ago
  If you want to write about LLMs I really strongly recommend getting an API key for the major vendors! It's really useful being able to run quick experiments like this one if you want to develop a deeper understanding of what they can and cannot do. You can also get an account with something like https://openrouter.ai/ https://openrouter.ai/ which gives you one key to use with multiple different backends. Or use GitHub Models which gives you free albeit limited access to a bunch at once. https://github.com/marketplace/models https://github.com/marketplace/models
  - alganet 1y ago
    I want to write about thinking crictically, specially but not limited to a software development context. Lots of people don't have resources to invest in LLMs (either self-hosted or not). They rely on what other people say. And people get caught in the hype all the time. As it turns out, lots of hype nowadays is around LLMs, so that's where I'll go. I was skeptic about LK99. Didn't had the resources to independently verify it. It doesn't mean I don't believe in superconductors or that I should have no say in it. Some of that hype will be justified, some will not. And that's exactly what I expect from this kind of technology.
    - simonw 1y ago
      At this point most of the top tier LLMs are available for free across most of the world. If people aren't experimenting with LLMs it's not due to financial cost, it's due to either time constraints (figuring this stuff out does take a bunch of work) or because they find the field either uninteresting or downright scary (or both).