6 ms·

1 isn't true. o1 doesn't have access to a Python interpreter unless you explicitly grant it access. If you call the OpenAI API for o1 and ask it to multiply tw

by simonw 1y ago

1 isn't true. o1 doesn't have access to a Python interpreter unless you explicitly grant it access. If you call the OpenAI API for o1 and ask it to multiply two large numbers it cannot use Python to help it. Try this: curl https://api.openai.com/v1/responses \ -H "Content-Type: application/json" \ -H "Authorization: Bearer $OPENAI_API_KEY" \ -d '{ "model": "o1", "input": "Multiply 87654321 × 98765432", "reasoning": { "effort": "medium", "summary": "detailed" } }' Here's what I got back just now: https://gist.github.com/simonw/a6438aabdca7eed3eec52ed7df64eff0?permalink_comment_id=5821856 https://gist.github.com/simonw/a6438aabdca7eed3eec52ed7df64e... o1 correctly answered the multiplication by running a long multiplication process entirely through reasoning tokens.

deleted 1y ago
[deleted]
alganet 1y ago
I see this: > "tool_choice": "auto" > "parallel_tool_calls": true Can you remake the API call explicitly asking it to not perform any tool calls?
- simonw 1y ago
  I'm doing that here. It only makes tool calls if you give it a JSON list of tools it can call. Those are its default settings whether or not there are tools configured. You can set tool_choice to the name of a specific tool in order to force it to use that tool. I added my comment here to show an example of an API call with Python enabled: https://news.ycombinator.com/item?id=45686779 https://news.ycombinator.com/item?id=45686779 Update: Looks like you can add "tool_choice": "none" to prevent even tools you have configured from being called. https://platform.openai.com/docs/api-reference/responses/create#responses-create-tool_choice https://platform.openai.com/docs/api-reference/responses/cre...
  - alganet 1y ago
    There are three possible generic values for `tool_choice`: none, auto and required. Can you remake the call explicitly using the value `none`? Maybe it's not using Python, but it's using something else. I think it's a good test. If you're right, then the response shouldn't change. Update: `auto` is ambiguous. It doesn't say whether is picking from your selection of tools or the pool of all available tools. Explicit is better than implicit. I think you should do the call with `none`, it can't hurt and it can prove me wrong.
    - simonw 1y ago
      I just ran it like this and got the same correct result: curl https://api.openai.com/v1/responses \ -H "Content-Type: application/json" \ -H "Authorization: Bearer $OPENAI_API_KEY" \ -d '{ "model": "o1", "input": "Multiply 87654321 × 98765432", "reasoning": { "effort": "medium", "summary": "detailed" }, "tool_choice": "none" }' Result: https://gist.github.com/simonw/52888b6546dcfc6a9dcc75bcf171bc7e https://gist.github.com/simonw/52888b6546dcfc6a9dcc75bcf171b... I promise you it is not using anything else. It is performing long multiplication entirely through model reasoning. (I suggest getting your own OpenAI API key so you can try these things yourself.)
      - simonw 1y ago
        And if you're still not convinced the definitive evidence is to run a reasoning model on your own machine. OpenAI's gpt-oss-20b is a 12GB download for LM Studio from https://lmstudio.ai/models/openai/gpt-oss-20b https://lmstudio.ai/models/openai/gpt-oss-20b It turns out it's powerful enough to solve this. Here's the thinking trace: We need product. Compute. Use approximate? Let's compute exactly. 87,654,321 * 98,765,432. We can use multiplication: Let's do 87,654,321 * 98,765,432. Compute using decomposition: 98,765,432 = 100,000,000 -1,234,568. So product = 87,654,321100,000,000 - 87,654,3211,234,568. First term: 8,765,432,100,000,000? Wait multiply: 87,654,321 *100,000,000 = 8,765,432,100,000,000. Now compute second term: 87,654,321 *1,234,568. Compute 1,234,568*87,654,321. Use maybe multiply 87,654,321 1,234,568 = 87,654,321(1,200,000+34,568). First: 1,200,000 = 87,654,32112100,000. 87,654,32112=1,051,851,852? Wait compute: 87,654,321*10=876,543,210; *2=175,308,642; sum =1,051,851,852. Then times100,000 =>105,185,185,200,000. Second: 34,568. Compute 87,654,32134,568 = 87,654,321*(30,000+4,568). 30,000: 87,654,3213=262,962,963; times10,000 =>2,629,629,630,000. 4,568: compute 87,654,3214,568 = *4000 + *500 + *60 + *8. 4000: 87,654,3214=350,617,284; times1000 =>350,617,284,000. *500: *5=438,271,605; times100 =>43,827,160,500. *60: *6=525,925,926; times10 =>5,259,259,260. *8: 700,834,568? Wait *8=700,834,568. Sum these: 350,617,284,000 +43,827,160,500 =394,444,444,500 +5,259,259,260 =399,703,703,760 +700,834,568 =400,404,538,328. Now add *30,000 part: 2,629,629,630,000 +400,404,538,328 =3,030,034,168,328. Now add to first part: 105,185,185,200,000 +3,030,034,168,328 =108,215,219,368,328. Thus product second term =108,215,219,368,328. Now subtract from first term: 8,765,432,100,000,000 -108,215,219,368,328 = 8,657,216,880,631,672. So answer: 8,657,216,880,631,672. And a screenshot: https://gist.github.com/simonw/a8929c0df5f2049816528715554208f0 https://gist.github.com/simonw/a8929c0df5f204981652871555420...
simonw 1y ago
I know this isn't using tools (e.g. the Python interpreter) because you have to turn those on explicitly. That's not actually supported for o1 in the API but you can do it for GPT-5 like this: curl https://api.openai.com/v1/responses \ -H "Content-Type: application/json" \ -H "Authorization: Bearer $OPENAI_API_KEY" \ -d '{ "model": "gpt-5", "input": "Multiply 87654321 × 98765432", "reasoning": { "effort": "medium", "summary": "detailed" }, "tools": [ { "type": "code_interpreter", "container": {"type": "auto"} } ] }' Here's the response: https://gist.github.com/simonw/c53c373fab2596c20942cfbb235af9ae https://gist.github.com/simonw/c53c373fab2596c20942cfbb235af... Note this bit where the code interpreter Python tool is called: { "id": "rs_080a5801ca14ad990068fa91f2779081a0ad166ee263153d98", "type": "reasoning", "summary": [ { "type": "summary_text", "text": "**Calculating large product**\n\nI see that I need to compute the product of two large numbers, which involves big integer multiplication. It\u2019s a straightforward task, and given that I can use Python, that seems like the best route to avoid any potential errors. The user specifically asked for this multiplication, so I\u2019ll go ahead and use the python tool for accurate analysis. Let\u2019s get started on that!" } ] }, { "id": "ci_080a5801ca14ad990068fa91f4dbe481a09eb646af049541c6", "type": "code_interpreter_call", "status": "completed", "code": "a = 87654321\r\nb = 98765432\r\na*b", "container_id": "cntr_68fa91f12f008191a359f1eeaed561290c438cc21b3fc083", "outputs": null }
deleted 1y ago
[deleted]

by simonw 1y ago

deleted 1y ago

[deleted]

alganet 1y ago

I see this: > "tool_choice": "auto" > "parallel_tool_calls": true Can you remake the API call explicitly asking it to not perform any tool calls?

simonw 1y ago
I'm doing that here. It only makes tool calls if you give it a JSON list of tools it can call. Those are its default settings whether or not there are tools configured. You can set tool_choice to the name of a specific tool in order to force it to use that tool. I added my comment here to show an example of an API call with Python enabled: https://news.ycombinator.com/item?id=45686779 https://news.ycombinator.com/item?id=45686779 Update: Looks like you can add "tool_choice": "none" to prevent even tools you have configured from being called. https://platform.openai.com/docs/api-reference/responses/create#responses-create-tool_choice https://platform.openai.com/docs/api-reference/responses/cre...
- alganet 1y ago
  There are three possible generic values for `tool_choice`: none, auto and required. Can you remake the call explicitly using the value `none`? Maybe it's not using Python, but it's using something else. I think it's a good test. If you're right, then the response shouldn't change. Update: `auto` is ambiguous. It doesn't say whether is picking from your selection of tools or the pool of all available tools. Explicit is better than implicit. I think you should do the call with `none`, it can't hurt and it can prove me wrong.
  - simonw 1y ago
    I just ran it like this and got the same correct result: curl https://api.openai.com/v1/responses \ -H "Content-Type: application/json" \ -H "Authorization: Bearer $OPENAI_API_KEY" \ -d '{ "model": "o1", "input": "Multiply 87654321 × 98765432", "reasoning": { "effort": "medium", "summary": "detailed" }, "tool_choice": "none" }' Result: https://gist.github.com/simonw/52888b6546dcfc6a9dcc75bcf171bc7e https://gist.github.com/simonw/52888b6546dcfc6a9dcc75bcf171b... I promise you it is not using anything else. It is performing long multiplication entirely through model reasoning. (I suggest getting your own OpenAI API key so you can try these things yourself.)
    - simonw 1y ago
      And if you're still not convinced the definitive evidence is to run a reasoning model on your own machine. OpenAI's gpt-oss-20b is a 12GB download for LM Studio from https://lmstudio.ai/models/openai/gpt-oss-20b https://lmstudio.ai/models/openai/gpt-oss-20b It turns out it's powerful enough to solve this. Here's the thinking trace: We need product. Compute. Use approximate? Let's compute exactly. 87,654,321 * 98,765,432. We can use multiplication: Let's do 87,654,321 * 98,765,432. Compute using decomposition: 98,765,432 = 100,000,000 -1,234,568. So product = 87,654,321100,000,000 - 87,654,3211,234,568. First term: 8,765,432,100,000,000? Wait multiply: 87,654,321 *100,000,000 = 8,765,432,100,000,000. Now compute second term: 87,654,321 *1,234,568. Compute 1,234,568*87,654,321. Use maybe multiply 87,654,321 1,234,568 = 87,654,321(1,200,000+34,568). First: 1,200,000 = 87,654,32112100,000. 87,654,32112=1,051,851,852? Wait compute: 87,654,321*10=876,543,210; *2=175,308,642; sum =1,051,851,852. Then times100,000 =>105,185,185,200,000. Second: 34,568. Compute 87,654,32134,568 = 87,654,321*(30,000+4,568). 30,000: 87,654,3213=262,962,963; times10,000 =>2,629,629,630,000. 4,568: compute 87,654,3214,568 = *4000 + *500 + *60 + *8. 4000: 87,654,3214=350,617,284; times1000 =>350,617,284,000. *500: *5=438,271,605; times100 =>43,827,160,500. *60: *6=525,925,926; times10 =>5,259,259,260. *8: 700,834,568? Wait *8=700,834,568. Sum these: 350,617,284,000 +43,827,160,500 =394,444,444,500 +5,259,259,260 =399,703,703,760 +700,834,568 =400,404,538,328. Now add *30,000 part: 2,629,629,630,000 +400,404,538,328 =3,030,034,168,328. Now add to first part: 105,185,185,200,000 +3,030,034,168,328 =108,215,219,368,328. Thus product second term =108,215,219,368,328. Now subtract from first term: 8,765,432,100,000,000 -108,215,219,368,328 = 8,657,216,880,631,672. So answer: 8,657,216,880,631,672. And a screenshot: https://gist.github.com/simonw/a8929c0df5f2049816528715554208f0 https://gist.github.com/simonw/a8929c0df5f204981652871555420...

simonw 1y ago

I know this isn't using tools (e.g. the Python interpreter) because you have to turn those on explicitly. That's not actually supported for o1 in the API but you can do it for GPT-5 like this: curl https://api.openai.com/v1/responses \ -H "Content-Type: application/json" \ -H "Authorization: Bearer $OPENAI_API_KEY" \ -d '{ "model": "gpt-5", "input": "Multiply 87654321 × 98765432", "reasoning": { "effort": "medium", "summary": "detailed" }, "tools": [ { "type": "code_interpreter", "container": {"type": "auto"} } ] }' Here's the response: https://gist.github.com/simonw/c53c373fab2596c20942cfbb235af9ae https://gist.github.com/simonw/c53c373fab2596c20942cfbb235af... Note this bit where the code interpreter Python tool is called: { "id": "rs_080a5801ca14ad990068fa91f2779081a0ad166ee263153d98", "type": "reasoning", "summary": [ { "type": "summary_text", "text": "**Calculating large product**\n\nI see that I need to compute the product of two large numbers, which involves big integer multiplication. It\u2019s a straightforward task, and given that I can use Python, that seems like the best route to avoid any potential errors. The user specifically asked for this multiplication, so I\u2019ll go ahead and use the python tool for accurate analysis. Let\u2019s get started on that!" } ] }, { "id": "ci_080a5801ca14ad990068fa91f4dbe481a09eb646af049541c6", "type": "code_interpreter_call", "status": "completed", "code": "a = 87654321\r\nb = 98765432\r\na*b", "container_id": "cntr_68fa91f12f008191a359f1eeaed561290c438cc21b3fc083", "outputs": null }

deleted 1y ago

[deleted]