4 ms·

OP here. Thanks for the thoughtful reply. Curious if you’ve measured o1’s accuracy and token cost with tool use enabled vs disabled? Wondering if python sandbox

OP here. Thanks for the thoughtful reply. Curious if you’ve measured o1’s accuracy and token cost with tool use enabled vs disabled? Wondering if python sandbox gives higher accuracy and lower cost, since internal reasoning chains are longer and pricier. [Edit] Its probably premature to argue without the above data, but if we assume tool use gives ~100% accuracy and reasoning-only ~90%, then that 10% gap might represent the loss in the probabilistic model: either from functional ambiguity in the model itself or symbolic ambiguity from tokenization?

simonw 1y ago
I am 100% sure that using Python is faster and cheaper. My o1 call in https://gist.github.com/simonw/a6438aabdca7eed3eec52ed7df64eff0?permalink_comment_id=5821856 https://gist.github.com/simonw/a6438aabdca7eed3eec52ed7df64e... used 16 input tokens and produced 2357 output tokens (1664 were reasoning). At o1's price that's 14 cents! https://www.llm-prices.com/#it=16&ot=2357&ic=15&cic=7.5&oc=60 https://www.llm-prices.com/#it=16&ot=2357&ic=15&cic=7.5&oc=6... I can't call o1 with the Python tool via the API, so I'll have to provide the price for the GPT-5 example in https://gist.github.com/simonw/c53c373fab2596c20942cfbb235af9ae https://gist.github.com/simonw/c53c373fab2596c20942cfbb235af... - that one was 777 input tokens and 140 output tokens. Why 777 input tokens? That's a bit of a mystery to me - my assumption is that a bunch of extra system prompt stuff gets stuffed on describing that coding tool. GPT-5 is hugely cheaper than o1 so that cost 0.22 cents (almost a quarter of a cent) - but if o1 ran with the same number of tokens it would only cost 1.94 cents: https://www.llm-prices.com/#it=777&ot=130&sel=gpt-5%2Co1-preview https://www.llm-prices.com/#it=777&ot=130&sel=gpt-5%2Co1-pre...
- QueensGambit 1y ago
  wow, tool use seems to reduce total tokens by ~4 to 10× and cost by orders of magnitude. I wonder what the accuracy difference would be. I'm going to try multiplying larger and larger numbers to see how accuracy compares between tool use and pure reasoning.
- QueensGambit 1y ago
  I’ve updated my article (replaced GPT-5 with ChatGPT-5 in this section): When you ask the latest model, ChatGPT-5 to multiply two large numbers, it doesn't calculate. It generates Python code, executes it in a sandbox, and returns the result. Unlike ChatGPT-3, which at least attempted arithmetic internally (and often failed), ChatGPT-5 delegates computation to external tools. [1] And added this note: [1] There are 2 ways to multiply numbers in GPT-5: - Python mode, which uses python sandbox as mentioned above - No tool mode, which uses internal reasoning Python mode is approximately 2x more accurate than no tool mode in FrontierMath (26.3% vs 13.5% accuracy on expert level math). Python mode is also 4x to 10x more cost effective than no tool mode. The GPT-5 API uses no-tool mode by default (tools must be explicitly enabled in API calls), while ChatGPT UI likely uses Python mode by default since Advanced Data Analysis is enabled by default for all Plus, Team, and Enterprise subscribers. This creates a significant cost optimization for OpenAI in the consumer product, while API users bear the full cost of inefficient reasoning unless they manually configure tool use. --- Thanks again for flagging the inaccuracy, Simon! If you think any part of this update still misrepresents the model behavior, I’d love your input.

simonw 1y ago

I am 100% sure that using Python is faster and cheaper. My o1 call in https://gist.github.com/simonw/a6438aabdca7eed3eec52ed7df64eff0?permalink_comment_id=5821856 https://gist.github.com/simonw/a6438aabdca7eed3eec52ed7df64e... used 16 input tokens and produced 2357 output tokens (1664 were reasoning). At o1's price that's 14 cents! https://www.llm-prices.com/#it=16&ot=2357&ic=15&cic=7.5&oc=60 https://www.llm-prices.com/#it=16&ot=2357&ic=15&cic=7.5&oc=6... I can't call o1 with the Python tool via the API, so I'll have to provide the price for the GPT-5 example in https://gist.github.com/simonw/c53c373fab2596c20942cfbb235af9ae https://gist.github.com/simonw/c53c373fab2596c20942cfbb235af... - that one was 777 input tokens and 140 output tokens. Why 777 input tokens? That's a bit of a mystery to me - my assumption is that a bunch of extra system prompt stuff gets stuffed on describing that coding tool. GPT-5 is hugely cheaper than o1 so that cost 0.22 cents (almost a quarter of a cent) - but if o1 ran with the same number of tokens it would only cost 1.94 cents: https://www.llm-prices.com/#it=777&ot=130&sel=gpt-5%2Co1-preview https://www.llm-prices.com/#it=777&ot=130&sel=gpt-5%2Co1-pre...

QueensGambit 1y ago
wow, tool use seems to reduce total tokens by ~4 to 10× and cost by orders of magnitude. I wonder what the accuracy difference would be. I'm going to try multiplying larger and larger numbers to see how accuracy compares between tool use and pure reasoning.
QueensGambit 1y ago
I’ve updated my article (replaced GPT-5 with ChatGPT-5 in this section): When you ask the latest model, ChatGPT-5 to multiply two large numbers, it doesn't calculate. It generates Python code, executes it in a sandbox, and returns the result. Unlike ChatGPT-3, which at least attempted arithmetic internally (and often failed), ChatGPT-5 delegates computation to external tools. [1] And added this note: [1] There are 2 ways to multiply numbers in GPT-5: - Python mode, which uses python sandbox as mentioned above - No tool mode, which uses internal reasoning Python mode is approximately 2x more accurate than no tool mode in FrontierMath (26.3% vs 13.5% accuracy on expert level math). Python mode is also 4x to 10x more cost effective than no tool mode. The GPT-5 API uses no-tool mode by default (tools must be explicitly enabled in API calls), while ChatGPT UI likely uses Python mode by default since Advanced Data Analysis is enabled by default for all Plus, Team, and Enterprise subscribers. This creates a significant cost optimization for OpenAI in the consumer product, while API users bear the full cost of inefficient reasoning unless they manually configure tool use. --- Thanks again for flagging the inaccuracy, Simon! If you think any part of this update still misrepresents the model behavior, I’d love your input.