5 ms·

Hi HN, OP here. I'd appreciate feedback from folks with deep model knowledge on a few technical claims in the essay. I want to make sure I'm getting the fundam

Hi HN, OP here. I'd appreciate feedback from folks with deep model knowledge on a few technical claims in the essay. I want to make sure I'm getting the fundamentals right. 1. On o1's arithmetic handling: I claim that when o1 multiplies large numbers, it generates Python code rather than calculating internally. I don't have full transparency into o1's internals. Is this accurate? 2. On model stagnation: I argue that fundamental model capabilities (especially code generation) have plateaued, and that tool orchestration is masking this. Do folks with hands-on experience building/evaluating models agree? 3. On alternative architectures: I suggest graph transformers that preserve semantic meaning at the word level as one possible path forward. For those working on novel architectures - what approaches look promising? Are graph-based architectures, sparse attention, or hybrid systems actually being pursued seriously in research labs? Would love to know your thoughts!

Workaccount2 1y ago
I don't think HN is a place for fact checking your blog.
- Terr_ 1y ago
  On the other hand, "that's technically wrong" is an extraordinarily popular staple of online geek discussion. :p
lawlessone 1y ago
All i can say as someone sorta lay is that math isn't an LLM's strength. Having them defer calculations to calculators or python code seems better than it guessing that 1+1 = 2 because it's past data says 1+1 = 2
cpa 1y ago
I don't think 2 is true: when OpenAI model won a gold medal in the math olympiads, it did so without tools or web search, just pure inference. Such a feat definitely would not have happened with o1.
- MoltenMan 1y ago
  True, but aren't the math (and competitive programming) achievements a bit different? They're specific models heavily RL'd on competition math problems. Obviously still ridiculously impressive, but if you haven't done competition math or programming before it's much more memorization of techniques than you might expect and it's much easier to RL on.
- simonw 1y ago
  Yeah, I confirmed this at the time. Neither OpenAI nor Gemini used tools as part of their IMO gold medal performances. Here's OpenAI's tweet about this: https://twitter.com/SebastienBubeck/status/1946577650405056722 https://twitter.com/SebastienBubeck/status/19465776504050567... > Just to spell it out as clearly as possible: a next-word prediction machine (because that's really what it is here, no tools no nothing) just produced genuinely creative proofs for hard, novel math problems at a level reached only by an elite handful of pre‑college prodigies. My notes: https://simonwillison.net/2025/Jul/19/openai-gold-medal-math-olympiad/ https://simonwillison.net/2025/Jul/19/openai-gold-medal-math... They DID use tools for the International Collegiate Programming Contest (ICPC) programming one though: https://twitter.com/ahelkky/status/1971652614950736194 https://twitter.com/ahelkky/status/1971652614950736194 > For OpenAI, the models had access to a code execution sandbox, so they could compile and test out their solutions. That was it though; no internet access.
  - emp17344 1y ago
    We still have next to no real information on how the models achieved the gold medal. It’s a little early to be confirming anything, especially when the main source is a Twitter thread initiated by a company known for “exaggerating” the truth.
    - simonw 1y ago
      If you're not going to believe researchers when they tell you how they did something then sure, we don't know how they did it. Given how much bad press OpenAI got just last week[1] when one one of their execs clumsily (and I would argue misleadingly) described a model achievement and then had to walk it back amid widespread headlines about their dishonesty, those researchers have a VERY strong incentive to tell the truth. [1] https://techcrunch.com/2025/10/19/openais-embarrassing-math/ https://techcrunch.com/2025/10/19/openais-embarrassing-math/
simonw 1y ago
1 isn't true. o1 doesn't have access to a Python interpreter unless you explicitly grant it access. If you call the OpenAI API for o1 and ask it to multiply two large numbers it cannot use Python to help it. Try this: curl https://api.openai.com/v1/responses \ -H "Content-Type: application/json" \ -H "Authorization: Bearer $OPENAI_API_KEY" \ -d '{ "model": "o1", "input": "Multiply 87654321 × 98765432", "reasoning": { "effort": "medium", "summary": "detailed" } }' Here's what I got back just now: https://gist.github.com/simonw/a6438aabdca7eed3eec52ed7df64eff0?permalink_comment_id=5821856 https://gist.github.com/simonw/a6438aabdca7eed3eec52ed7df64e... o1 correctly answered the multiplication by running a long multiplication process entirely through reasoning tokens.
- deleted 1y ago
  [deleted]
- alganet 1y ago
  I see this: > "tool_choice": "auto" > "parallel_tool_calls": true Can you remake the API call explicitly asking it to not perform any tool calls?
  - simonw 1y ago
    I'm doing that here. It only makes tool calls if you give it a JSON list of tools it can call. Those are its default settings whether or not there are tools configured. You can set tool_choice to the name of a specific tool in order to force it to use that tool. I added my comment here to show an example of an API call with Python enabled: https://news.ycombinator.com/item?id=45686779 https://news.ycombinator.com/item?id=45686779 Update: Looks like you can add "tool_choice": "none" to prevent even tools you have configured from being called. https://platform.openai.com/docs/api-reference/responses/create#responses-create-tool_choice https://platform.openai.com/docs/api-reference/responses/cre...
    - alganet 1y ago
      There are three possible generic values for `tool_choice`: none, auto and required. Can you remake the call explicitly using the value `none`? Maybe it's not using Python, but it's using something else. I think it's a good test. If you're right, then the response shouldn't change. Update: `auto` is ambiguous. It doesn't say whether is picking from your selection of tools or the pool of all available tools. Explicit is better than implicit. I think you should do the call with `none`, it can't hurt and it can prove me wrong.
anonymoushn 1y ago
I don't really know what you mean by "preserve semantic meaning at the word level." The significant misunderstanding about tokenization present elsewhere in the article is concerning, given that the proposed path forward is to do with replacing tokenization somehow.
- remich 1y ago
  Right, words don't have semantic meaning on their own, that meaning is derived from surrounding context. "Cat" is both an animal and a bash command.
ACCount37 1y ago
Wrong on every count, basically. 1. You can enable or disable tool use in most APIs. Generally, tools such as web search and Python interpreter give models an edge. The same is true for humans, so, no surprise. At the frontier, model performance keeps climbing - both with tool use enabled and with it disabled. 2. Model capabilities keep improving. Frontier models of today are both more capable at their peak, and pack more punch for their weight, figuratively and literally. Capability per trained model weight and capability per unit of inference compute are both rising. This is reflected directly in model pricing - "GPT-4 level of performance" is getting cheaper over time. 3. We're 3 years into the AI revolution. If I had ten bucks for every "breakthrough new architecture idea" I've seen in a meanwhile, I'd be able to buy a full GB200 NVL72 with that. As a rule: those "breakthroughs" aren't that. At best, they offer some incremental or area-specific improvements that could find their way into frontier models eventually. Think +4% performance across the board, or +30% to usable context length for the same amount of inference memory/compute, or a full generational leap but only in challenging image understanding tasks. There are some promising hybrid approaches, but none that do away with "autoregressive transformer with attention" altogether. So if you want a shiny new architecture to appear out of nowhere and bail you out of transformer woes? Prepare to be disappointed.
- throwthrowrow 1y ago
  Question #1 was on the model's ability to handle arithmetic. The answer to question seems to be unrelated, at least to me: "you can enable or disable tool use in most APIs". The original question still stands: do recent LLMs have an inherent knowledge of arithmetic, or do they have to offload the calculation to some other non-LLM system?
  - ACCount37 1y ago
    The knowledge was never the bottleneck for that, not since the days of GPT-3. The ability to execute on it was. Which includes, among other things, the underappreciated metacognitive skill of "being able to decide when to do math quick and dirty, in one forward pass, and when to write it out explicitly and solve it step by step". Today's frontier LLMs can do that. A lot of training for "reasoning" is just training for "execute on your knowledge reliably". They usually can solve math problems with no tool calls. But they will tool call for more complex math when given an option to.
XenophileJKO 1y ago
Point 2 is 1000% not true, the models have both gotten better at the overall act of coding, but have also gotten WAY better at USING tools. This isn't tool orchestration frameworks, this is knowing how and when to use tools effectively and it is largely inside the model. I would also say this is a fundamental model capability. This improved think->act->sense loop that they now form, exponentially increases the possible utility of the models. We are just starting to see this with gpt-5 and the 4+ series of Claude models.
- emp17344 1y ago
  Yes, the models have gotten better at using tools because tech companies have poured an insane amount of money into improving tools and integrating them with LLMs. Is this because the models have actually improved, or because the tools and integration methods have improved? I don’t think anyone actually knows.
  - XenophileJKO 1y ago
    The models have improved. They are using "arbitrary tools" better.
    - emp17344 1y ago
      I don’t know what you mean, because arbitrary tools don’t integrate with LLMs in the first place. Are you referring to MCP?
  - remich 1y ago
    But, isn't improving tools and the LLM's integration with them improving the model? Caveat that we don't fully understand how human intelligence works, but with humans it's generally true that skills are not static or siloed. Improving in one area can generate dividends in others. It's like how some professional football players improve their games by taking ballet lessons. Two very different skills, but the incorporation of one improves the other as well as the whole. I would argue that narrowly focusing on LLM performance via benchmarks before tool use is incorporated is interesting, but not particularly relevant to whether they are transformative, or even useful, as products.
mirekrusin 1y ago
Reasoning model doesn't imply tool calling – those shouldn't be conflated. Reasoning just means more implicit chain-of-thought. It can be emulated by non reasoning model by explicitly constructing prompt to perform longer step by step thought process. With reasoning models it just happens implicitly, some models allow for control over reasoning effort with special tokens. Those models are simply fine tuned to do it themselves without explicit dialogue from the user. Tool calling happens primarily on the client side. Research/web access mode etc made available by some providers (based on tool calling that they handle themselves) is not a property of a model, can be enabled on any model. Nothing plateaued from where I'm standing – new models are being trained, releases happen frequently with impressive integration speed. New models outperform previous ones. Models gain multi modality etc. Regarding alternative architectures – there are new ones proposed all the time. It's not easy to verify all of them at scale. Some ideas that are extending current state of art architectures end up in frontier models - but it takes time to train so lag does exist. There are also a lot of improvements that are hidden from public by commercial companies.
Legend2440 1y ago
>I claim that when o1 multiplies large numbers, it generates Python code rather than calculating internally. I don't have full transparency into o1's internals. Is this accurate? Both reasoning and non-reasoning models may choose to use the Python interpreter to solve math problems. This isn't hidden from the user; it will show the interpreter ("Analyzing...") and you can click on it to see the code it ran. It can also solve math problems by working through them step-by-step. In this case it will do long multiplication using the pencil-and-paper method, and it will show its work.
mxkopy 1y ago
Not affiliated with anyone, but I think the likes of OptNet (differentiable constraint optimization) are soon going to play a role in developing AI with precise deductive reasoning. More broadly I think what we’re looking for at the end of the day, AGI, is going come about from a diaspora of methods capturing the diverse aspects of what we recognize as intelligence. ‘Precise deductive reasoning’ is one capability out of many. Attention isn’t all you need, neither is compression, convex programming, what have you. The perceived “smoothness” or “unity” of our intelligence is an illusion like virtual memory hiding cache, and building it is going to look a lot more like stitching these capabilities together than deriving some deep and elegant equation.
kgeist 1y ago
Try running local LLMs like Qwen3 yourself. They can calculate accurately in their reasoning traces even if you don't give them access to coding tools. In fact, even mid-range models (32b params) under 4-bit quantization can perform pretty well. No need to make guesses, you can try it yourself!

Workaccount2 1y ago

I don't think HN is a place for fact checking your blog.

Terr_ 1y ago
On the other hand, "that's technically wrong" is an extraordinarily popular staple of online geek discussion. :p

lawlessone 1y ago

All i can say as someone sorta lay is that math isn't an LLM's strength. Having them defer calculations to calculators or python code seems better than it guessing that 1+1 = 2 because it's past data says 1+1 = 2

cpa 1y ago

I don't think 2 is true: when OpenAI model won a gold medal in the math olympiads, it did so without tools or web search, just pure inference. Such a feat definitely would not have happened with o1.

MoltenMan 1y ago
True, but aren't the math (and competitive programming) achievements a bit different? They're specific models heavily RL'd on competition math problems. Obviously still ridiculously impressive, but if you haven't done competition math or programming before it's much more memorization of techniques than you might expect and it's much easier to RL on.
simonw 1y ago
Yeah, I confirmed this at the time. Neither OpenAI nor Gemini used tools as part of their IMO gold medal performances. Here's OpenAI's tweet about this: https://twitter.com/SebastienBubeck/status/1946577650405056722 https://twitter.com/SebastienBubeck/status/19465776504050567... > Just to spell it out as clearly as possible: a next-word prediction machine (because that's really what it is here, no tools no nothing) just produced genuinely creative proofs for hard, novel math problems at a level reached only by an elite handful of pre‑college prodigies. My notes: https://simonwillison.net/2025/Jul/19/openai-gold-medal-math-olympiad/ https://simonwillison.net/2025/Jul/19/openai-gold-medal-math... They DID use tools for the International Collegiate Programming Contest (ICPC) programming one though: https://twitter.com/ahelkky/status/1971652614950736194 https://twitter.com/ahelkky/status/1971652614950736194 > For OpenAI, the models had access to a code execution sandbox, so they could compile and test out their solutions. That was it though; no internet access.
- emp17344 1y ago
  We still have next to no real information on how the models achieved the gold medal. It’s a little early to be confirming anything, especially when the main source is a Twitter thread initiated by a company known for “exaggerating” the truth.
  - simonw 1y ago
    If you're not going to believe researchers when they tell you how they did something then sure, we don't know how they did it. Given how much bad press OpenAI got just last week[1] when one one of their execs clumsily (and I would argue misleadingly) described a model achievement and then had to walk it back amid widespread headlines about their dishonesty, those researchers have a VERY strong incentive to tell the truth. [1] https://techcrunch.com/2025/10/19/openais-embarrassing-math/ https://techcrunch.com/2025/10/19/openais-embarrassing-math/

simonw 1y ago

1 isn't true. o1 doesn't have access to a Python interpreter unless you explicitly grant it access. If you call the OpenAI API for o1 and ask it to multiply two large numbers it cannot use Python to help it. Try this: curl https://api.openai.com/v1/responses \ -H "Content-Type: application/json" \ -H "Authorization: Bearer $OPENAI_API_KEY" \ -d '{ "model": "o1", "input": "Multiply 87654321 × 98765432", "reasoning": { "effort": "medium", "summary": "detailed" } }' Here's what I got back just now: https://gist.github.com/simonw/a6438aabdca7eed3eec52ed7df64eff0?permalink_comment_id=5821856 https://gist.github.com/simonw/a6438aabdca7eed3eec52ed7df64e... o1 correctly answered the multiplication by running a long multiplication process entirely through reasoning tokens.

deleted 1y ago
[deleted]
alganet 1y ago
I see this: > "tool_choice": "auto" > "parallel_tool_calls": true Can you remake the API call explicitly asking it to not perform any tool calls?
- simonw 1y ago
  I'm doing that here. It only makes tool calls if you give it a JSON list of tools it can call. Those are its default settings whether or not there are tools configured. You can set tool_choice to the name of a specific tool in order to force it to use that tool. I added my comment here to show an example of an API call with Python enabled: https://news.ycombinator.com/item?id=45686779 https://news.ycombinator.com/item?id=45686779 Update: Looks like you can add "tool_choice": "none" to prevent even tools you have configured from being called. https://platform.openai.com/docs/api-reference/responses/create#responses-create-tool_choice https://platform.openai.com/docs/api-reference/responses/cre...
  - alganet 1y ago
    There are three possible generic values for `tool_choice`: none, auto and required. Can you remake the call explicitly using the value `none`? Maybe it's not using Python, but it's using something else. I think it's a good test. If you're right, then the response shouldn't change. Update: `auto` is ambiguous. It doesn't say whether is picking from your selection of tools or the pool of all available tools. Explicit is better than implicit. I think you should do the call with `none`, it can't hurt and it can prove me wrong.

anonymoushn 1y ago

I don't really know what you mean by "preserve semantic meaning at the word level." The significant misunderstanding about tokenization present elsewhere in the article is concerning, given that the proposed path forward is to do with replacing tokenization somehow.

remich 1y ago
Right, words don't have semantic meaning on their own, that meaning is derived from surrounding context. "Cat" is both an animal and a bash command.

ACCount37 1y ago

Wrong on every count, basically. 1. You can enable or disable tool use in most APIs. Generally, tools such as web search and Python interpreter give models an edge. The same is true for humans, so, no surprise. At the frontier, model performance keeps climbing - both with tool use enabled and with it disabled. 2. Model capabilities keep improving. Frontier models of today are both more capable at their peak, and pack more punch for their weight, figuratively and literally. Capability per trained model weight and capability per unit of inference compute are both rising. This is reflected directly in model pricing - "GPT-4 level of performance" is getting cheaper over time. 3. We're 3 years into the AI revolution. If I had ten bucks for every "breakthrough new architecture idea" I've seen in a meanwhile, I'd be able to buy a full GB200 NVL72 with that. As a rule: those "breakthroughs" aren't that. At best, they offer some incremental or area-specific improvements that could find their way into frontier models eventually. Think +4% performance across the board, or +30% to usable context length for the same amount of inference memory/compute, or a full generational leap but only in challenging image understanding tasks. There are some promising hybrid approaches, but none that do away with "autoregressive transformer with attention" altogether. So if you want a shiny new architecture to appear out of nowhere and bail you out of transformer woes? Prepare to be disappointed.

throwthrowrow 1y ago
Question #1 was on the model's ability to handle arithmetic. The answer to question seems to be unrelated, at least to me: "you can enable or disable tool use in most APIs". The original question still stands: do recent LLMs have an inherent knowledge of arithmetic, or do they have to offload the calculation to some other non-LLM system?
- ACCount37 1y ago
  The knowledge was never the bottleneck for that, not since the days of GPT-3. The ability to execute on it was. Which includes, among other things, the underappreciated metacognitive skill of "being able to decide when to do math quick and dirty, in one forward pass, and when to write it out explicitly and solve it step by step". Today's frontier LLMs can do that. A lot of training for "reasoning" is just training for "execute on your knowledge reliably". They usually can solve math problems with no tool calls. But they will tool call for more complex math when given an option to.

XenophileJKO 1y ago

Point 2 is 1000% not true, the models have both gotten better at the overall act of coding, but have also gotten WAY better at USING tools. This isn't tool orchestration frameworks, this is knowing how and when to use tools effectively and it is largely inside the model. I would also say this is a fundamental model capability. This improved think->act->sense loop that they now form, exponentially increases the possible utility of the models. We are just starting to see this with gpt-5 and the 4+ series of Claude models.

emp17344 1y ago
Yes, the models have gotten better at using tools because tech companies have poured an insane amount of money into improving tools and integrating them with LLMs. Is this because the models have actually improved, or because the tools and integration methods have improved? I don’t think anyone actually knows.
- XenophileJKO 1y ago
  The models have improved. They are using "arbitrary tools" better.
  - emp17344 1y ago
    I don’t know what you mean, because arbitrary tools don’t integrate with LLMs in the first place. Are you referring to MCP?
- remich 1y ago
  But, isn't improving tools and the LLM's integration with them improving the model? Caveat that we don't fully understand how human intelligence works, but with humans it's generally true that skills are not static or siloed. Improving in one area can generate dividends in others. It's like how some professional football players improve their games by taking ballet lessons. Two very different skills, but the incorporation of one improves the other as well as the whole. I would argue that narrowly focusing on LLM performance via benchmarks before tool use is incorporated is interesting, but not particularly relevant to whether they are transformative, or even useful, as products.

mirekrusin 1y ago

Reasoning model doesn't imply tool calling – those shouldn't be conflated. Reasoning just means more implicit chain-of-thought. It can be emulated by non reasoning model by explicitly constructing prompt to perform longer step by step thought process. With reasoning models it just happens implicitly, some models allow for control over reasoning effort with special tokens. Those models are simply fine tuned to do it themselves without explicit dialogue from the user. Tool calling happens primarily on the client side. Research/web access mode etc made available by some providers (based on tool calling that they handle themselves) is not a property of a model, can be enabled on any model. Nothing plateaued from where I'm standing – new models are being trained, releases happen frequently with impressive integration speed. New models outperform previous ones. Models gain multi modality etc. Regarding alternative architectures – there are new ones proposed all the time. It's not easy to verify all of them at scale. Some ideas that are extending current state of art architectures end up in frontier models - but it takes time to train so lag does exist. There are also a lot of improvements that are hidden from public by commercial companies.

Legend2440 1y ago

>I claim that when o1 multiplies large numbers, it generates Python code rather than calculating internally. I don't have full transparency into o1's internals. Is this accurate? Both reasoning and non-reasoning models may choose to use the Python interpreter to solve math problems. This isn't hidden from the user; it will show the interpreter ("Analyzing...") and you can click on it to see the code it ran. It can also solve math problems by working through them step-by-step. In this case it will do long multiplication using the pencil-and-paper method, and it will show its work.

mxkopy 1y ago

Not affiliated with anyone, but I think the likes of OptNet (differentiable constraint optimization) are soon going to play a role in developing AI with precise deductive reasoning. More broadly I think what we’re looking for at the end of the day, AGI, is going come about from a diaspora of methods capturing the diverse aspects of what we recognize as intelligence. ‘Precise deductive reasoning’ is one capability out of many. Attention isn’t all you need, neither is compression, convex programming, what have you. The perceived “smoothness” or “unity” of our intelligence is an illusion like virtual memory hiding cache, and building it is going to look a lot more like stitching these capabilities together than deriving some deep and elegant equation.

kgeist 1y ago

Try running local LLMs like Qwen3 yourself. They can calculate accurately in their reasoning traces even if you don't give them access to coding tools. In fact, even mid-range models (32b params) under 4-bit quantization can perform pretty well. No need to make guesses, you can try it yourself!