4 ms·

This is pretty recent - the survey they ran (99 respondents) was August 18 to September 23 2025 and the field observations (watching developers for 45 minute th

by simonw 11mo ago

This is pretty recent - the survey they ran (99 respondents) was August 18 to September 23 2025 and the field observations (watching developers for 45 minute then a 30 minute interview, 13 participants) were August 1 to October 3. The models were mostly GPT-5 and Claude Sonnet 4. The study was too early to catch the 5.x Codex or Claude 4.5 models (bar one mention of Sonnet 4.5.) This is notable because a lot of academic papers take 6-12 months to come out, by which time the LLM space has often moved on by an entire model generation.

dheera 11mo ago
> academic papers take 6-12 months to come out It takes about 6 months to figure out how to get LaTeX to position figures where you want them, and then another 6 months to fight with reviewers
- zeristor 11mo ago
  Couldn't AI help with the LaTeX? Cutting it down to 6 minutes
  - jsrozner 11mo ago
    I have found it to be pretty bad at formatting tables
joenot443 11mo ago
Thanks Simon - always quick on the draw. Off your intuition, do you think the same study with Codex 5.2 and Opus 4.5 would see even better results?
- simonw 11mo ago
  Depends on the participants. If they're cutting-edge LLM users then yes, I think so. If they continue to use LLMs like they would have back in the first half of 2025 I'm not sure if a difference would be noticeable.
  - mkozlows 11mo ago
    I'm not remotely cutting edge (just switched from Cursor to Codex CLI, have no fancy tooling infrastructure, am not even vaguely considering git worktrees as a means of working), but Opus 4.5 and 5.2 Codex are both so clearly more competent than previous models that I've started just telling them to do high-level things rather than trying to break things down and give them subtasks. If people are really set in their ways, maybe they won't try anything beyond what old models can do, and won't notice a difference, but who's had time to get set in their ways with this stuff?
    - christophilus 11mo ago
      I mostly agree, but today, Opus 4.5 via Claude code did something pretty dumb stuff in my codebase— N queries where one would do, deep array comparison where a reference equality check would suffice, very complex web of nested conditionals which a competent developer would have never written, some edge cases where the backend endpoints didn’t properly verify user permissions before overwriting data, etc. It’s still hit or miss. The product “worked” when I tested it as a black box, but the code had a lot of rot in it already. Maybe that stuff no longer matters. Maybe it does. Time will tell.
      - remich 11mo ago
        I have had a lot of success lately when working with Opus 4.5 using both the Beads task tracking system and the array of skills under the umbrella of Bad Dave's Robot Army. I don't have a link handy, but you should be able to find it on GitHub. I use the specialized skills for different review tasks (like Architecture Review, Performance Review, Security Review, etc.) on every completed task in addition to my own manual review, and I find that that helps to keep things from getting out of hand.
      - ManuelKiessling 11mo ago
        As someone who’s responsible for some very clean codebases and some codebases that grew over many years, warts and all, I always wonder if being subjected to large amounts of not-exactly-wonderful code has the same effect on an LLM that it arguably also has on human developers (myself included occasionally): that they subconsciously lower their normally high bar for quality a bit, as in „well there‘s quite some smells here, let’s go a bit with the flow and not overdo the quality“.
reactordev 11mo ago
I knew in October the game had changed. Thanks for keeping us in the know.
- mikasisiki 11mo ago
  I'm not sure what you mean by “the game has changed.” If you’re referring to Opus 4.5, it’s somewhat better, but it’s far from game-changing.
  - reactordev 11mo ago
    You’re looking in from the outside. I’m on the inside. This next generation of models will show. It’s about to get wild. We now have extremely large context windows, we now have memory, we now have recall, we now can put an agent to the task for 24 hours.
trq126154 11mo ago
[flagged]
bbor 11mo ago
I’m glad someone else noticed the time frames — turns out the lead author here has published 28 distinct preprints in the past 60 days, almost all of which are marked as being officially published already/soon. Certainly some scientists are just absurdly efficient and all 28 involved teams, but that’s still a lot. Personally speaking, this gives me second thoughts about their dedication to truly accurately measuring something as notoriously tricky as corporate SWE performance. Any number of cut corners in a novel & empirical study like this would be hard to notice from the final product, especially for casual readers…TBH, the clickbait title doesn’t help either! I don’t have a specific critique on why 4 months is definitely too short to do it right tho. Just vibe-reviewing, I guess ;)
- aaronblohowiak 11mo ago
  are they a PI with a lab? in this field, does the PI get first or last author?
  - deleted 11mo ago
    [deleted]
ActionHank 11mo ago
For what it’s worth I know this is likely intended to read as the new generation of models will somehow better than any paper will be able to gauge, that hasn’t been my experience. Results are getting worse and less accurate, hell, I even had Claude drop some Chinese into a response out of the blue one day.
- danielbln 11mo ago
  I can absolutely not corroborate this, Opus 4.5 has been nothing but stellar.
- mannycalavera42 11mo ago
  same here. While getting a commandline for ffmpeg instead of giving me the option "soft-knee" it used "soft-膝" (where 膝 is the chinese for knee) was easy to spot and figure out but still... pretty rubbishy ¯ \ _ (ツ) _ / ¯
utopiah 11mo ago
> academic papers take 6-12 months to come out, by which time the LLM space has often moved on by an entire model generation. This is a recurring argument which I don't understand. Doesn't it simply mean that whatever conclusion they did was valid then? The research process is about approximating a better description of a phenomenon to understand it. It's not about providing a definitive answer. Being "an entire model generation" behind would be important if fundamental problems, e.g. no more hallucinations, would be solved but if it's going from incremental changes then most likely the conclusions remain correct. Which fundamental change (I don't think labeling newer models as "better" is sufficient) do you believe invalidate their conclusions in this specific context?
- soulofmischief 11mo ago
  2025 has been a wild year for agentic coding models. Cutting-edge models in January 2025 don't hold a candle to cutting edge models in December 2025. Just the jump from Sonnet 3.5 to 3.7 to 4.5, and Opus 4.5 has been pretty massive in terms of holistic reasoning, deep knowledge as well as better procedural and architectural adherence. GPT-5 Pro convinced me to pay $200/mo for an OpenAI subscription. Regular 5.2 models, and 5.2 codex, are leagues better than GPT-4 when it comes to solving problems procedurally, using tools, and deep discussion of scientific, mathematic, philosophical and engineering problems. Models have increasingly longer context, especially some Google models. OpenAI has released very good image models, and great editing-focused image models in general have been released. Predictably better multimodal inference over the short term is unlocking many cool near-term possibilities. Additionally, we have seen some incredible open source and open weight models released this year. Some fully commercially viable without restriction. And more and more smaller TTS/STT projects are in active development, with a few notable releases this year. Honestly, the landscape at the end of the year is impressive. There has been great work all over the place, almost too much to keep up with. I'm very interested in the Genie models and a few others. For an idea: At the beginning of the year, I was mildly successful getting at coding models to make changes in some of my codebases, but the more esoteric problems were out of reach. Progress in general was deliberate and required a lot of manual intervention. By comparison, in the last week I've prototyped six applications at levels that would take me days to weeks individually, often developing multiple at the same time, monitoring agentic workflows and intervening only when necessary, relying on long preproduction phases with architectural discussions and development of documentation, requirements, SDDs... and detailed code review and refactoring processes to ensure adherence to constraints. I'm morphing from a very busy solo developer into a very busy product manager.

by simonw 11mo ago

dheera 11mo ago

> academic papers take 6-12 months to come out It takes about 6 months to figure out how to get LaTeX to position figures where you want them, and then another 6 months to fight with reviewers

zeristor 11mo ago
Couldn't AI help with the LaTeX? Cutting it down to 6 minutes
- jsrozner 11mo ago
  I have found it to be pretty bad at formatting tables

joenot443 11mo ago

Thanks Simon - always quick on the draw. Off your intuition, do you think the same study with Codex 5.2 and Opus 4.5 would see even better results?

simonw 11mo ago
Depends on the participants. If they're cutting-edge LLM users then yes, I think so. If they continue to use LLMs like they would have back in the first half of 2025 I'm not sure if a difference would be noticeable.
- mkozlows 11mo ago
  I'm not remotely cutting edge (just switched from Cursor to Codex CLI, have no fancy tooling infrastructure, am not even vaguely considering git worktrees as a means of working), but Opus 4.5 and 5.2 Codex are both so clearly more competent than previous models that I've started just telling them to do high-level things rather than trying to break things down and give them subtasks. If people are really set in their ways, maybe they won't try anything beyond what old models can do, and won't notice a difference, but who's had time to get set in their ways with this stuff?
  - christophilus 11mo ago
    I mostly agree, but today, Opus 4.5 via Claude code did something pretty dumb stuff in my codebase— N queries where one would do, deep array comparison where a reference equality check would suffice, very complex web of nested conditionals which a competent developer would have never written, some edge cases where the backend endpoints didn’t properly verify user permissions before overwriting data, etc. It’s still hit or miss. The product “worked” when I tested it as a black box, but the code had a lot of rot in it already. Maybe that stuff no longer matters. Maybe it does. Time will tell.
    - remich 11mo ago
      I have had a lot of success lately when working with Opus 4.5 using both the Beads task tracking system and the array of skills under the umbrella of Bad Dave's Robot Army. I don't have a link handy, but you should be able to find it on GitHub. I use the specialized skills for different review tasks (like Architecture Review, Performance Review, Security Review, etc.) on every completed task in addition to my own manual review, and I find that that helps to keep things from getting out of hand.
    - ManuelKiessling 11mo ago
      As someone who’s responsible for some very clean codebases and some codebases that grew over many years, warts and all, I always wonder if being subjected to large amounts of not-exactly-wonderful code has the same effect on an LLM that it arguably also has on human developers (myself included occasionally): that they subconsciously lower their normally high bar for quality a bit, as in „well there‘s quite some smells here, let’s go a bit with the flow and not overdo the quality“.

reactordev 11mo ago

I knew in October the game had changed. Thanks for keeping us in the know.

mikasisiki 11mo ago
I'm not sure what you mean by “the game has changed.” If you’re referring to Opus 4.5, it’s somewhat better, but it’s far from game-changing.
- reactordev 11mo ago
  You’re looking in from the outside. I’m on the inside. This next generation of models will show. It’s about to get wild. We now have extremely large context windows, we now have memory, we now have recall, we now can put an agent to the task for 24 hours.

trq126154 11mo ago

[flagged]

bbor 11mo ago

I’m glad someone else noticed the time frames — turns out the lead author here has published 28 distinct preprints in the past 60 days, almost all of which are marked as being officially published already/soon. Certainly some scientists are just absurdly efficient and all 28 involved teams, but that’s still a lot. Personally speaking, this gives me second thoughts about their dedication to truly accurately measuring something as notoriously tricky as corporate SWE performance. Any number of cut corners in a novel & empirical study like this would be hard to notice from the final product, especially for casual readers…TBH, the clickbait title doesn’t help either! I don’t have a specific critique on why 4 months is definitely too short to do it right tho. Just vibe-reviewing, I guess ;)

aaronblohowiak 11mo ago
are they a PI with a lab? in this field, does the PI get first or last author?
- deleted 11mo ago
  [deleted]

ActionHank 11mo ago

For what it’s worth I know this is likely intended to read as the new generation of models will somehow better than any paper will be able to gauge, that hasn’t been my experience. Results are getting worse and less accurate, hell, I even had Claude drop some Chinese into a response out of the blue one day.

danielbln 11mo ago
I can absolutely not corroborate this, Opus 4.5 has been nothing but stellar.
mannycalavera42 11mo ago
same here. While getting a commandline for ffmpeg instead of giving me the option "soft-knee" it used "soft-膝" (where 膝 is the chinese for knee) was easy to spot and figure out but still... pretty rubbishy ¯ \ _ (ツ) _ / ¯

utopiah 11mo ago

> academic papers take 6-12 months to come out, by which time the LLM space has often moved on by an entire model generation. This is a recurring argument which I don't understand. Doesn't it simply mean that whatever conclusion they did was valid then? The research process is about approximating a better description of a phenomenon to understand it. It's not about providing a definitive answer. Being "an entire model generation" behind would be important if fundamental problems, e.g. no more hallucinations, would be solved but if it's going from incremental changes then most likely the conclusions remain correct. Which fundamental change (I don't think labeling newer models as "better" is sufficient) do you believe invalidate their conclusions in this specific context?

soulofmischief 11mo ago
2025 has been a wild year for agentic coding models. Cutting-edge models in January 2025 don't hold a candle to cutting edge models in December 2025. Just the jump from Sonnet 3.5 to 3.7 to 4.5, and Opus 4.5 has been pretty massive in terms of holistic reasoning, deep knowledge as well as better procedural and architectural adherence. GPT-5 Pro convinced me to pay $200/mo for an OpenAI subscription. Regular 5.2 models, and 5.2 codex, are leagues better than GPT-4 when it comes to solving problems procedurally, using tools, and deep discussion of scientific, mathematic, philosophical and engineering problems. Models have increasingly longer context, especially some Google models. OpenAI has released very good image models, and great editing-focused image models in general have been released. Predictably better multimodal inference over the short term is unlocking many cool near-term possibilities. Additionally, we have seen some incredible open source and open weight models released this year. Some fully commercially viable without restriction. And more and more smaller TTS/STT projects are in active development, with a few notable releases this year. Honestly, the landscape at the end of the year is impressive. There has been great work all over the place, almost too much to keep up with. I'm very interested in the Genie models and a few others. For an idea: At the beginning of the year, I was mildly successful getting at coding models to make changes in some of my codebases, but the more esoteric problems were out of reach. Progress in general was deliberate and required a lot of manual intervention. By comparison, in the last week I've prototyped six applications at levels that would take me days to weeks individually, often developing multiple at the same time, monitoring agentic workflows and intervening only when necessary, relying on long preproduction phases with architectural discussions and development of documentation, requirements, SDDs... and detailed code review and refactoring processes to ensure adherence to constraints. I'm morphing from a very busy solo developer into a very busy product manager.