4 ms·

Every so often I try out a GPT model for coding again, and manage to get tricked by the very sparse conversation style into thinking it's great for a couple of

by p1necone 11mo ago

Every so often I try out a GPT model for coding again, and manage to get tricked by the very sparse conversation style into thinking it's great for a couple of days (when it says nothing and then finishes producing code with a 'I did x, y and z' with no stupid 'you're absolutely' right sucking up and it works, it feels very good). But I always realize it's just smoke and mirrors - the actual quality of the code and the failure modes and stuff are just so much worse than claude and gemini.

logicchains 11mo ago
I find for difficult questions math and design questions GPT5 tends to produce better answers than Claude and Gemini.
- munk-a 11mo ago
  Could you clarify what you mean by design questions? I do agree that GPT5 tends to have a better agentic dispatch style for math questions but I've found it has really struggled with data model design.
jpalomaki 11mo ago
Can you give some concrete example of programming problem task GPT fails to solve? Interested, because I’ve been getting pretty good results with different tasks using the Codex.
- cmarschner 11mo ago
  Completely failed for me running the code it changed in a docker container i keep running. Claude did it flawlessly. It absolutely rocks at code reviews but ir‘s terrible in comparison generating code
  - peab 11mo ago
    It really depends on what kind of code. I've found it incredible for frontend dev, and for scripts. It falls apart in more complex projects and monorepos
- throwaway31131 11mo ago
  I posted this example before but academic papers on algorithms often have pseudo code but no actual code. I thought it would be handy to use AI to make the code from the paper so a few months ago I tried to use Claude (not GPT, because I only have access to Claude) to recreate C++ code to implement the algorithms in this paper as practice for me in LLM use and it didn’t go well. https://users.cs.duke.edu/~reif/paper/chen/graph/graph.pdf https://users.cs.duke.edu/~reif/paper/chen/graph/graph.pdf
  - threeducks 11mo ago
    I just tried it with GPT-5.1-Codex. The compression ratio is not amazing, so not sure if it really worked, but at least it ran without errors. A few ideas how to make it work for you: 1. You gave a link to a PDF, but you did not describe how you provided the content of the PDF to the model. It might only have read the text with something like pdftotext, which for this PDF results in a garbled mess. It is safer to convert the pages to PNG (e.g. with pdftoppm) and let the model read it from the pages. A prompt like "Transcribe these pages as markdown." should be sufficient. If you can not see what the model did, there is a chance it made things up. 2. You used C++, but Python is much easier to write. You can tell the model to translate the code to C++ once it works in Python. 3. Tell the model to write unit tests to verify that the individual components work as intended. 4. Use Agent Mode and tell the model to print something and to judge whether the output is sensible, so it can debug the code.
    - throwaway31131 11mo ago
      Interesting. Thanks for the suggestions.
- gloosx 11mo ago
  Try to ask it to write some GLSL shaders. Just describe what you want to see and then try to run the shaders it outputs. It can output a UV-map or the simple gradient right, but when it comes to shaders a bit more complex it most of the time will not compile or run properly, sometimes mix GLSL versions, sometimes just straight make up things which don't work or output what you want.
- kriro 11mo ago
  Library/API conflicts are the biggest pain point for me usually. Especially breaking changes. RLlib (currently 2.41.0) and Gymnasium (currently 0.29.0+) have ended in circles many times for me because they tend to be out of sync (for multi-agent environments). My go to test now is a simple hello world type card game like war, competitive multi-agent with rllib and gymnasium (pettingzoo tends to cause even more issues). Claude Sonnet 4.5 was able to figure out a way to resolve it eventually (around 7 fixes) and I let it create an rllib.md with all the fixes and pitfalls and am curious if feeding this file to the next experiment will lead to a one-shot. GPT-5 struggled more but haven't tried Codex on this yet so it's not exactly fair. All done with Copilot in agent mode, just prompting, no specs or anything.
tartoran 11mo ago
I'm starting with Claude at work but did have an okay experience with OpenAi so far. For clearly delimited tasks it does produce working code more often than not. I've seen some improvement on their side compared to say, last year. For something more complex and not clearly defined in advance, yes, it does produce plausible garbage and it goes off the rails a lot. I was migrating a project and asked ChatGPT to analyze the original code base and produce a migration plan. The result seemed good and encouraging because I didn't know much about that project at that time. But I ended up taking a different route and when I finished the migration (with bits of help from ChatGPT) I looked at the original migration plan out of curiosity since I had become more familiar with the project by now. And the migration plan was an absolutely useless and senseless hallucination.
- wahnfrieden 11mo ago
  Use Codex for coding work
herpdyderp 11mo ago
On the contrary, I cannot use the top Gemini and Claude models because their outputs are so out place and hard to integrate with my code bases. The GPT 5 models integrate with my code base's existing patterns seamlessly.
- inquirerGeneral 11mo ago
  You realize on some level all of these sort of anecdotes, though, are simply random coincidence .
- ta12653421 11mo ago
  Supply some relevant files of your codebase in the ClaudeAI project area in the right part of the browser. Usually it will understand your architecture, patterns, principles
  - herpdyderp 11mo ago
    I'm using AI in-editor, all the models have full access to my code base.
findjashua 11mo ago
NME at all - 5.1 codex has been the best by far.
- manmal 11mo ago
  How can you stand the excruciating slowness? Claude Code is running circles around codex. The most mundane tasks make it think for a minute before doing anything.
  - wahnfrieden 11mo ago
    By learning to parallelize my work. This also solved my problem with slow Xcode builds.
    - manmal 11mo ago
      Well you can’t edit files while Xcode is building or the compiler will throw up, so I‘m wondering what you mean here. You can’t even run swift test in 2 agents at the same time, because swift serializes access for some reason. Whenever I have more than 1 agent run Swift tests in a loop to fix things, and another one to build something, the latter will disturb the former and I need to cancel. And then there’s a lot of work that can’t be parallelized, like complex git rebases - well you can do other things in a worktree, but good luck merging that after you‘ve changed everything in the repo. Codex is really really bad at git.
      - spongebobstoes 11mo ago
        I use the web ui, easy to parallelize stuff to 90% done. manually finish the last 10% and a quick test
        wahnfrieden 11mo ago
        For Xcode projects?
      - wahnfrieden 11mo ago
kshacker 11mo ago
I am a novice programmer -- I have programmed for 35+ years now but I build and lose the skills moving between coder to manager to sales -- multiple times. Fresh IC since last week again :) I have coded starting with Fortran, RPG and COBOL and I have also coded Java and Scala. I know modern architecture but haven't done enough grunt work to make it work or to debug (and fix) a complex problem. Needless to say sometimes my eyes glaze over the code. And I write some code for my personal enjoyment, and I gave it to Claude 6-8 months back for improvement, it gave me a massive change log and it was quite risky so abandoned it. I tried this again with Gemini last week, I was more prepared and asked it to improve class by class, and for whatever reasons I got better answers -- changed code, with explanations, and when I asked it to split the refactor in smaller steps, it did so. Was a joy working on this over the thanksgiving holidays. It could break the changes in small pieces, talk through them as I evolved concepts learned previously, took my feedback and prioritization, and also gave me nuanced explanation of the business objectives I was trying to achieve. This is not to downplay claude, that is just the sequence of events narration. So while it may or may not work well for experienced programmers, it is such a helpful tool for people who know the domain or the concepts (or both) and struggle with details, since the tool can iron out a lot of details for you. My goal now is to have another project for winter holidays and then think through 4-6 hour AI assisted refactors over the weekends. Do note that this is a project of personal interest so not spending weekends for the big man.
- bovermyer 11mo ago
  I have never considered trying to apply Claude/Gemini/etc. to Fortran or COBOL. That would be interesting.
  - kshacker 11mo ago
    I was just giving my history :) but yes I am sure this could actually get us out of the COBOL lock-in which requires 70 years old programmers to continue working. The last article I could find on this is from 2020 though: https://www.cnbc.com/2020/04/06/new-jersey-seeks-cobol-programmers-to-fix-unemployment-system.html https://www.cnbc.com/2020/04/06/new-jersey-seeks-cobol-progr...
sharyphil 11mo ago
You're absolutely right! Somehow it doesn't get on my nerves (unlike Gemini with "Of course").
bsder 11mo ago
At this point you are now forced to use the "AI"s as code search tools--and it annoys me to no end. The problem is that the "AI"s can cough up code examples based upon proprietary codebases that you, as an individual, have no access to. That creates a significant quality differential between coders who only use publicly available search (Google, Github, etc.) vs those who use "AI" systems.
- dieortin 11mo ago
  How would the AIs have access to proprietary codebases?
  - milutinovici 11mo ago
    Microsoft owns github
stevedonovan 11mo ago
I've been getting great results from Codex. Can be a bit slow, but gets there. Writes good Rust, powers through integration test generation. So (again) we are just sharing anecdata
CheeseFromLidl 11mo ago
Same experience here. The more commonly known the stuff it regurgitates is, the fewer errors. But if you venture into RF electronics or embedded land, beware of it turning into a master of bs. Which makes sense for something that isn’t AI but LLM.

logicchains 11mo ago

I find for difficult questions math and design questions GPT5 tends to produce better answers than Claude and Gemini.

munk-a 11mo ago
Could you clarify what you mean by design questions? I do agree that GPT5 tends to have a better agentic dispatch style for math questions but I've found it has really struggled with data model design.

jpalomaki 11mo ago

Can you give some concrete example of programming problem task GPT fails to solve? Interested, because I’ve been getting pretty good results with different tasks using the Codex.

cmarschner 11mo ago
Completely failed for me running the code it changed in a docker container i keep running. Claude did it flawlessly. It absolutely rocks at code reviews but ir‘s terrible in comparison generating code
- peab 11mo ago
  It really depends on what kind of code. I've found it incredible for frontend dev, and for scripts. It falls apart in more complex projects and monorepos
throwaway31131 11mo ago
I posted this example before but academic papers on algorithms often have pseudo code but no actual code. I thought it would be handy to use AI to make the code from the paper so a few months ago I tried to use Claude (not GPT, because I only have access to Claude) to recreate C++ code to implement the algorithms in this paper as practice for me in LLM use and it didn’t go well. https://users.cs.duke.edu/~reif/paper/chen/graph/graph.pdf https://users.cs.duke.edu/~reif/paper/chen/graph/graph.pdf
- threeducks 11mo ago
  I just tried it with GPT-5.1-Codex. The compression ratio is not amazing, so not sure if it really worked, but at least it ran without errors. A few ideas how to make it work for you: 1. You gave a link to a PDF, but you did not describe how you provided the content of the PDF to the model. It might only have read the text with something like pdftotext, which for this PDF results in a garbled mess. It is safer to convert the pages to PNG (e.g. with pdftoppm) and let the model read it from the pages. A prompt like "Transcribe these pages as markdown." should be sufficient. If you can not see what the model did, there is a chance it made things up. 2. You used C++, but Python is much easier to write. You can tell the model to translate the code to C++ once it works in Python. 3. Tell the model to write unit tests to verify that the individual components work as intended. 4. Use Agent Mode and tell the model to print something and to judge whether the output is sensible, so it can debug the code.
  - throwaway31131 11mo ago
    Interesting. Thanks for the suggestions.
gloosx 11mo ago
Try to ask it to write some GLSL shaders. Just describe what you want to see and then try to run the shaders it outputs. It can output a UV-map or the simple gradient right, but when it comes to shaders a bit more complex it most of the time will not compile or run properly, sometimes mix GLSL versions, sometimes just straight make up things which don't work or output what you want.
kriro 11mo ago
Library/API conflicts are the biggest pain point for me usually. Especially breaking changes. RLlib (currently 2.41.0) and Gymnasium (currently 0.29.0+) have ended in circles many times for me because they tend to be out of sync (for multi-agent environments). My go to test now is a simple hello world type card game like war, competitive multi-agent with rllib and gymnasium (pettingzoo tends to cause even more issues). Claude Sonnet 4.5 was able to figure out a way to resolve it eventually (around 7 fixes) and I let it create an rllib.md with all the fixes and pitfalls and am curious if feeding this file to the next experiment will lead to a one-shot. GPT-5 struggled more but haven't tried Codex on this yet so it's not exactly fair. All done with Copilot in agent mode, just prompting, no specs or anything.

tartoran 11mo ago

I'm starting with Claude at work but did have an okay experience with OpenAi so far. For clearly delimited tasks it does produce working code more often than not. I've seen some improvement on their side compared to say, last year. For something more complex and not clearly defined in advance, yes, it does produce plausible garbage and it goes off the rails a lot. I was migrating a project and asked ChatGPT to analyze the original code base and produce a migration plan. The result seemed good and encouraging because I didn't know much about that project at that time. But I ended up taking a different route and when I finished the migration (with bits of help from ChatGPT) I looked at the original migration plan out of curiosity since I had become more familiar with the project by now. And the migration plan was an absolutely useless and senseless hallucination.

wahnfrieden 11mo ago
Use Codex for coding work

herpdyderp 11mo ago

On the contrary, I cannot use the top Gemini and Claude models because their outputs are so out place and hard to integrate with my code bases. The GPT 5 models integrate with my code base's existing patterns seamlessly.

inquirerGeneral 11mo ago
You realize on some level all of these sort of anecdotes, though, are simply random coincidence .
ta12653421 11mo ago
Supply some relevant files of your codebase in the ClaudeAI project area in the right part of the browser. Usually it will understand your architecture, patterns, principles
- herpdyderp 11mo ago
  I'm using AI in-editor, all the models have full access to my code base.

findjashua 11mo ago

NME at all - 5.1 codex has been the best by far.

manmal 11mo ago
How can you stand the excruciating slowness? Claude Code is running circles around codex. The most mundane tasks make it think for a minute before doing anything.
- wahnfrieden 11mo ago
  By learning to parallelize my work. This also solved my problem with slow Xcode builds.
  - manmal 11mo ago
    Well you can’t edit files while Xcode is building or the compiler will throw up, so I‘m wondering what you mean here. You can’t even run swift test in 2 agents at the same time, because swift serializes access for some reason. Whenever I have more than 1 agent run Swift tests in a loop to fix things, and another one to build something, the latter will disturb the former and I need to cancel. And then there’s a lot of work that can’t be parallelized, like complex git rebases - well you can do other things in a worktree, but good luck merging that after you‘ve changed everything in the repo. Codex is really really bad at git.
    - spongebobstoes 11mo ago
      I use the web ui, easy to parallelize stuff to 90% done. manually finish the last 10% and a quick test
      - wahnfrieden 11mo ago
        For Xcode projects?
    - wahnfrieden 11mo ago

kshacker 11mo ago

I am a novice programmer -- I have programmed for 35+ years now but I build and lose the skills moving between coder to manager to sales -- multiple times. Fresh IC since last week again :) I have coded starting with Fortran, RPG and COBOL and I have also coded Java and Scala. I know modern architecture but haven't done enough grunt work to make it work or to debug (and fix) a complex problem. Needless to say sometimes my eyes glaze over the code. And I write some code for my personal enjoyment, and I gave it to Claude 6-8 months back for improvement, it gave me a massive change log and it was quite risky so abandoned it. I tried this again with Gemini last week, I was more prepared and asked it to improve class by class, and for whatever reasons I got better answers -- changed code, with explanations, and when I asked it to split the refactor in smaller steps, it did so. Was a joy working on this over the thanksgiving holidays. It could break the changes in small pieces, talk through them as I evolved concepts learned previously, took my feedback and prioritization, and also gave me nuanced explanation of the business objectives I was trying to achieve. This is not to downplay claude, that is just the sequence of events narration. So while it may or may not work well for experienced programmers, it is such a helpful tool for people who know the domain or the concepts (or both) and struggle with details, since the tool can iron out a lot of details for you. My goal now is to have another project for winter holidays and then think through 4-6 hour AI assisted refactors over the weekends. Do note that this is a project of personal interest so not spending weekends for the big man.

bovermyer 11mo ago
I have never considered trying to apply Claude/Gemini/etc. to Fortran or COBOL. That would be interesting.
- kshacker 11mo ago
  I was just giving my history :) but yes I am sure this could actually get us out of the COBOL lock-in which requires 70 years old programmers to continue working. The last article I could find on this is from 2020 though: https://www.cnbc.com/2020/04/06/new-jersey-seeks-cobol-programmers-to-fix-unemployment-system.html https://www.cnbc.com/2020/04/06/new-jersey-seeks-cobol-progr...

sharyphil 11mo ago

You're absolutely right! Somehow it doesn't get on my nerves (unlike Gemini with "Of course").

bsder 11mo ago

At this point you are now forced to use the "AI"s as code search tools--and it annoys me to no end. The problem is that the "AI"s can cough up code examples based upon proprietary codebases that you, as an individual, have no access to. That creates a significant quality differential between coders who only use publicly available search (Google, Github, etc.) vs those who use "AI" systems.

dieortin 11mo ago
How would the AIs have access to proprietary codebases?
- milutinovici 11mo ago
  Microsoft owns github

stevedonovan 11mo ago

I've been getting great results from Codex. Can be a bit slow, but gets there. Writes good Rust, powers through integration test generation. So (again) we are just sharing anecdata

CheeseFromLidl 11mo ago

Same experience here. The more commonly known the stuff it regurgitates is, the fewer errors. But if you venture into RF electronics or embedded land, beware of it turning into a master of bs. Which makes sense for something that isn’t AI but LLM.