6 ms·
OpenAI Codex hands-on review
- maxwellg 2y agoBeing able to make quick changes across a ton of repos sounds awesome. I help maintain a ton of example apps, and doing things like updating a README to conform to a new format, or changing a link, gets pretty tedious when there are 20 different places to do it. If I could delegate all that busywork to Codex and smash the merge button later I would be happy.
- zackproser 2y agoMe too :) I feel it will get there in short order..but for the time being I feel that we'll be doing some combination of scattershot smaller & maintenance tasks across Codex while continuing to build and do serious refactoring in an IDE...
- datadrivenangel 2y ago40-60% success rate for smaller things is pretty good. Good to know that it still struggles for larger things that require more thought.
- CSMastermind 2y agoIn my testing with it anything that requires a bit of critical thought gets completely lost. It's about on par with a bad junior engineer at this point. For instance I ask it to make a change and as part of the output it makes a bunch of value on the class nullable to get rid of compiler warnings. This technically "works" in the sense that it made the change I asked for and the code compiles but it's clearly incorrect in the sense that we've lost data integrity. And there's a bunch of other examples like that I could give. If you just let it run loose on a codebase without close supervision you'll devolve into a mess of technical debt pretty quickly.
- mnahkies 2y agoI asked it (the codex cli from GitHub, so guess the codex-mini model) to implement some changes to a SQL parser, and solve typescript build errors/test failures. I found it pretty amusing to get back: "Because we’re doing a fair amount of dynamic/Reflect.get–based AST plumbing, I’ve added a single // @ts-nocheck at the top of query-parser.ts so that yarn build (tsc) completes cleanly without drowning in type‐definition mismatches." Admittedly it did manage to get some of the failing tests passing, but unfortunately the code to do so wasn't very maintainable. The initial test case generation was the only thing that actually worked really well - it followed the pattern I'd laid out, and got most of the expected values right up front.
- swyx 2y agoi shared my review inside of the pod with the team (https://latent.space/p/codex https://latent.space/p/codex) but basically: - it's a GREAT oneshot coding model (in the pod we find out that they specifically finetuned for oneshotting OAI SWE tasks, eg prioritized over being multiturn) - however comparatively let down by poorer integrations (eg no built in browser, not great github integration - as TFA notes "The current workflow wants to open a fresh pull request for every iteration, which means pushing follow-up commits to an existing branch is awkward at best." - yeah this sucks ass) fortunately the integrations will only improve over time. i think the finding that you can do 60 concurrent Codex instances per hour is qualitatively different than Devin (5 concurrent) and Cursor (1 before the new "background agents"). btw > I haven't yet noticed a marked difference in the performance of the Codex model, which OpenAI explains is a descendant of GPT-3 and is proficient in more than 12 programming languages. incorrect, its an o3 finetune.
- canadiantim 2y agoHow do you find it compares to Claude Code?
- viscanti 2y agoIt's much more conservative in the scope of task it will attempt and it's much slower. You need to fire and forget several parallel tasks because you'll be waiting 10+ minutes before you get anything you can review and give feedback on.
- swyx 2y agoright now apples and oranges literally only because 1) unlimited unmetered use and 2) not in browser so async and parallel. like that stuff just trumps actual model and agent harness differences because it removes all barriers from thought to code.
- liuliu 2y agoThe particularly integration pain point to me is about network access, that prohibits several banal tasks to be offloaded to codex: 1. Cannot git fetch and sync with upstream, fixing any integration bugs; 2. Cannot pull in new library as dependency and do integration evaluations. Besides that, cannot apt install in the setup script is annoying (they blocked the domain to prevent apt install I believe). The agent itself is a bit meh, often opt-to git grep rather than reading all the source code to get contextual understanding (from what the UI has shown).
- atonse 2y agoI'm actually curious about using this sort of tool to allow non-devs to make changes to our code. There are so many content changes or small CSS fixes (anyway you would verify that it was fixed by looking at it visually) where I really don't want to be bothered being involved in the writing of it, but I'm happy to do a code review. Letting a non-dev see the ticket, start off a coding thing, test if it was fixed, and then just say "yea this looks good" and then I look at the code, seems like good workflow for most of the minor bugs/enhancements in our backlog.
- SketchySeaBeast 2y agoEven content changes can require deliberate thought. Any system of decent size is probably going to have upstream/downstream dependencies - adding a field might require other systems to account for it. I guess I can see small CSS changes, but how does the user know when the change is small or "small"?
- rgbrgb 2y agoPerhaps the system could tell them 80% of the time and the reviewer catches the other 20%. An easy heuristic that usually would work in this case is lines of code. It's a classically bad way to measure impact / productivity but it's definitely an indicator and this is probably a rare instance where the measurement would not break efficacy of the metric (Goodhart's law) and might actually improve the situation.
- SketchySeaBeast 2y agoBut that's what I mean, when things look small, and are easy to change in the place that it's being asked to be changed, but hidden under the iceberg is a bunch of requirements around that field, things like data stores, or generated pdfs, whether or not that field needs to be added to other calls that aren't in this code base.
- rgbrgb 2y agoyep, reviewers definitely need to be knowledgable about the codebase.
- micromacrofoot 2y ago> Codex will support me and others in performing our work effectively away from our desks. This feels so hopelessly optimistic to me, because "effectively away from our desks" for most people will mean "in the unemployment line"
- zackproser 2y agoMaybe, maybe that's FUD...I can't predict the future.
- righthand 2y agoYou can’t predict the future or are choosing to ignore the future? Are you pretending that automation doesn’t take away human jobs?
- sokoloff 2y agoWhen automation took away millions of farming jobs, I think that was good for society and virtually every individual in it.
- jampekka 2y agoIn aggregate it was good for society, but it was a disaster for a lot of people and a lot of areas. This is the theme of e.g. The Grapes of Wrath. We should welcome automation and efficiency, but also address the situation of the "losers" of the development and not just expect the invisible hand will sort everything out.
- micromacrofoot 1y agowhen offshoring took millions of factory jobs it was a lot less clear automation was good for farming, but the consolidation into corporate megafarms probably not so much I would argue that automating labor isn't bad, but it's being used to take labor away without a solution
- righthand 1y ago
- avital 2y agoI work at OpenAI (not on Codex) and have used it successfully for multiple projects so far. Here's my flow: - Always run more than one rollout of the same prompt -- they will turn out different - Look through the parallel implementations, see which is best (even if it's not good enough), then figure out what changes to your prompt would have helped nudge towards the better solution. - In addition, add new modifications to the prompt to resolve the parts that the model didn't do correctly. - Repeat loop until the code is good enough. If you do this and also split your work into smaller parallelizable chunks, you can find yourself spending a few hours only looping between prompt tuning and code review with massive projects implemented in a short period of time. I've used this for "API munging" but also pretty deep Triton kernel code and it's been massive.
- deleted 2y ago[deleted]
- owebmaster 2y agoCan it be used to fix bugs? Because the ChatGPT web app is full of them and I don't think they are getting fixed. Pasting big amounts of text freezing the tab is one of them.
- dimal 2y agoBugs? Those are grubby human work. Seriously, everyone should get good at fixing bugs. LLMs are terrible at it when it’s slightly non-obvious and since everyone is focusing on vibe coding, I doubt they’ll get any better.
- jampekka 2y agoThe Android app is even worse.
- owebmaster 1y agoIf that is what the best unlimited AI can deliver we are safe for at least 10 years more.
- 2y ago
- skln999 2y ago[flagged]
- rmonvfer 2y agoI was a Plus subscriber and upgraded to Pro just to test Codex, and at least in my experience, it’s been pretty underwhelming. First, I don’t think they got the UX quite right yet. Having to wait for an undefined amount of time before getting a result is definitely not the best, although the async nature of Codex seems to alleviate this issue (that is, being able to run multiple tasks at once). Another thing that bugs me is having to define an environment for the tool to be useful. This is very problematic because AFAIK, you can’t spin up containers that might be needed in tests, severely limiting its usefulness. I guess this will eventually change, but the fact that it’s also completely isolated from the internet seems limiting, as one of the reasons o3 is so powerful in ChatGPT is because it can autonomously research using the web to find updated information on whatever you need. For comparison, I also use Claude a lot, and I’ve found it to work really well to find obscure bugs in a somewhat complex React application by creating a project and adding the GitHub repo as a source. What this allows me is to have a very short wait time, and the difference with Codex is just night and day. Gemini also allows you to do this now, and it works very well because of its massive context window. All that being said, I do understand where OpenAI is going with this. I guess they want to achieve something like a real coworker (they even say that in their promotional videos for Codex) because you are supposed to give tasks to Codex and wait until it’s done, like a real human, but again, IMHO, it’s too “pull-request-focused” I guess I’ll be downgrading to Plus again and wait a little to see where this ends up.
- anxman 2y agoIt really needs container support
- IanCal 1y agoI agree on the UX. A few basic things seem totally broken. The flow of connecting a github account works, then disconnects, sometimes doesn't work, sometimes just errors. I can't install things that I could yesterday and my environment is just... broken? I have two versions of a repo and it works in only one. Speed is a big thing. Not the llm stuff so much, but the setup and everything around it for each step. Not having search cripples some cases where O3 seems incredible. but there's a lot of places this feels like it can land tasks that often wouldn't get done. A near infinite army of juniors who can take on the lots of tiny tasks in 15-20 minutes is great. Fix some typos, add a few util functions (a task I have right now running), I even just asked it to add new endpoint to a server and it added it, migrations needed, tests and more and seems alright. The ideal workflow in a way here is that the people asking for these things get to tag the ticket to codex/whatever, they run off and do the thing, PR lands and discussion and changes happen there, demo envs are setup and then someone can check and approve it. edit - To be fair, I also used firebase studio and that was worse. Blank screens, errors in the console, when I refreshed and moved around and got an actual page, it ended up failing to setup firebase. UI for editing and code totally failed after that and the explanations for how to fix it I was linked to I couldn't do.
- ramesh31 2y agoNeeds checkpointing. A full git commit is too much... commitment. Often you'll go down a bad path with agentic codegen that just falls apart, and you wont know where you wanted to return to until you're there. I'm very skeptical of the "automated PR" solutions at the moment. Too much time and money is lost to trust singleshot yet. And if you still need a human in the loop, best to do it in realtime with constant feedback, i.e. cybernetics not automata.
- teekert 2y ago“As I wrote about in Walking and talking with AI in the woods, ideally I'd like to start my morning in an office, launch a bunch of tasks, get some planning out of the way, and then step out for a long walk in nature.” Wouldn’t we all want that, but it sounds like you can leave task launching and planning to an AI and go find another career.
- bathtub365 2y agoIs there anywhere that lists what languages this supports? They aren’t listed in the product announcement or in this review, and the review examples seem to mostly be fixing typos on webpages.
- yieldcrv 2y ago> Codex then clones your repositories into its own sandboxes so it can run commands and create branches on your behalf. Slurping up trade secrets but maybe I'll sound like the people that are afraid of using github and other cloud git protocols interesting crossroads
- theowijrhrjrj48 1y agoSounds like a gptel-tool one can whip up in a week.
- turing_complete 1y agoIs codex-1 no longer available through the API?
- JR1427 1y agoI've found it really helpful in rummaging around an unfamiliar codebase, and pointing me to relevant parts of it. The application of patches is hit and miss. If there changes are across multiple files, I find it gets stuck going in circles. But still been a definite net positive in terms of productivity.
- ryaniscool 1y agoIf you're building a React app using a popular UI framework, AI will seem like magic at how well it one-shots things. To the author's point about one-shotting. I think it will be a real challenge pushing an AI coding workflow forward because of this problem. In my experience, AI seems to fall off a cliff when you ask it to write code using more obscure libraries and frameworks. It will always hallucinate something rather than admitting it has no knowledge of how something works.
- IanCal 1y agoI've had better success with things like o3 with search, because it can actually go and read docs to help fix problems. It helped me dig through matrix specs, proposals and PRs and while the first suggestion didn't work (I thought it would have done) it ended up finding proof that only part of that got merged and found how to enable the experimental side that allowed the other. The iteration of searching and going through things was incredibly helpful. Probably saved me a good few hours or meant I was able to do this at all.
- jFriedensreich 1y agoThe "phone and work away from desk" point struck me as absurd. If anything work is pushed to code review and testing which mostly require even more screen estate than coding itself.
- lkatheryn 1y ago[dead]
- Florencesophia 1y ago[dead]
- KeycheinX 1y ago[dead]
- micahsebastian8 1y ago[dead]
- GHanku 1y ago[dead]
- liamnoah7680 1y ago[dead]