13 ms·

A Research Preview of Codex

511 points by meetpateltech 1y ago

deleted 1y ago
[deleted]
haffi112 1y ago
(watching live) I'm wondering how it performs on the METR benchmark (https://metr.org/blog/2025-03-19-measuring-ai-ability-to-complete-long-tasks/ https://metr.org/blog/2025-03-19-measuring-ai-ability-to-com...).
deleted 1y ago
[deleted]
- deleted 1y ago
  [deleted]
colesantiago 1y ago
I think the benchmark test for these programming agents that I would like to see an Agent making a flawless PR or patch to the BSD / Linux kernel. This should be possible today and surely Linus would also see this in the future.
- _kb 1y ago
  There’s a fairly pragmatic discussion in that exact topic with Linus here: https://youtu.be/VHHT6W-N0ak https://youtu.be/VHHT6W-N0ak.
deleted 1y ago
[deleted]
- deleted 1y ago
  [deleted]
_jayhack_ 1y ago
[flagged]
- esafak 1y ago
  An AI not for the IDE... for PMs?
tptacek 1y ago
Maddening: "codex" is also the name of their open-source Claude-Code-alike, and was previously the name of an at-the-time frontier coding model. It's like they name things just to fuck with us.
- tekacs 1y ago
  So -- that client-side thing is _technically_ called `codex-cli` (in the parent 'codex' repo, which looks like a monorepo?). Still super confusing, though! I feel like companies working with and shipping LLMs would do well to remember that it's not just humans who get confused by this, but LLMs themselves... it makes for a painful time, sending off a request and noting that a third of the way into its reasoning that the model has gotten tow things with almost-identical names confused.
  - tough 1y ago
    they also have a dual implementation on rust and typescript there's codex-rs in that monorepo
    - fabmilo 1y ago
      more excited about the rust impl than the typescript one.
      - tptacek 1y ago
        Besides packaging of their releases, what possible difference could that make in this problem domain?
        tough 1y ago
        I just think it's nice to have open source code to reference so maybe he meant just in that -educational- way, certainly more to learn from the rust one than the TS one for most folks? even if the problem-space doesn't require system-level safety code indeed
bilal4hmed 1y ago
[flagged]
- deleted 1y ago
  [deleted]
prhn 1y ago
Is anyone using any of these tools to write non boilerplate code? I'm very interested. In my experience ChatGPT and Gemini are absolutely terrible at these types of things. They are constantly wrong. I know I'm not saying anything new, but I'm waiting to personally experience an LLM that does something useful with any of the code I give it. These tools aren't useless. They're great as search engines and pointing me in the right direction. They write dumb bash scripts that save me time here and there. That's it. And it's hilarious to me how these people present these tools. It generates a bunch of code, and then you spend all your time auditing and fixing what is expected to be wrong. That's not the type of code I'm putting in my company's code base, and I could probably write the damn code more correctly in less time than it takes to review for expected errors. What am I missing?
- icapybara 1y ago
  It’s probably what you’re asking. You can’t just say “write me an app”, you have to break a big problem into small problems for it.
- spariev 1y ago
  I think it all depends on your platform and use cases. In my experience AI tools work best with Python and JS/Typescript and some simple use cases (web apps, basic data science etc). Also, I've found they can be of great help with refactorings and cases when you need to do something similar to already existing code, but with a twist or change.
- volkk 1y ago
  you might be missing small things to create more guardrails like effective prompting and maintaining what's been done using files, carefully controlling context, committing often in-between changes, but largely, you're not missing anything. i use AI constantly, but always for subtasks of a larger complicated thing that my brain has thought through. and often use higher cost models to help me abstractly think through complex things/point me in the right directions. personally, i've always operated in a codebase in a way that i _need_ to understand how things work for me to be productive and make the right decisions. I operate the same way with AI. every change is carefully reviewed, if it's dumb, i make it redo it and explain why it's dumb. and if it gets caught in a loop, i reset the context and try to reframe the problem. overall, i'm definitely more productive, but if you truly want to be hands off--you're in for a very bad time. i've been there. lastly, some codebases don't work well with AI. I was working on a problem that was a bit more novel/out there and no model could solve it. Just yapped endlessly about these complex, very potentially smart sounding solutions that did absolutely nothing. went all the way to o1-pro. the craziest part to me was the fact that across claude, deepseek and openai, they used the same specific vernacular for this particular problem which really highlights how a lot of these models are just a mish-mash of the same underlying architecture/internet data. some of these models use responses from other models for their training data, which to me is like incest. you won't get good genetical results
energy123 1y ago
Where can I read OpenAI's promise that it won't use the repos I upload for training?
- deleted 1y ago
  [deleted]
alvis 1y ago
Is it surprising? Hmm perhaps nope. But is it better than cursor etc? Hmm perhaps it’s a wrong question. Feels like codex is for product managers to fix bugs without touching any developer resources. Then it’s insanely surprising!
- gbalduzzi 1y ago
  It sounds nice, but are product managers able to spot regressions or other potential issues (performance, data protection, legal, etc) in the codex result?
  - alvis 1y ago
    If codex can analyze the whole code base, I can’t see why not? I can even imagine one can set up a CI task that any committed code must pass all sort of legal/data protection requirements too
    - kenjackson 1y ago
      Exactly this. In fact the product manager should be the one that knows what the set of checks that need to be done over the code base. You need a dev though to do make sure the last mile is doing what you expect it to do.
- bhl 1y ago
  I've been contracting with a startup. The bottleneck is not the lack of tools; it's agency. There's so much work, it becomes work to assign and organize work. But now who's going to do that work? Still engineers.
deleted 1y ago
[deleted]
ilaksh 1y ago
As someone who works on his own open source agent framework/UI (https://github.com/runvnc/mindroot https://github.com/runvnc/mindroot), it's kind of interesting how announcements from vendors tend to mirror features that I am working on. For example, in the last month or so, I added a job queue plugin. The ability to run multiple tasks that they demoed today is quite similar. The issue I ran into with users is that without Enterprise plans, complex tasks run into rate limits when trying to run concurrently. So I am adding an ability to have multiple queues, with each possibly using different models and/or providers, to get around rate limits. By the way, my system has features that are somewhat similar not only to this tool they are showing but also things like Manus. It is quite rough around the edges though because I am doing 100% of it myself. But it is MIT Licensed and it would be great if any developer on the planet wanted to contribute anything.
- deleted 1y ago
  [deleted]
asadm 1y ago
Is there an open source version of this? that essentially uses microvms to git clone my repo and essentially run codex-cli or equivalent and sends me a PR. I made one for github action but it's not as realtime and is 2 years old now: https://github.com/asadm/chota https://github.com/asadm/chota
- illnewsthat 1y ago
  I haven't checked in on it recently, but maybe a similar open-source option would be https://github.com/All-Hands-AI/OpenHands https://github.com/All-Hands-AI/OpenHands A not open-source option this looks close to is also https://githubnext.com/projects/copilot-workspace https://githubnext.com/projects/copilot-workspace (released April 2024, but I'm not sure it's gotten any significant updates since)
  - asadm 1y ago
    oh openDevin became openHANDS. Interestingly, I committed the LICENSE file to that repo haha
    - tough 1y ago
      did they relicense too w the rename?
      - asadm 1y ago
        no, original MIT still remains yay.
simianwords 1y ago
I wonder if tools like these are best for semi structured refactors like upgrade to python3, migrate to postgres etc
btbuildem 1y ago
> To balance safety and utility, Codex was trained to identify and precisely refuse requests aimed at development of malicious software, while clearly distinguishing and supporting legitimate tasks. I can't say I am a big fan of neutering these paradigm-shifting tools according to one culture's code of ethics / way of doing business / etc. One man's revolutionary is another's enemy combatant and all that. What if we need top-notch malware to take down the robot dogs lobbing mortars at our madmaxian compound?!
- amarcheschi 1y ago
  If I had to guess, only for the general public they'll be neutered, not for the 3 letters agencies
  - pixl97 1y ago
    TLA's have very few of their own coders, they contract everything out. Now I'm sure OAI will lend an unrestricted model to groups that pay large private contracts they won't disclose.
- lumenwrites 1y ago
  You gotta think about it in terms of cost vs benefit. How much damage will a malicious AI do, vs how much value will you get out of non-neutered model?
  - deleted 1y ago
    [deleted]
- deleted 1y ago
  [deleted]
- GolfPopper 1y ago
  >What if we need top-notch malware to take down the robot dogs lobbing mortars at our madmaxian compound?! I wouldn't sweat it. According to it's developers, Codex understands 'malicious software', it has just been trained to say, "But I won't do that" when such requests are made to it. Judging from the recent past [1][2] getting LLMs to bypass such safeguards is pretty easy. 1.https://hiddenlayer.com/innovation-hub/novel-universal-bypass-for-all-major-llms/ https://hiddenlayer.com/innovation-hub/novel-universal-bypas... 2.https://cyberpress.org/researchers-bypass-safeguards-in-17-popular-llm-models/?amp=1 https://cyberpress.org/researchers-bypass-safeguards-in-17-p...
scudsworth 1y ago
pleased to see a paragraph-long comment in the examples. now thats good coding.
- 2OEH8eoCRo0 1y ago
  More generated slop for a real human to sift through. Can I get an ai summary of that comment?
kleiba 1y ago
Just curious: is your company happy sharing their code-base with an AI provider? Or are you using a local installation?
- asadm 1y ago
  why not? OpenAI won't be stupid to look at my code and be that vulnerable legally. It ain't worth it.
  - KaiserPro 1y ago
    They literally scraped half of youtube, made a library to extract the audio and released it as whisper. Of _course_ they are training on your shit.
    - asadm 1y ago
      thats publicly accessible shit. my code is trade secret and IP. I would litigate that shit if a line I wrote ends up in public model, easiest money to be made.
      - frontalier 1y ago
        you will litigate nothing, and if you do you'll be dismissed anyway. https://en.wikipedia.org/wiki/Shira_Perlmutter#Firing https://en.wikipedia.org/wiki/Shira_Perlmutter#Firing
      - KaiserPro 1y ago
        > my code is trade secret and IP Which is the same law that protects "publicly accessible shit". In the same way that you can't legally just rip MP3s off the internet[1], or copy movies and distribute them. [1] unless you're rich.
- bhl 1y ago
  Cursor has enterprise mode which forces a data privacy feature.
tough 1y ago
so i just upgraded to pro plan but yet https://chatgpt.com/codex https://chatgpt.com/codex doesnt work for me and asks me to -try chatgpt pro- and shows me the upsell modal, even if already on the higher tier sigh
- modeless 1y ago
  You mean Pro? It's only in the $200 Pro tier.
  - tough 1y ago
    Yes sorry meant pro, I just enabled on Settings > Connectors > Github hoping that makes it work ... still doesnt work, is it geo-restricted maybe? idk
- rapfaria 1y ago
  They said Plus soon, not today.
- jiocrag 1y ago
  same here. Paying for Pro ($200) but the "try it" link just leads to the Pro sign up page, where it says I'm already on Pro. Hyper intelligent coding agents, but can't make their website work.
  - tough 1y ago
    > Hyper intelligent coding agents, but can't make their website work. I know right also no human to contact on support... tempted to cancel the sub lol i'll give them 24h
- fear91 1y ago
  Same here, paying for Pro but I just get redirected to vanilla version...
  - piskov 1y ago
    > will be rolling ≠ available now to all pro users
alvis 1y ago
I used to work for a bank and the legal team used to ping us to make tiny changes to the app for compliance related issues. Now they can fix themselves. I think they’d be very proud and happy
- ajkjk 1y ago
  Hopefully nobody lets legal touch anything without the ability to run the code to test it, plus code reviews. So probably not.
  - eru 1y ago
    I'm not sure what you are on about? You can make arbitrary teams, like legal, make PRs. You would still have the proper owners of the project agree whether they take the PRs. Either by human review and/or by any other review process they set up.
    - ajkjk 1y ago
      I am on about the fact that you are imagining something working in a way that it does not work in practice. You have two choices: 1. A developer makes a PR. They build the app and run it themselves to make sure it works, does what they intended, and nothing unexpected happens, and report on their testing in the PR. There's also a suite of automated tests. Between these you are confident that you can rubber-stamp it without touching the change yourself. But this requires them being able to run the code and intelligently think about it themselves, which AI cannot do. 2. A non-developer uses an LLM to make a PR. The change passes the tests, but you have no other validation about it because it was done by a bot that can't think about how the app actually works. As reviewer you have to pull down the change to run and validate it yourself. Now you are doing the same amount of work as before, except that instead of you telling the LLM to make the change, someone else did. The only difference is maybe that the activation energy for doing the work was avoided. Which, fine, is non-negligible. But let's not pretend like in the latter case legal "made the change". They did the 1% upfront work of asking the LLM to find where the change goes, and left you the other 99% of actually shepherding it through. The only changes that will work for at all are, like, updating copy/strings/icons---and honestly, at my last job at least, we already let legal (and product, etc) do stuff like that. I suppose the LLM might save them having to figure out how to use Git, at least. You might imagine there's some software out there where the automated tests are so thorough that you can trust a code change to not break anything if it passes the tests. I have personally never seen such a thing. And in practice many tests validate against strings and other small feature-level details, meaning that the kinds of code changes that other orgs are making are going to be touching the tests as well, so human verification is still required.
skovati 1y ago
I'm curious how many ICs are truly excited about these advancements in coding agents. It seems to me the general trend is we become more like PMs managing agents and reviewing PRs, all for the sake of productivity gains. I imagine many engineers are like myself in that they got into programming because they liked tinkering and hacking and implementation details, all of which are likely to be abstracted over in this new era of prompting.
- awestroke 1y ago
  At the end of the day, it's your job to deliver value. If a tool allows you to deliver more faster, without sacrificing quality, it's your responsibility to use that tool. You'll just have to make sure you can fully take responsibility for the end deliverables. And these tools are not only useful for writing the final code
  - enjoylife 1y ago
    > these tools are not only useful for writing the final code This sparked a thought in how a large part of the job is often the work needed to demonstrate impact. I think this aspect is often overlooked by some of the good engineers not yet taking advantage of the AI tooling. LLM loops may not yet be good enough to produce shippable code by themselves, but they sure are capable to help reduce the overhead of these up and out communicative tasks.
    - tough 1y ago
      you mean like hacking a first POC with AI to sell a product/feature internally to get buy-in from the rest of the team before actually shipping production version of it?
  - whyowhy3484939 1y ago
    It's actually not. My job description does not say "deliver value" and nobody talks about my work like that so I'm not quite sure what to make of that. > without sacrificing quality Right.. > it's your responsibility to use that tool Again, it's actually not. It's my responsibility to do my job, not to make my boss' - or his boss' - car nicer. I know that's what we all know will create "job security" but let's not conflate these things. My job is to do my end of the bargain. My boss' job is paying me for doing that. If he deems it necessary to force me to use AI bullshit, I will of course, but it is definitely not my responsibility to do so autonomously.
simianwords 1y ago
Does any one how the quality drops with size of codebase?
yanis_t 1y ago
So it's looking like it's only running in the cloud, that is it will push commits to my remote repo before I have a chance to see if it works? When I'm using aider, after it make a commit what I do, I then immediately run git reset HEAD^ and then git diff (actually I use github desktop client to see the diff) to evaluate what exactly it did, and if I like it or not. Then I usually make some adjustments and only after that commit and push.
- flakiness 1y ago
  You can think of this as a managed (cloud) version of their codex command line tool, which runs locally on your laptop. The secret sauce here seems like their new model, but I expect it to come to API at some point.
- codemac 1y ago
  watch the live stream, it shows you the diff as the completed task, you decide whether or not to generate a github pr when you see the diff.
- danielbln 1y ago
  You may want to pass --no-auto-commits to Aider if you peel them off HEAD afterwards anyway.
- TechDebtDevin 1y ago
  This takes all the soul out of programming, not sure why anyone likes to use llms like this.
  - sumedh 1y ago
    Because from a business point of view, they just want a feature to be built, they really care about the code if it works.
- sagarpatil 1y ago
  It generates a PR. You decide if you want to merge, clone it locally, test, edit and then merge the code.
adamTensor 1y ago
not buying windsurf then???
- motoxpro 1y ago
  This would be the why of that acquisition as this needs a more integrated UI. Guessing by the speed at which this came out, this was in the works long before that acquisition.
  - adamTensor 1y ago
    it is not even clear *if* they are going to buy windsurf at all. And thats a big if. This might've just been the 'why' that deal is not happening.
    - shmoogy 1y ago
      This probably came out to beat Google I/O or something similar - odd Friday release otherwise.
ianbutler 1y ago
Im super curious to see how this actually does at finding significant bugs, we've been working in the space on https://www.bismuth.sh https://www.bismuth.sh for a while and one of the things we're focused on is deep validation of the code being outputted. There's so many of these "vibe coding" tools and there has to be real engineering rigor at some point. I saw them demo "find the bug" but the bugs they found were pretty superficial and thats something we've seen in our internal benchmark from both Devin and Cursor. A lot of noise and false positives or superficial fixes.
- deleted 1y ago
  [deleted]
orliesaurus 1y ago
Why hasn't Github released this? Why it's OpenAI releasing this?!
- adpirz 1y ago
  It's on their roadmap: https://github.blog/news-insights/product-news/github-copilot-the-agent-awakens/#project-padawan-swe-agents-on-github https://github.blog/news-insights/product-news/github-copilo... But they aren't moving nearly as fast as OpenAI. And it remains to be seen if first mover will mean anything.
- taytus 1y ago
  Github moves too slow, and OpenAI moves too fast.
- danielbln 1y ago
  GitHub has released this, it's called Copilot Agent.
johnjwang 1y ago
Some engineers on my team at Assembled and I have been a part of the alpha test of Codex, and I'll say it's been quite impressive. We’ve long used local agents like Cursor and Claude Code, so we didn’t expect too much. But Codex shines in a few areas: Parallel task execution: You can batch dozens of small edits (refactors, tests, boilerplate) and run them concurrently without context juggling. It's super nice to run a bunch of tasks at the same time (something that's really hard to do in Cursor, Cline, etc.) It kind of feels like a junior engineer on steroids, you just need to point it at a file or function, specify the change, and it scaffolds out most of a PR. You still need to do a lot of work to get it production ready, but it's as if you have an infinite number of junior engineers at your disposal now all working on different things. Model quality is good, but hard to say it's that much better than other models. In side-by-side tests with Cursor + Gemini 2.5-pro, naming, style and logic are relatively indistinguishable, so quality meets our bar but doesn’t yet exceed it.
- fourside 1y ago
  > You still need to do a lot of work to get it production ready, but it's as if you have an infinite number of junior engineers at your disposal now all working on different things. One issue with junior devs is that because they’re not fully autonomous, you have to spend a non trivial amount of time guiding them and reviewing their code. Even if I had easy access to a lot of them, pretty quickly that overhead would become the bottleneck. Did you think that managing a lot of these virtual devs could get overwhelming or are they pretty autonomous?
  - fabrice_d 1y ago
    They wrote "You still need to do a lot of work to get it production ready". So I would say it's not much better than real colleagues. Especially since junior devs will improve to a point they don't need your hand holding (remember you also were a junior at some point), which is not proven will happen with AI tools.
    - bmcahren 1y ago
      Counter-point A: AI coding assistance tools are rapidly advancing at a clip that is inarguably faster than humans. Counter-point B: AI does not get tired, does not need space, does not need catering to their experience. AI is fine being interrupted and redirected. AI is fine spending two days on something that gets overwritten and thrown away (no morale loss).
tough 1y ago
can someone give me a test prompt to one-shot something in go for testing? (Im trying something) what would be an impressive program that an agent should be able to one-shot in one go?
blixt 1y ago
They mentioned "microVM" in the live stream. Notably there's no browser or internet access. It makes sense, running specialized Firecracker/Unikraft/etc microkernels is way faster and cheaper so you can scale it up. But there will be a big technical scalability difficulty jump from this to the "agents with their own computers". ChatGPT Operator already does have a browser, so they definitely can do this, but I imagine the demand is orders of magnitudes different. There must be room for a Modal/Cloudflare/etc infrastructure company that focuses only on providing full-fledged computer environments specifically for AI with forking/snapshotting (pause/resume), screen access, human-in-the-loop support, and so forth, and it would be very lucrative. We have browser-use, etc, but they don't (yet) capture the whole flow.
- ushakov 1y ago
  we offer this with E2B Desktop Demo: https://surf.e2b.dev https://surf.e2b.dev SDK: https://github.com/e2b-dev/desktop https://github.com/e2b-dev/desktop
- thundergolfer 1y ago
  It's not our only focus at Modal but it's a big focus![1] Code agents are the killer use case for LLMs right now, and this complements our GPU inference and training capabilities. I'm quietly betting that agents increase the leverage of deterministic, reproducible devbox tech (eg. Nix, lockfiles, package mirroring), and this will end up being a huge win for us human engineers too. 1. https://modal.com/use-cases/sandboxes https://modal.com/use-cases/sandboxes
sudohalt 1y ago
When it runs the code I assume it does so via a docker container, does anyone know how it is configured? Assuming the user hasn't specified an AGENTS.md file or a Dockerfile in the repo. Does it generate it via LLM based on the repo, and what it thinks is needed? Does it use static analysis (package.json, requirements txt, etc)? Do they just have a super generic Dockerfile that can handle most envs? Combination of different things?
- ilaksh 1y ago
  I think they mentioned it was a similar environment to what it trains on, so maybe they have a default Dockerfile. Of course containers can also install additional packages or at least python packages.
  - nkko 1y ago
    Yes, and one test failed as it missed pydantic dependency
- hansonw 1y ago
  More about that here! https://platform.openai.com/docs/codex#advanced-configuration https://platform.openai.com/docs/codex#advanced-configuratio...
  - sudohalt 1y ago
    Thanks!
  - sudohalt 1y ago
    It seems LLMs are doing a lot of the heavy lifting figuring out the exact test, build, lint commands to run (even if the AGENTS.md file gives it direction and hints). I wonder if there are any plans to support user defined build, test, and pre commit commands to avoid unnecessary cost and keep it deterministic. Also wonder how monolith repos (or distinct but related repos) are supported, does it run everything in one container or loop through the envs that are edited? I assume one easy next step is to just run GitHub Actions in the container since everything is defined there (assuming the user set it up)
bionhoward 1y ago
What about privacy, training opt out? What about using it for AI / developing models that compete with our new overlords? Seems like using this is just asking to get rug pulled for competing with em when they release something that competes with your thing. Am I just an old who’s crowing about nothing? It’s ok for them to tell us we own outputs we can’t use to compete with em?
- piskov 1y ago
  What the video: there is an explicit switch at one of the steps about (not) allowing to train on your repo.
  - lurking_swe 1y ago
    That’s nice. And we trust that it does what it says because…? The AI company (openai, anthropic, etc) pinky promised? Have we seen their source code? How do you know they don’t train? Facebook has been caught in recent DOJ hearings breaking the law with how they run their business, just as one example. They claimed under oath, previously, to not be doing X, and then years later there was proof they did exactly that. https://youtu.be/7ZzxxLqWKOE?si=_FD2gikJkSH1V96r https://youtu.be/7ZzxxLqWKOE?si=_FD2gikJkSH1V96r A companies “word” means nothing imo. None of this makes sense if i’m being honest. Unless you personally have a negotiated contract with the provider, and can somehow be certain they are doing what they claim, and can later sue for damages, all of this is just crossing your fingers and hoping for the best.
    - tough 1y ago
      On the other hand you can enable explicit sharing of your data and get a few million free tokens daily
    - wilg 1y ago
      If you don't trust the company your opt-out strategy is much easier, you simply do not authorize them to access your code.
    - deleted 1y ago
ofirpress 1y ago
[I'm one of the co-creators of SWE-bench] The team managed to improve on the already very strong o3 results on SWE-bench, but it's interesting that we're just seeing an improvement of a few percentage points. I wonder if getting to 85% from 75% on Verified is going to take as long as it took to get from 20% to 75%.
- Snuggly73 1y ago
  I can be completely off base, but it feels to me like benchmaxxing is going on with swe-bench. Look at the results from multi swe bench - https://multi-swe-bench.github.io/#/ https://multi-swe-bench.github.io/#/ swe polybench - https://amazon-science.github.io/SWE-PolyBench/ https://amazon-science.github.io/SWE-PolyBench/ Kotlin bench - https://firebender.com/leaderboard https://firebender.com/leaderboard
  - Bjorkbat 1y ago
    I kind of had the feeling LLMs would be better at Python vs other languages, but wow, the difference on Multi SWE is pretty crazy.
  - ofirpress 1y ago
    Not sure what you mean by benchmaxxing but we think there's still a lot of useful signals you can infer from SWE-bench-style benchmarking. We also have SWE-bench Multimodal which adds a twist I haven't seen elsewhere: https://www.swebench.com/multimodal.html https://www.swebench.com/multimodal.html
    - Snuggly73 1y ago
      I mean that there is the possibility that swe bench is being specifically targeted for training and the results may not reflect real world performance.
  - kristianp 1y ago
    Maybe a lot of the difference we see between peoples comments about how useful AI is for their coding, is a function of what language they're using. Python coders may love it, Go coders not much at all.
nadis 1y ago
In the preview video, I appreciated Katy Shi's comment on "I think this is a reflection of where engineering work has moved over the past where a lot of my time now is spent reviewing code rather than writing it." Preview video from Open AI: https://www.youtube.com/watch?v=hhdpnbfH6NU&t=878s https://www.youtube.com/watch?v=hhdpnbfH6NU&t=878s As I think about what "AI-native" or just the future of building software loos like, its interesting to me that - right now - developers are still just reading code and tests rather than looking at simulations. While a new(ish) concept for software development, simulations could provide a wider range of outcomes and, especially for the front end, are far easier to evaluate than just code/tests alone. I'm biased because this is something I've been exploring but it really hit me over the head looking at the Codex launch materials.
- ai-christianson 1y ago
  > rather than looking at simulations You mean like automated test suites?
  - tough 1y ago
    automated visual fuzzy-testing with some self-reinforcement loops There's already library's for QA testing and VLM's can give critique on a series of screenshots automated by a playwright script per branch
    - ai-christianson 1y ago
      Cool. Putting vision in the loop is a great idea. Ambitious idea, but I like it.
      - tough 1y ago
        SmolVLM, Gemma, LlaVa, in case you wanna play with some of the ones i've tried. https://huggingface.co/blog/smolvlm https://huggingface.co/blog/smolvlm recently both llama.cpp and ollama got better support for them too, which makes this kind of integration with local/self-hosted models now more attainable/less expensive
DGAP 1y ago
If you still don't think software engineering as a high paying job is over, I don't know what to tell you.
- whyowhy3484939 1y ago
  It's high paying?
- Reubend 1y ago
  It might be over in the future, but it's still fine for the next few years at the minimum.
- dreeves 1y ago
  My counterargument: https://agifriday.substack.com/p/jevons https://agifriday.substack.com/p/jevons Short version: Jevons's paradox means that the more coding you automate away for developers (like with compilers in the past), the more in-demand those developers are. (Until AGI when all bets are off, of course.)
asdev 1y ago
is the point of this to actually assign tasks to an AI to complete end to end? Every task I do with AI requires atleast some bit of hand holding, sometimes reprompting etc. So I don't see why I would want to run tasks in parallel, I don't think it would increase throughput. Curious if others have better experiences with this
- nmca 1y ago
  with a bad ai it is pointless, with a good ai it is powerful. codex-1 has been quite good in my experience
- masterj 1y ago
  The example use-cases in the videos are pretty compelling and much smaller scope. “Here’s an error reported to the oncall. Give a try fixing it” (Could be useful even if it fails) Refactor this small piece I noticed while doing something else. Small-scoped stuff that likely wouldn’t get done otherwise. I wouldn’t ask LLMs for full-features in a real codebase but these examples seem within the scope of what they might be able to accomplish end-to-end
- sagarpatil 1y ago
  I am working with a 3rd party API (Exa.ai) and I hacked together a python script. I ran a remote agent to do these tasks simultaneously (augment.new, I’m not affiliated, I have early access) Agent 1: write tests, make sure all the tests pass. Agent 2: concert python script to fastapi Agent 3: create frontend based on fastapi endpoints I get a PR, I check code and see if it works and then merge to main. All three PR’s worked flawlessly (front end wasn’t pretty).
RhysabOweyn 1y ago
I believe that code from one of these things will eventually cause a disaster affecting the capital owners. Then all of a sudden you will need a PE license, ABET degree, 5 years working experience, etc. to call yourself a software engineer. It would not even be historically unique. Charlatans are the reason that lawyers, medical doctors, and civil engineers have to go through lots of education, exams, and vocational training to get into their profession. AI will probably force software engineering as a profession into that category as well. On the other hand, if your job was writing code at certain companies whose profits were based on shoving ads in front of people then I would agree that no one will care if it is written by a machine or not. The days of those jobs making >$200k a year are numbered.
- deleted 1y ago
  [deleted]
- alfalfasprout 1y ago
  Even ads have risk. Customer service has risk. The widespread proliferation of this stuff is a legal minefield waiting to be stepped on.
- lispisok 1y ago
  I wish but I dont think we could be any futher away from professionalizing like engineering/law/accounting/medicine. There was a deliberate effort to flood the field and lower salaries and developers were so full of hubris and thought there was infinite demand for their labor and went along with it and still are. Maybe some are learning given the job market the last few years. Despite software being in everything and harm to the public due to bad software has materialized every developer seems vehemently against professionalizing. Do you want a surgeon that went to surgeon bootcamp because "you dont need all those years in medical school to learn how to remove an appendix"? Do you even want an accountant who went to accountant bootcamp to do your taxes?
  - RhysabOweyn 1y ago
    Obviously there is no way to really predict when this would happen, but I don't think it will be up to developers to decide whether it happens or not. In Texas for example, the legislature forced engineering to be professionalized (or regulated) in an emergency session after a school in a well off area exploded in a gas explosion (https://en.wikipedia.org/wiki/New_London_School_explosion#Investigation_and_legislation https://en.wikipedia.org/wiki/New_London_School_explosion#In...). I also do not think this is limited to software engineering. Medical doctors and accountants have faced the squeeze in recent years too. There are tons of (bad) DO med schools opening up across the country that will be flooding the field before long, nurse practitioners and physicians assistants get to do more and more work that only doctors got to do, and more and more accounting is being offshored. The question is when things get so bad that even the powerful decide to actually do something about it.
SketchySeaBeast 1y ago
Is this the same idea as when we switched to multicore machines? The rate of change on the capabilities of a single agent has slowed enough now the only way for OpenAI to appearing to be making decent progress is to have many?
ionwake 1y ago
Im sorry if Im being silly, but I have paid for the Pro version, $200 a month, everytime I click on Try Codex, it takes me to a pricing page with the "Team Plan" https://chatgpt.com/codex#pricing https://chatgpt.com/codex#pricing. Is this still rolling out? I dont need the team plan too do I? I have been using openAI products for years now and I am keen to try but I have no idea what I am doing wrong.
- mr_north_london 1y ago
  It's still rolling out
  - ionwake 1y ago
    Thx for the reply, Im in london too ( atm )
- jdee 1y ago
  im the same, and it appeared for me 2 mins ago. looks like its still rolling out
  - ionwake 1y ago
    cool it appeared - I wa sjsut worried it was a payment issue. thanks guys.
- throwaway314155 1y ago
  They do this with every major release. Never going to understand why.
hintymad 1y ago
I remember HN had a repeating popular post on the the most important data structures. They are all the basic ones that a first-year college student can learn. The youngest one was skiplist, which was invented in 1990. When I was a student, my class literally read the original paper and implemented the data structure and analyzed the complexity in our first data structure course. This seems imply that the software engineering as a profession has been quite mature and saturated for a while, to the point that a model can predict most of the output. Yes, yes, I know there are thousands of advanced algorithms and amazing systems in production. It's just that the market does not need millions of engineers for such advanced skills. Unless we get yet another new domain like cloud or like internet, I'm afraid the core value of software engineers: trailblazing for new business scenarios, will continue diminishing and being marginalized by AI. As a result, we get way less demand for our job, and many of us will either take a lower pay, or lose our jobs for extended time.
deleted 1y ago
[deleted]
theappsecguy 1y ago
I am so damn tired of all the AI garbage shoved down our throats every day. Can't wait for all of it to crash and burn.
- jetsetk 1y ago
  At some point the bubble will pop
  - boshalfoshal 1y ago
    Higher chance of AGI/most SWEs being out of a job for an extended period of time than the bubble popping imo.
    - missedthecue 1y ago
      Agree with this. It's so strange to me that out of the all the anti-AI arguments one can derive, the anti-AIers have settled on riding with the "it's useless" argument, when it's plainly one of the most useful tools created since the desktop computer.
      - Collisteru 1y ago
        These people remind me of the people in the late 19th century who were scared of electricity because they thought it could remove all the air from rooms and suffocate people.
    - greatpostman 1y ago
      Way higher chance
fullstackchris 1y ago
Reading these threads its clear to me people are so cooked and no longer understand (or perhaps never did) understand the simple process of how source code is shared, built, and merged together with multiple editors has ever worked
swisniewski 1y ago
Has anyone else been able to get "secrets" to work? They seem to be injected fine in the "environment setup" but don't seem to be injected when running tasks against the enviornment. This consistently repros even if I delete and re-create the enviornment and archive and resubmit the task.
bmcahren 1y ago
So there's this thing called "Setup Scripts" but they don't explicitly say these are equivalent to AWS Metadata and configured inside of Codex web interface - not a setup.sh or a package.json preinstall declaration. I wasted several hours (and lots of compute where Codex was as confused as I was) trying to figure out how to convince codex to pnpm install.
- gpeal 1y ago
  Codex engineer here. Can you elaborate on what was confusing? I would love to make it more clear.
  - rahulyc 1y ago
    I didn't realize the setup script had to be done in the UI over in the environment tab. I assumed it would be reading something like setup.sh from the codebase. Docs could make that more clear
  - nater5000 1y ago
    I just found this thread while trying to figure out where to set the setup script. The docs should probably make that more clear.
CSMastermind 1y ago
I really wish they'd add support for Gitlab or better any arbitrary git repo.
- eru 1y ago
  Well, Microsoft bought GitHub and Microsoft sort-of co-owns OpenAI, too.
ZeroCool2u 1y ago
"23 SWE-Bench Verified samples that were not runnable on our internal infrastructure were excluded." What does that mean? Surely this should have a bit more elaboration. If you're just excluding a double digit number of tasks in the benchmark as uncompleted, that should be reflected in the scores.
solresol 1y ago
I'm not sure what's wrong with me, but I just wasted several hours wrestling codex to make it behave. Here's my workflow that keeps failing: - it writes some code. It looks good a first glance - I push it to github - automated tests on github show that there's a problem - go back to codex and ask it to fix it - it does stuff. It looks good again. Now what do I do? If I ask it to push again to github, then it will often create a pull request that doesn't include stuff from the first pull request, but it's not a pull request that stacks on top of the previous pull request, it's a pull request that stacks on top of main. When asked to write something that called out to gpt-4.1-mini, it used openai.ChatCompletion.create (!?!!?) I just found myself using claude to fix codex's mistakes.
- landl0rd 1y ago
  [flagged]
- csomar 1y ago
  I think you need to run the tests locally before you push the PR. I actually think you need to (somehow?) make this part of the generation process before Codex proposes the changes.
- bargainbin 1y ago
  I have this issue with Devin. Given my limited knowledge of how these work, I believe there is simply too much context for it to take a holistic view of the task and finish accordingly. If both OpenAI and Devin are falling into the same pattern then that’s a good indication there’s a fundamental problem to be solved here.
- fcoury 1y ago
  I upgraded to Pro just because of Codex and I am really not impressed. Granted, I am using rust so that may be the issue (or skill issue on my end too). One of the things I am constantly struggling with is that the containers they use are having issues to fetch anything from the internet: error: failed to get `anyhow` as a dependency of package `yawl-core v0.1.0 (/wor kspace/yawl/core)` Caused by: download of config.json failed Caused by: failed to download from `https://index.crates.io/config.json` Caused by: [7] Could not connect to server (Failed to connect to proxy port 8080 after 30 65 ms: Could not connect to server) Hopefully they fix this and it gets better with time, but I am not going to renew past this month otherwise.
bearjaws 1y ago
I spend maybe $20 a month on Claude code, $200 a month is hard to swallow. I don't understand why OAI puts their alpha release products under a $200 a month plan instead of just charging for tokens.
- Bjorkbat 1y ago
  My weird theory: the alpha releases are more expensive than people realize, and OpenAI can afford to launch money-losing alphas on Pro because few users have Pro accounts and the lose less money per Pro user. If anyone could use Codex, you’d want to price the tokens at a rate where you don’t lose money, or at least not too much money. If you price the tokens too high, then not only are people not using Codex, but customers and investors alike begin to lose faith, they no longer “feel the AGI”.
zrg 1y ago
I've been experimenting with providers offering similar functionality for the last year and it is really vastly superior experience this codex like approach than cursor, devin etc

511 points by meetpateltech 1y ago

deleted 1y ago

[deleted]

(watching live) I'm wondering how it performs on the METR benchmark (https://metr.org/blog/2025-03-19-measuring-ai-ability-to-complete-long-tasks/ https://metr.org/blog/2025-03-19-measuring-ai-ability-to-com...).

deleted 1y ago

[deleted]

deleted 1y ago
[deleted]

colesantiago 1y ago

I think the benchmark test for these programming agents that I would like to see an Agent making a flawless PR or patch to the BSD / Linux kernel. This should be possible today and surely Linus would also see this in the future.

_kb 1y ago
There’s a fairly pragmatic discussion in that exact topic with Linus here: https://youtu.be/VHHT6W-N0ak https://youtu.be/VHHT6W-N0ak.

deleted 1y ago

[deleted]

deleted 1y ago
[deleted]

_jayhack_ 1y ago

[flagged]

esafak 1y ago
An AI not for the IDE... for PMs?

tptacek 1y ago

Maddening: "codex" is also the name of their open-source Claude-Code-alike, and was previously the name of an at-the-time frontier coding model. It's like they name things just to fuck with us.

tekacs 1y ago
So -- that client-side thing is _technically_ called `codex-cli` (in the parent 'codex' repo, which looks like a monorepo?). Still super confusing, though! I feel like companies working with and shipping LLMs would do well to remember that it's not just humans who get confused by this, but LLMs themselves... it makes for a painful time, sending off a request and noting that a third of the way into its reasoning that the model has gotten tow things with almost-identical names confused.
- tough 1y ago
  they also have a dual implementation on rust and typescript there's codex-rs in that monorepo
  - fabmilo 1y ago
    more excited about the rust impl than the typescript one.
    - tptacek 1y ago
      Besides packaging of their releases, what possible difference could that make in this problem domain?
      - tough 1y ago
        I just think it's nice to have open source code to reference so maybe he meant just in that -educational- way, certainly more to learn from the rust one than the TS one for most folks? even if the problem-space doesn't require system-level safety code indeed

bilal4hmed 1y ago

[flagged]

deleted 1y ago
[deleted]

prhn 1y ago

Is anyone using any of these tools to write non boilerplate code? I'm very interested. In my experience ChatGPT and Gemini are absolutely terrible at these types of things. They are constantly wrong. I know I'm not saying anything new, but I'm waiting to personally experience an LLM that does something useful with any of the code I give it. These tools aren't useless. They're great as search engines and pointing me in the right direction. They write dumb bash scripts that save me time here and there. That's it. And it's hilarious to me how these people present these tools. It generates a bunch of code, and then you spend all your time auditing and fixing what is expected to be wrong. That's not the type of code I'm putting in my company's code base, and I could probably write the damn code more correctly in less time than it takes to review for expected errors. What am I missing?

icapybara 1y ago
It’s probably what you’re asking. You can’t just say “write me an app”, you have to break a big problem into small problems for it.
spariev 1y ago
I think it all depends on your platform and use cases. In my experience AI tools work best with Python and JS/Typescript and some simple use cases (web apps, basic data science etc). Also, I've found they can be of great help with refactorings and cases when you need to do something similar to already existing code, but with a twist or change.
volkk 1y ago
you might be missing small things to create more guardrails like effective prompting and maintaining what's been done using files, carefully controlling context, committing often in-between changes, but largely, you're not missing anything. i use AI constantly, but always for subtasks of a larger complicated thing that my brain has thought through. and often use higher cost models to help me abstractly think through complex things/point me in the right directions. personally, i've always operated in a codebase in a way that i _need_ to understand how things work for me to be productive and make the right decisions. I operate the same way with AI. every change is carefully reviewed, if it's dumb, i make it redo it and explain why it's dumb. and if it gets caught in a loop, i reset the context and try to reframe the problem. overall, i'm definitely more productive, but if you truly want to be hands off--you're in for a very bad time. i've been there. lastly, some codebases don't work well with AI. I was working on a problem that was a bit more novel/out there and no model could solve it. Just yapped endlessly about these complex, very potentially smart sounding solutions that did absolutely nothing. went all the way to o1-pro. the craziest part to me was the fact that across claude, deepseek and openai, they used the same specific vernacular for this particular problem which really highlights how a lot of these models are just a mish-mash of the same underlying architecture/internet data. some of these models use responses from other models for their training data, which to me is like incest. you won't get good genetical results

energy123 1y ago

Where can I read OpenAI's promise that it won't use the repos I upload for training?

deleted 1y ago
[deleted]

alvis 1y ago

Is it surprising? Hmm perhaps nope. But is it better than cursor etc? Hmm perhaps it’s a wrong question. Feels like codex is for product managers to fix bugs without touching any developer resources. Then it’s insanely surprising!

gbalduzzi 1y ago
It sounds nice, but are product managers able to spot regressions or other potential issues (performance, data protection, legal, etc) in the codex result?
- alvis 1y ago
  If codex can analyze the whole code base, I can’t see why not? I can even imagine one can set up a CI task that any committed code must pass all sort of legal/data protection requirements too
  - kenjackson 1y ago
    Exactly this. In fact the product manager should be the one that knows what the set of checks that need to be done over the code base. You need a dev though to do make sure the last mile is doing what you expect it to do.
bhl 1y ago
I've been contracting with a startup. The bottleneck is not the lack of tools; it's agency. There's so much work, it becomes work to assign and organize work. But now who's going to do that work? Still engineers.

deleted 1y ago

[deleted]

ilaksh 1y ago

As someone who works on his own open source agent framework/UI (https://github.com/runvnc/mindroot https://github.com/runvnc/mindroot), it's kind of interesting how announcements from vendors tend to mirror features that I am working on. For example, in the last month or so, I added a job queue plugin. The ability to run multiple tasks that they demoed today is quite similar. The issue I ran into with users is that without Enterprise plans, complex tasks run into rate limits when trying to run concurrently. So I am adding an ability to have multiple queues, with each possibly using different models and/or providers, to get around rate limits. By the way, my system has features that are somewhat similar not only to this tool they are showing but also things like Manus. It is quite rough around the edges though because I am doing 100% of it myself. But it is MIT Licensed and it would be great if any developer on the planet wanted to contribute anything.

deleted 1y ago
[deleted]

asadm 1y ago

Is there an open source version of this? that essentially uses microvms to git clone my repo and essentially run codex-cli or equivalent and sends me a PR. I made one for github action but it's not as realtime and is 2 years old now: https://github.com/asadm/chota https://github.com/asadm/chota

illnewsthat 1y ago
I haven't checked in on it recently, but maybe a similar open-source option would be https://github.com/All-Hands-AI/OpenHands https://github.com/All-Hands-AI/OpenHands A not open-source option this looks close to is also https://githubnext.com/projects/copilot-workspace https://githubnext.com/projects/copilot-workspace (released April 2024, but I'm not sure it's gotten any significant updates since)
- asadm 1y ago
  oh openDevin became openHANDS. Interestingly, I committed the LICENSE file to that repo haha
  - tough 1y ago
    did they relicense too w the rename?
    - asadm 1y ago
      no, original MIT still remains yay.

simianwords 1y ago

I wonder if tools like these are best for semi structured refactors like upgrade to python3, migrate to postgres etc

btbuildem 1y ago

> To balance safety and utility, Codex was trained to identify and precisely refuse requests aimed at development of malicious software, while clearly distinguishing and supporting legitimate tasks. I can't say I am a big fan of neutering these paradigm-shifting tools according to one culture's code of ethics / way of doing business / etc. One man's revolutionary is another's enemy combatant and all that. What if we need top-notch malware to take down the robot dogs lobbing mortars at our madmaxian compound?!

amarcheschi 1y ago
If I had to guess, only for the general public they'll be neutered, not for the 3 letters agencies
- pixl97 1y ago
  TLA's have very few of their own coders, they contract everything out. Now I'm sure OAI will lend an unrestricted model to groups that pay large private contracts they won't disclose.
lumenwrites 1y ago
You gotta think about it in terms of cost vs benefit. How much damage will a malicious AI do, vs how much value will you get out of non-neutered model?
- deleted 1y ago
  [deleted]
deleted 1y ago
[deleted]
GolfPopper 1y ago
>What if we need top-notch malware to take down the robot dogs lobbing mortars at our madmaxian compound?! I wouldn't sweat it. According to it's developers, Codex understands 'malicious software', it has just been trained to say, "But I won't do that" when such requests are made to it. Judging from the recent past [1][2] getting LLMs to bypass such safeguards is pretty easy. 1.https://hiddenlayer.com/innovation-hub/novel-universal-bypass-for-all-major-llms/ https://hiddenlayer.com/innovation-hub/novel-universal-bypas... 2.https://cyberpress.org/researchers-bypass-safeguards-in-17-popular-llm-models/?amp=1 https://cyberpress.org/researchers-bypass-safeguards-in-17-p...

scudsworth 1y ago

pleased to see a paragraph-long comment in the examples. now thats good coding.

2OEH8eoCRo0 1y ago
More generated slop for a real human to sift through. Can I get an ai summary of that comment?

kleiba 1y ago

Just curious: is your company happy sharing their code-base with an AI provider? Or are you using a local installation?

asadm 1y ago
why not? OpenAI won't be stupid to look at my code and be that vulnerable legally. It ain't worth it.
- KaiserPro 1y ago
  They literally scraped half of youtube, made a library to extract the audio and released it as whisper. Of _course_ they are training on your shit.
  - asadm 1y ago
    thats publicly accessible shit. my code is trade secret and IP. I would litigate that shit if a line I wrote ends up in public model, easiest money to be made.
    - frontalier 1y ago
      you will litigate nothing, and if you do you'll be dismissed anyway. https://en.wikipedia.org/wiki/Shira_Perlmutter#Firing https://en.wikipedia.org/wiki/Shira_Perlmutter#Firing
    - KaiserPro 1y ago
      > my code is trade secret and IP Which is the same law that protects "publicly accessible shit". In the same way that you can't legally just rip MP3s off the internet[1], or copy movies and distribute them. [1] unless you're rich.
bhl 1y ago
Cursor has enterprise mode which forces a data privacy feature.

tough 1y ago

so i just upgraded to pro plan but yet https://chatgpt.com/codex https://chatgpt.com/codex doesnt work for me and asks me to -try chatgpt pro- and shows me the upsell modal, even if already on the higher tier sigh

modeless 1y ago
You mean Pro? It's only in the $200 Pro tier.
- tough 1y ago
  Yes sorry meant pro, I just enabled on Settings > Connectors > Github hoping that makes it work ... still doesnt work, is it geo-restricted maybe? idk
rapfaria 1y ago
They said Plus soon, not today.
jiocrag 1y ago
same here. Paying for Pro ($200) but the "try it" link just leads to the Pro sign up page, where it says I'm already on Pro. Hyper intelligent coding agents, but can't make their website work.
- tough 1y ago
  > Hyper intelligent coding agents, but can't make their website work. I know right also no human to contact on support... tempted to cancel the sub lol i'll give them 24h
fear91 1y ago
Same here, paying for Pro but I just get redirected to vanilla version...
- piskov 1y ago
  > will be rolling ≠ available now to all pro users

alvis 1y ago

I used to work for a bank and the legal team used to ping us to make tiny changes to the app for compliance related issues. Now they can fix themselves. I think they’d be very proud and happy

ajkjk 1y ago
Hopefully nobody lets legal touch anything without the ability to run the code to test it, plus code reviews. So probably not.
- eru 1y ago
  I'm not sure what you are on about? You can make arbitrary teams, like legal, make PRs. You would still have the proper owners of the project agree whether they take the PRs. Either by human review and/or by any other review process they set up.
  - ajkjk 1y ago
    I am on about the fact that you are imagining something working in a way that it does not work in practice. You have two choices: 1. A developer makes a PR. They build the app and run it themselves to make sure it works, does what they intended, and nothing unexpected happens, and report on their testing in the PR. There's also a suite of automated tests. Between these you are confident that you can rubber-stamp it without touching the change yourself. But this requires them being able to run the code and intelligently think about it themselves, which AI cannot do. 2. A non-developer uses an LLM to make a PR. The change passes the tests, but you have no other validation about it because it was done by a bot that can't think about how the app actually works. As reviewer you have to pull down the change to run and validate it yourself. Now you are doing the same amount of work as before, except that instead of you telling the LLM to make the change, someone else did. The only difference is maybe that the activation energy for doing the work was avoided. Which, fine, is non-negligible. But let's not pretend like in the latter case legal "made the change". They did the 1% upfront work of asking the LLM to find where the change goes, and left you the other 99% of actually shepherding it through. The only changes that will work for at all are, like, updating copy/strings/icons---and honestly, at my last job at least, we already let legal (and product, etc) do stuff like that. I suppose the LLM might save them having to figure out how to use Git, at least. You might imagine there's some software out there where the automated tests are so thorough that you can trust a code change to not break anything if it passes the tests. I have personally never seen such a thing. And in practice many tests validate against strings and other small feature-level details, meaning that the kinds of code changes that other orgs are making are going to be touching the tests as well, so human verification is still required.

skovati 1y ago

I'm curious how many ICs are truly excited about these advancements in coding agents. It seems to me the general trend is we become more like PMs managing agents and reviewing PRs, all for the sake of productivity gains. I imagine many engineers are like myself in that they got into programming because they liked tinkering and hacking and implementation details, all of which are likely to be abstracted over in this new era of prompting.

awestroke 1y ago
At the end of the day, it's your job to deliver value. If a tool allows you to deliver more faster, without sacrificing quality, it's your responsibility to use that tool. You'll just have to make sure you can fully take responsibility for the end deliverables. And these tools are not only useful for writing the final code
- enjoylife 1y ago
  > these tools are not only useful for writing the final code This sparked a thought in how a large part of the job is often the work needed to demonstrate impact. I think this aspect is often overlooked by some of the good engineers not yet taking advantage of the AI tooling. LLM loops may not yet be good enough to produce shippable code by themselves, but they sure are capable to help reduce the overhead of these up and out communicative tasks.
  - tough 1y ago
    you mean like hacking a first POC with AI to sell a product/feature internally to get buy-in from the rest of the team before actually shipping production version of it?
- whyowhy3484939 1y ago
  It's actually not. My job description does not say "deliver value" and nobody talks about my work like that so I'm not quite sure what to make of that. > without sacrificing quality Right.. > it's your responsibility to use that tool Again, it's actually not. It's my responsibility to do my job, not to make my boss' - or his boss' - car nicer. I know that's what we all know will create "job security" but let's not conflate these things. My job is to do my end of the bargain. My boss' job is paying me for doing that. If he deems it necessary to force me to use AI bullshit, I will of course, but it is definitely not my responsibility to do so autonomously.

simianwords 1y ago

Does any one how the quality drops with size of codebase?

yanis_t 1y ago

So it's looking like it's only running in the cloud, that is it will push commits to my remote repo before I have a chance to see if it works? When I'm using aider, after it make a commit what I do, I then immediately run git reset HEAD^ and then git diff (actually I use github desktop client to see the diff) to evaluate what exactly it did, and if I like it or not. Then I usually make some adjustments and only after that commit and push.

flakiness 1y ago
You can think of this as a managed (cloud) version of their codex command line tool, which runs locally on your laptop. The secret sauce here seems like their new model, but I expect it to come to API at some point.
codemac 1y ago
watch the live stream, it shows you the diff as the completed task, you decide whether or not to generate a github pr when you see the diff.
danielbln 1y ago
You may want to pass --no-auto-commits to Aider if you peel them off HEAD afterwards anyway.
TechDebtDevin 1y ago
This takes all the soul out of programming, not sure why anyone likes to use llms like this.
- sumedh 1y ago
  Because from a business point of view, they just want a feature to be built, they really care about the code if it works.
sagarpatil 1y ago
It generates a PR. You decide if you want to merge, clone it locally, test, edit and then merge the code.

adamTensor 1y ago

not buying windsurf then???

motoxpro 1y ago
This would be the why of that acquisition as this needs a more integrated UI. Guessing by the speed at which this came out, this was in the works long before that acquisition.
- adamTensor 1y ago
  it is not even clear *if* they are going to buy windsurf at all. And thats a big if. This might've just been the 'why' that deal is not happening.
  - shmoogy 1y ago
    This probably came out to beat Google I/O or something similar - odd Friday release otherwise.

ianbutler 1y ago

Im super curious to see how this actually does at finding significant bugs, we've been working in the space on https://www.bismuth.sh https://www.bismuth.sh for a while and one of the things we're focused on is deep validation of the code being outputted. There's so many of these "vibe coding" tools and there has to be real engineering rigor at some point. I saw them demo "find the bug" but the bugs they found were pretty superficial and thats something we've seen in our internal benchmark from both Devin and Cursor. A lot of noise and false positives or superficial fixes.

deleted 1y ago
[deleted]

orliesaurus 1y ago

Why hasn't Github released this? Why it's OpenAI releasing this?!

adpirz 1y ago
It's on their roadmap: https://github.blog/news-insights/product-news/github-copilot-the-agent-awakens/#project-padawan-swe-agents-on-github https://github.blog/news-insights/product-news/github-copilo... But they aren't moving nearly as fast as OpenAI. And it remains to be seen if first mover will mean anything.
taytus 1y ago
Github moves too slow, and OpenAI moves too fast.
danielbln 1y ago
GitHub has released this, it's called Copilot Agent.

johnjwang 1y ago

Some engineers on my team at Assembled and I have been a part of the alpha test of Codex, and I'll say it's been quite impressive. We’ve long used local agents like Cursor and Claude Code, so we didn’t expect too much. But Codex shines in a few areas: Parallel task execution: You can batch dozens of small edits (refactors, tests, boilerplate) and run them concurrently without context juggling. It's super nice to run a bunch of tasks at the same time (something that's really hard to do in Cursor, Cline, etc.) It kind of feels like a junior engineer on steroids, you just need to point it at a file or function, specify the change, and it scaffolds out most of a PR. You still need to do a lot of work to get it production ready, but it's as if you have an infinite number of junior engineers at your disposal now all working on different things. Model quality is good, but hard to say it's that much better than other models. In side-by-side tests with Cursor + Gemini 2.5-pro, naming, style and logic are relatively indistinguishable, so quality meets our bar but doesn’t yet exceed it.

fourside 1y ago
> You still need to do a lot of work to get it production ready, but it's as if you have an infinite number of junior engineers at your disposal now all working on different things. One issue with junior devs is that because they’re not fully autonomous, you have to spend a non trivial amount of time guiding them and reviewing their code. Even if I had easy access to a lot of them, pretty quickly that overhead would become the bottleneck. Did you think that managing a lot of these virtual devs could get overwhelming or are they pretty autonomous?
- fabrice_d 1y ago
  They wrote "You still need to do a lot of work to get it production ready". So I would say it's not much better than real colleagues. Especially since junior devs will improve to a point they don't need your hand holding (remember you also were a junior at some point), which is not proven will happen with AI tools.
  - bmcahren 1y ago
    Counter-point A: AI coding assistance tools are rapidly advancing at a clip that is inarguably faster than humans. Counter-point B: AI does not get tired, does not need space, does not need catering to their experience. AI is fine being interrupted and redirected. AI is fine spending two days on something that gets overwritten and thrown away (no morale loss).

tough 1y ago

can someone give me a test prompt to one-shot something in go for testing? (Im trying something) what would be an impressive program that an agent should be able to one-shot in one go?

blixt 1y ago

They mentioned "microVM" in the live stream. Notably there's no browser or internet access. It makes sense, running specialized Firecracker/Unikraft/etc microkernels is way faster and cheaper so you can scale it up. But there will be a big technical scalability difficulty jump from this to the "agents with their own computers". ChatGPT Operator already does have a browser, so they definitely can do this, but I imagine the demand is orders of magnitudes different. There must be room for a Modal/Cloudflare/etc infrastructure company that focuses only on providing full-fledged computer environments specifically for AI with forking/snapshotting (pause/resume), screen access, human-in-the-loop support, and so forth, and it would be very lucrative. We have browser-use, etc, but they don't (yet) capture the whole flow.

ushakov 1y ago
we offer this with E2B Desktop Demo: https://surf.e2b.dev https://surf.e2b.dev SDK: https://github.com/e2b-dev/desktop https://github.com/e2b-dev/desktop
thundergolfer 1y ago
It's not our only focus at Modal but it's a big focus![1] Code agents are the killer use case for LLMs right now, and this complements our GPU inference and training capabilities. I'm quietly betting that agents increase the leverage of deterministic, reproducible devbox tech (eg. Nix, lockfiles, package mirroring), and this will end up being a huge win for us human engineers too. 1. https://modal.com/use-cases/sandboxes https://modal.com/use-cases/sandboxes

sudohalt 1y ago

When it runs the code I assume it does so via a docker container, does anyone know how it is configured? Assuming the user hasn't specified an AGENTS.md file or a Dockerfile in the repo. Does it generate it via LLM based on the repo, and what it thinks is needed? Does it use static analysis (package.json, requirements txt, etc)? Do they just have a super generic Dockerfile that can handle most envs? Combination of different things?

ilaksh 1y ago
I think they mentioned it was a similar environment to what it trains on, so maybe they have a default Dockerfile. Of course containers can also install additional packages or at least python packages.
- nkko 1y ago
  Yes, and one test failed as it missed pydantic dependency
hansonw 1y ago
More about that here! https://platform.openai.com/docs/codex#advanced-configuration https://platform.openai.com/docs/codex#advanced-configuratio...
- sudohalt 1y ago
  Thanks!
- sudohalt 1y ago
  It seems LLMs are doing a lot of the heavy lifting figuring out the exact test, build, lint commands to run (even if the AGENTS.md file gives it direction and hints). I wonder if there are any plans to support user defined build, test, and pre commit commands to avoid unnecessary cost and keep it deterministic. Also wonder how monolith repos (or distinct but related repos) are supported, does it run everything in one container or loop through the envs that are edited? I assume one easy next step is to just run GitHub Actions in the container since everything is defined there (assuming the user set it up)

bionhoward 1y ago

What about privacy, training opt out? What about using it for AI / developing models that compete with our new overlords? Seems like using this is just asking to get rug pulled for competing with em when they release something that competes with your thing. Am I just an old who’s crowing about nothing? It’s ok for them to tell us we own outputs we can’t use to compete with em?

piskov 1y ago
What the video: there is an explicit switch at one of the steps about (not) allowing to train on your repo.
- lurking_swe 1y ago
  That’s nice. And we trust that it does what it says because…? The AI company (openai, anthropic, etc) pinky promised? Have we seen their source code? How do you know they don’t train? Facebook has been caught in recent DOJ hearings breaking the law with how they run their business, just as one example. They claimed under oath, previously, to not be doing X, and then years later there was proof they did exactly that. https://youtu.be/7ZzxxLqWKOE?si=_FD2gikJkSH1V96r https://youtu.be/7ZzxxLqWKOE?si=_FD2gikJkSH1V96r A companies “word” means nothing imo. None of this makes sense if i’m being honest. Unless you personally have a negotiated contract with the provider, and can somehow be certain they are doing what they claim, and can later sue for damages, all of this is just crossing your fingers and hoping for the best.
  - tough 1y ago
    On the other hand you can enable explicit sharing of your data and get a few million free tokens daily
  - wilg 1y ago
    If you don't trust the company your opt-out strategy is much easier, you simply do not authorize them to access your code.
  - deleted 1y ago

ofirpress 1y ago

[I'm one of the co-creators of SWE-bench] The team managed to improve on the already very strong o3 results on SWE-bench, but it's interesting that we're just seeing an improvement of a few percentage points. I wonder if getting to 85% from 75% on Verified is going to take as long as it took to get from 20% to 75%.

Snuggly73 1y ago
I can be completely off base, but it feels to me like benchmaxxing is going on with swe-bench. Look at the results from multi swe bench - https://multi-swe-bench.github.io/#/ https://multi-swe-bench.github.io/#/ swe polybench - https://amazon-science.github.io/SWE-PolyBench/ https://amazon-science.github.io/SWE-PolyBench/ Kotlin bench - https://firebender.com/leaderboard https://firebender.com/leaderboard
- Bjorkbat 1y ago
  I kind of had the feeling LLMs would be better at Python vs other languages, but wow, the difference on Multi SWE is pretty crazy.
- ofirpress 1y ago
  Not sure what you mean by benchmaxxing but we think there's still a lot of useful signals you can infer from SWE-bench-style benchmarking. We also have SWE-bench Multimodal which adds a twist I haven't seen elsewhere: https://www.swebench.com/multimodal.html https://www.swebench.com/multimodal.html
  - Snuggly73 1y ago
    I mean that there is the possibility that swe bench is being specifically targeted for training and the results may not reflect real world performance.
- kristianp 1y ago
  Maybe a lot of the difference we see between peoples comments about how useful AI is for their coding, is a function of what language they're using. Python coders may love it, Go coders not much at all.

nadis 1y ago

In the preview video, I appreciated Katy Shi's comment on "I think this is a reflection of where engineering work has moved over the past where a lot of my time now is spent reviewing code rather than writing it." Preview video from Open AI: https://www.youtube.com/watch?v=hhdpnbfH6NU&t=878s https://www.youtube.com/watch?v=hhdpnbfH6NU&t=878s As I think about what "AI-native" or just the future of building software loos like, its interesting to me that - right now - developers are still just reading code and tests rather than looking at simulations. While a new(ish) concept for software development, simulations could provide a wider range of outcomes and, especially for the front end, are far easier to evaluate than just code/tests alone. I'm biased because this is something I've been exploring but it really hit me over the head looking at the Codex launch materials.

ai-christianson 1y ago
> rather than looking at simulations You mean like automated test suites?
- tough 1y ago
  automated visual fuzzy-testing with some self-reinforcement loops There's already library's for QA testing and VLM's can give critique on a series of screenshots automated by a playwright script per branch
  - ai-christianson 1y ago
    Cool. Putting vision in the loop is a great idea. Ambitious idea, but I like it.
    - tough 1y ago
      SmolVLM, Gemma, LlaVa, in case you wanna play with some of the ones i've tried. https://huggingface.co/blog/smolvlm https://huggingface.co/blog/smolvlm recently both llama.cpp and ollama got better support for them too, which makes this kind of integration with local/self-hosted models now more attainable/less expensive

DGAP 1y ago

If you still don't think software engineering as a high paying job is over, I don't know what to tell you.

whyowhy3484939 1y ago
It's high paying?
Reubend 1y ago
It might be over in the future, but it's still fine for the next few years at the minimum.
dreeves 1y ago
My counterargument: https://agifriday.substack.com/p/jevons https://agifriday.substack.com/p/jevons Short version: Jevons's paradox means that the more coding you automate away for developers (like with compilers in the past), the more in-demand those developers are. (Until AGI when all bets are off, of course.)

asdev 1y ago

is the point of this to actually assign tasks to an AI to complete end to end? Every task I do with AI requires atleast some bit of hand holding, sometimes reprompting etc. So I don't see why I would want to run tasks in parallel, I don't think it would increase throughput. Curious if others have better experiences with this

nmca 1y ago
with a bad ai it is pointless, with a good ai it is powerful. codex-1 has been quite good in my experience
masterj 1y ago
The example use-cases in the videos are pretty compelling and much smaller scope. “Here’s an error reported to the oncall. Give a try fixing it” (Could be useful even if it fails) Refactor this small piece I noticed while doing something else. Small-scoped stuff that likely wouldn’t get done otherwise. I wouldn’t ask LLMs for full-features in a real codebase but these examples seem within the scope of what they might be able to accomplish end-to-end
sagarpatil 1y ago
I am working with a 3rd party API (Exa.ai) and I hacked together a python script. I ran a remote agent to do these tasks simultaneously (augment.new, I’m not affiliated, I have early access) Agent 1: write tests, make sure all the tests pass. Agent 2: concert python script to fastapi Agent 3: create frontend based on fastapi endpoints I get a PR, I check code and see if it works and then merge to main. All three PR’s worked flawlessly (front end wasn’t pretty).

RhysabOweyn 1y ago

I believe that code from one of these things will eventually cause a disaster affecting the capital owners. Then all of a sudden you will need a PE license, ABET degree, 5 years working experience, etc. to call yourself a software engineer. It would not even be historically unique. Charlatans are the reason that lawyers, medical doctors, and civil engineers have to go through lots of education, exams, and vocational training to get into their profession. AI will probably force software engineering as a profession into that category as well. On the other hand, if your job was writing code at certain companies whose profits were based on shoving ads in front of people then I would agree that no one will care if it is written by a machine or not. The days of those jobs making >$200k a year are numbered.

deleted 1y ago
[deleted]
alfalfasprout 1y ago
Even ads have risk. Customer service has risk. The widespread proliferation of this stuff is a legal minefield waiting to be stepped on.
lispisok 1y ago
I wish but I dont think we could be any futher away from professionalizing like engineering/law/accounting/medicine. There was a deliberate effort to flood the field and lower salaries and developers were so full of hubris and thought there was infinite demand for their labor and went along with it and still are. Maybe some are learning given the job market the last few years. Despite software being in everything and harm to the public due to bad software has materialized every developer seems vehemently against professionalizing. Do you want a surgeon that went to surgeon bootcamp because "you dont need all those years in medical school to learn how to remove an appendix"? Do you even want an accountant who went to accountant bootcamp to do your taxes?
- RhysabOweyn 1y ago
  Obviously there is no way to really predict when this would happen, but I don't think it will be up to developers to decide whether it happens or not. In Texas for example, the legislature forced engineering to be professionalized (or regulated) in an emergency session after a school in a well off area exploded in a gas explosion (https://en.wikipedia.org/wiki/New_London_School_explosion#Investigation_and_legislation https://en.wikipedia.org/wiki/New_London_School_explosion#In...). I also do not think this is limited to software engineering. Medical doctors and accountants have faced the squeeze in recent years too. There are tons of (bad) DO med schools opening up across the country that will be flooding the field before long, nurse practitioners and physicians assistants get to do more and more work that only doctors got to do, and more and more accounting is being offshored. The question is when things get so bad that even the powerful decide to actually do something about it.

SketchySeaBeast 1y ago

Is this the same idea as when we switched to multicore machines? The rate of change on the capabilities of a single agent has slowed enough now the only way for OpenAI to appearing to be making decent progress is to have many?

ionwake 1y ago

Im sorry if Im being silly, but I have paid for the Pro version, $200 a month, everytime I click on Try Codex, it takes me to a pricing page with the "Team Plan" https://chatgpt.com/codex#pricing https://chatgpt.com/codex#pricing. Is this still rolling out? I dont need the team plan too do I? I have been using openAI products for years now and I am keen to try but I have no idea what I am doing wrong.

mr_north_london 1y ago
It's still rolling out
- ionwake 1y ago
  Thx for the reply, Im in london too ( atm )
jdee 1y ago
im the same, and it appeared for me 2 mins ago. looks like its still rolling out
- ionwake 1y ago
  cool it appeared - I wa sjsut worried it was a payment issue. thanks guys.
throwaway314155 1y ago
They do this with every major release. Never going to understand why.

hintymad 1y ago

I remember HN had a repeating popular post on the the most important data structures. They are all the basic ones that a first-year college student can learn. The youngest one was skiplist, which was invented in 1990. When I was a student, my class literally read the original paper and implemented the data structure and analyzed the complexity in our first data structure course. This seems imply that the software engineering as a profession has been quite mature and saturated for a while, to the point that a model can predict most of the output. Yes, yes, I know there are thousands of advanced algorithms and amazing systems in production. It's just that the market does not need millions of engineers for such advanced skills. Unless we get yet another new domain like cloud or like internet, I'm afraid the core value of software engineers: trailblazing for new business scenarios, will continue diminishing and being marginalized by AI. As a result, we get way less demand for our job, and many of us will either take a lower pay, or lose our jobs for extended time.

deleted 1y ago

[deleted]

theappsecguy 1y ago

I am so damn tired of all the AI garbage shoved down our throats every day. Can't wait for all of it to crash and burn.

jetsetk 1y ago
At some point the bubble will pop
- boshalfoshal 1y ago
  Higher chance of AGI/most SWEs being out of a job for an extended period of time than the bubble popping imo.
  - missedthecue 1y ago
    Agree with this. It's so strange to me that out of the all the anti-AI arguments one can derive, the anti-AIers have settled on riding with the "it's useless" argument, when it's plainly one of the most useful tools created since the desktop computer.
    - Collisteru 1y ago
      These people remind me of the people in the late 19th century who were scared of electricity because they thought it could remove all the air from rooms and suffocate people.
  - greatpostman 1y ago
    Way higher chance

fullstackchris 1y ago

Reading these threads its clear to me people are so cooked and no longer understand (or perhaps never did) understand the simple process of how source code is shared, built, and merged together with multiple editors has ever worked

swisniewski 1y ago

Has anyone else been able to get "secrets" to work? They seem to be injected fine in the "environment setup" but don't seem to be injected when running tasks against the enviornment. This consistently repros even if I delete and re-create the enviornment and archive and resubmit the task.

bmcahren 1y ago

So there's this thing called "Setup Scripts" but they don't explicitly say these are equivalent to AWS Metadata and configured inside of Codex web interface - not a setup.sh or a package.json preinstall declaration. I wasted several hours (and lots of compute where Codex was as confused as I was) trying to figure out how to convince codex to pnpm install.

gpeal 1y ago
Codex engineer here. Can you elaborate on what was confusing? I would love to make it more clear.
- rahulyc 1y ago
  I didn't realize the setup script had to be done in the UI over in the environment tab. I assumed it would be reading something like setup.sh from the codebase. Docs could make that more clear
- nater5000 1y ago
  I just found this thread while trying to figure out where to set the setup script. The docs should probably make that more clear.

CSMastermind 1y ago

I really wish they'd add support for Gitlab or better any arbitrary git repo.

eru 1y ago
Well, Microsoft bought GitHub and Microsoft sort-of co-owns OpenAI, too.

ZeroCool2u 1y ago

"23 SWE-Bench Verified samples that were not runnable on our internal infrastructure were excluded." What does that mean? Surely this should have a bit more elaboration. If you're just excluding a double digit number of tasks in the benchmark as uncompleted, that should be reflected in the scores.

solresol 1y ago

I'm not sure what's wrong with me, but I just wasted several hours wrestling codex to make it behave. Here's my workflow that keeps failing: - it writes some code. It looks good a first glance - I push it to github - automated tests on github show that there's a problem - go back to codex and ask it to fix it - it does stuff. It looks good again. Now what do I do? If I ask it to push again to github, then it will often create a pull request that doesn't include stuff from the first pull request, but it's not a pull request that stacks on top of the previous pull request, it's a pull request that stacks on top of main. When asked to write something that called out to gpt-4.1-mini, it used openai.ChatCompletion.create (!?!!?) I just found myself using claude to fix codex's mistakes.

landl0rd 1y ago
[flagged]
csomar 1y ago
I think you need to run the tests locally before you push the PR. I actually think you need to (somehow?) make this part of the generation process before Codex proposes the changes.
bargainbin 1y ago
I have this issue with Devin. Given my limited knowledge of how these work, I believe there is simply too much context for it to take a holistic view of the task and finish accordingly. If both OpenAI and Devin are falling into the same pattern then that’s a good indication there’s a fundamental problem to be solved here.
fcoury 1y ago
I upgraded to Pro just because of Codex and I am really not impressed. Granted, I am using rust so that may be the issue (or skill issue on my end too). One of the things I am constantly struggling with is that the containers they use are having issues to fetch anything from the internet: error: failed to get `anyhow` as a dependency of package `yawl-core v0.1.0 (/wor kspace/yawl/core)` Caused by: download of config.json failed Caused by: failed to download from `https://index.crates.io/config.json` Caused by: [7] Could not connect to server (Failed to connect to proxy port 8080 after 30 65 ms: Could not connect to server) Hopefully they fix this and it gets better with time, but I am not going to renew past this month otherwise.

bearjaws 1y ago

I spend maybe $20 a month on Claude code, $200 a month is hard to swallow. I don't understand why OAI puts their alpha release products under a $200 a month plan instead of just charging for tokens.

Bjorkbat 1y ago
My weird theory: the alpha releases are more expensive than people realize, and OpenAI can afford to launch money-losing alphas on Pro because few users have Pro accounts and the lose less money per Pro user. If anyone could use Codex, you’d want to price the tokens at a rate where you don’t lose money, or at least not too much money. If you price the tokens too high, then not only are people not using Codex, but customers and investors alike begin to lose faith, they no longer “feel the AGI”.

zrg 1y ago

I've been experimenting with providers offering similar functionality for the last year and it is really vastly superior experience this codex like approach than cursor, devin etc