9 ms·

Harness engineering: Leveraging Codex in an agent-first world

• bko 4mo ago
  > We had weeks to ship what ended up being a million lines of code... Five months later, the repository contains on the order of a million lines of code across application logic, infrastructure, tooling, documentation, and internal developer utilities. Over that period, roughly 1,500 pull requests have been opened and merged with a small team of just three engineers driving Codex. This translates to an average throughput of 3.5 PRs per engineer per day, and surprisingly the throughput has increased as the team has grown to now seven engineers. Importantly, this wasn’t output for output’s sake: the product has been used by hundreds of users internally, including daily internal power users. That's an insane level of throughput. What's a good baseline? Prior to agentic coding, whats the typical number of PRs engineers were expected to push? Maybe a 2-10? Do people feel the software has gotten better in the last 6 months? The number of engs is prob the same so we should expect maybe 5x faster cycle in major software apps, but I don't see it. The AI apps do change very fast but given its a very new field, I'd expect as much. But outside of that, I don't see it.
  • krackers 4mo ago
    They never specified what exactly the product was, without which it's impossible to judge the post. For some reason most of the uses of "agents" are to build yet other AI products, it's turtles all the way down. Maybe that says more about the field of harnesses than it does about the power of "agents".
    • becomevocal 4mo ago
      Feels like the active discovery going on is trying to understand what is computer vs what is AI, for every product. Agents help a ton with the discovery, but the act of building a product needs a deeper level of thought and validation to make it actually better than what came before. So IMO what you see is people still learning what needs to be understood and crafted first hand to make a product better (including economics) We’ll get there if more of us try
    • theptip 4mo ago
      There is a sense in which it doesn’t matter at all; many of the limitations of agents in large codebases are just the context management challenges. So proving that you can cohere and progress at O(1m) is a useful scale observation. “Can I use agents in my 1m line codebase?” There is of course another sense in which the output quality is the only thing that matters. “Can I use agents to build a 1m line codebase that I want to maintain going forward.” I take this as being exclusively a tech demo of the former. Quality (feature velocity, bugs, scalability) is not demonstrated.
  • Aperocky 4mo ago
    > ended up being a million lines of code This almost reeks of "I've never cleaned up our code base because there is too much code, and didn't even bother having agents/LLM cleaning them up". You almost never need a million lines of code - this includes your software, infra, testing and operational tools. You didn't ship the linux kernel in 3 weeks and you know it. The code is already speghetti and it achieve the basic functions OK but it will harder and harder to simplify and untangle and maintain.
    • girvo 4mo ago
      Yeah I cannot see how "we shipped 1 million lines of code in three weeks" is... something to be proud of haha
    • bombcar 4mo ago
      Even the linux kernel doesn't need millions of lines of code; most of the actual LOC is device drivers, and you don't need all of them, you just need the ones for the devices you have.
      • Chu4eeno 4mo ago
        And Linux maintainers are actively pushing to radically cut down on the LOC by eliminating drivers etc.
    • faustin 4mo ago
      They directly address routine code cleanup and regularly paying down technical debt near the end of the article.
      • Aperocky 4mo ago
        I stand corrected, but the LOC being advertised still make me doubt the efficacy of their process.
    • zahlman 4mo ago
      As a point of reference, 1MLOC is about the size of the entire Python standard library including tests, as well as stuff like IDLE. (Well, the Python part of the code. There's about half that much again of C in Modules/ .)
  • torben-friis 4mo ago
    Here's a fun one: firefox lists its current count at about 2.5M LOC, from roughly 1M commits during the years. You end up with about 3 lines added per commit, which is not ridiculous when you consider that most would be editions rather than full additions. Here, we have 1500 PRs and 1M LOC, which is about 650 added LOC per PR. Remember, not 650 lines total in the PR, but +650 balance after additions-removals. Fun questions for attentive readers: - What does a project growing at a rate of one full firefox-codebase worth of LOC per year look like, a decade down the line? - What does the line count say about the verbosity of the tool, and what does it say about outcomes that the purpose of the project isn't clearly disclosed? - Do we have reasons to care about LOC in a world where we don't write code manually? What happens to token usage numbers when the codebase is significantly larger? - If it was confirmed that LLM usage blows up your line count, what's the implication for codebases that want to return to manual coding after months of usage? (Say, because the tool gets expensive).
    • CleanCoder 4mo ago
      When I got to the 1M LOC I involuntarily paused feeling like this must be satire.
    • therealdrag0 4mo ago
      Does the Firefox LOC include ALL forms of text: infrastructure (Firefox doesn’t have), documentation, developer scripts,tests, etc? How is the test coverage of Firefox?
    • stult 4mo ago
      > - Do we have reasons to care about LOC in a world where we don't write code manually? What happens to token usage numbers when the codebase is significantly larger? Yes, at least to the extent that we care about context windows and tokens consumed by coding agents processing code that is ultimately irrelevant to their assigned task. Anecdotally, I've found keeping file sizes small has been important for agentic coding not just to maintain human readability, but also for optimizing agent performance, precisely because it limits the amount of incidental context they load while working a problem, because they generally load entire files rather than just parsing the part relevant to their current assignment as a human might. That smaller file size thus reduces input noise and the LLM generates a tighter solution, which in turn reduces input noise for future solutions. Or at least this strategy avoids a death spiral into exploding context length. I expect (but cannot currently prove) that keeping overall LOC down yields similar benefits even when file sizes are kept small because it spares the LLM from parsing potentially relevant files that prove irrelevant to its current task.
  • aleqs 4mo ago
    > should expect maybe 5x faster cycle in major software apps To what end and what would that even look like though? Enshittifying everything at maximum speed? The apps/platforms I use regularly - GitHub, Spotify, Google maps (just to name a few), have gotten noticeably shittier in recent times.
    • linsomniac 4mo ago
      >GitHub, Spotify, Google maps (just to name a few), have gotten noticeably shittier in recent times. What if AI lets you create new versions of those tools, but without the enshitification? I say that being in the "soaking" stage of using AI to rebuild a shitty software project in 70KLOC over about 2 weeks of spare time, so this may not be as theoretical as you might think.
      • NoraCodes 4mo ago
        > What if AI lets you create new versions of those tools, but without the enshitification? I'm not sure I fully understand what you're saying here. Isn't the value of these tools almost entirely independent of their actual software? That is, we have many good open source, self-hostable forges (Forgejo, sr.ht, etc.), lots of great music player software (Jellyfin, Symphonium, etc.), and decent maps software (OsmAnd and Organic Maps). People use GitHub, Spotify, and Google Maps -- perhaps even _put up_ with their often bad/glitchy software -- because of network effects (all three) and content/licensing partnerships (Spotify/GMaps). That proprietary data isn't something AI can help you with, right?
        • nostrademons 4mo ago
          AI coders are great for making scrapers, possibly because AI companies use their own tools to make an awful lot of scrapers.
        • linsomniac 4mo ago
          It really depends on the use-case. For example, my most starred github repo is a tool to convert Spotify playlists to YouTube Music (that was done pre-AI). Github depends on what issues you have with it, what your use case is, and whether you can leverage some of the network effects via API from the github source. Maps, same story.
  • ai-roundup 4mo ago
    [dead]
  • jakolaptu 4mo ago
    It is likely better because AI agents make access to domain knowledge easier. However, I would wager that the problem is people don’t remember the code well. The problems are going to be long-term as the pace of change increases. If you think about it, successful products rely on designing well-thought-out experiences, customer discovery (see all the Forward-Deployed Enginneer job listings at OpenAI) so the code velocity somewhat becomes irrelevant. If you’re solving the right problem and you’ve got a good team then competitive advantage comes from somewhere OUTSIDE of code velocity. The more important question I think is does faster code yield more value long-term? At the moment, it’s like yeah we do 3.5 pull requests per day. I’m thinking, great, good for you. You could also combine three pull requests into one and then you’re doing 1 per day. This is quantitative data that doesn’t really mean anything tangible.
  • dchftcs 4mo ago
    This is a lot tamer than what Claude Code's team claims tbf.
  • techblueberry 4mo ago
    I’ve been vibe coding a lot over the past year or so, and I think I’m going to stop. In fact, I sort of want to challenge myself to see, can I go back to a sort of the fork in the road with the old copilot autocomplete workflow and really maximize that. Be in the drivers seat for most of the code being written, but find ways to use AI to really enhance the flow state / remove blockers. Tools only minimal actual code generation.
    • 59nadir 4mo ago
      I would be very impressed with someone who's been vibecoding "a lot" for about a year who could then go back to being fully in the loop for even 50%. I would even say I'd expect withdrawal symptoms at that point. The dopamine hits are core to why people even do vibecoding (or vibecoding-in-a-dress/spec-driven development) and why they tend to overestimate its output so much. Hell, it's core to all forms of LLM-assisted development (because it feels like magic), but most of the other forms are more value, less delusion.
      • techblueberry 4mo ago
        I actually don’t find vibe coding satisfying is one of the many reasons I’m going back. I feel a little of what you’re talking about, but I’m a nerd. I like to code. But I’m not dismissing your concern. Because it is one of the reasons I’m making this decision. I’m a professional. I’m not just here to feel good I’m here to do a good job over the course of a career. I think all in, when you think about writing good maintainable, software, learning, staying mentally sharp, and speed put together. Vibe coding could be less effective and maybe even in the aggregate “slower”.
      • chasd00 4mo ago
        The dopamine hit is real, I feel like that was identified early on by OpenAI and probably lit a fire to get ChatGPT in the hands of the public. Bf Skinner (I think) is the one who narrowed in on variable ratio reward systems to maximize operant conditioning. An LLM, with hallucinations and imperfections, is the perfect variable ratio reward system. It’s no wonder they’re getting pushed so hard along with a consumption based pricing model. Whether you’re a human, rat, plant, bacteria there’s no real defense against that kind of conditioning. First hit on Google https://www.simplypsychology.org/operant-conditioning.html https://www.simplypsychology.org/operant-conditioning.html
  • AbrahamParangi 4mo ago
    The average efficiency improvement is closer to something like 2-3x per Anthropic’s numbers and this is only the rate at which software can advance. Do you expect to notice if 12 months of software engineering on a project you’re following gets done in 6 months? I suspect not. The root cause is that the acceleration is pareto distributed so the modern engineering team at the moment looks like one 10x engineer, one 5x engineer, and the rest are approximately 1.5x engineers.
  • ano-ther 4mo ago
    It feels like the update cadence has indeed sped up. But not necessarily quality. Looking at MS Office I notice a lot of small changes recently that are mostly annoying. Things like Word comments losing the focus after you @-tagged a colleague, needing to click the Outlook search field twice before you can enter text, Outlook mobile date picker losing its ability to show your and attendee's availability. So it looks like lots of throughput, but unfortunately breaking features that work. Or wasting time on things that don’t matter such as the status bar of OneDrive search circling around the input field.
  • hanzeweiasa 4mo ago
    [flagged]
  • HorizonXP 4mo ago
    [flagged]
  • endomorphosis 4mo ago
    I have been building an entire operating system ( not figuratively) Prior to ai autocomplete 500 loc a day and then with ai autocomplete I could do 2500 a day and now 50k is pretty normal. Walking around tech week with my phone yielded 150k this week
  • getnormality 4mo ago
    We've known for decades that output metrics like LOC/day are very bad measures of real productivity in software. But they seem to be back in vogue in the age of AI, because AI is so good at maxing these useless metrics, and we need to show how impressive our AI is and how impressive our usage of AI is.
  • bicepjai 4mo ago
    It’s sad we back at measuring code quality with lines of code
• Sarkie 4mo ago
  I would never dare put that in production
• jlintc 4mo ago
  [flagged]
• darepublic 4mo ago
  Codex pushed an update that made my old threads inaccessible. This takes a million of lines to put out a half baked crud app?
• rfw300 4mo ago
  I understand that the’ve written zero lines of code for this application, but would it kill them to write a few lines of the blog post by hand? Forcing readers to wade through an unceasing string of LLM clichés demonstrates the opposite of the point you’re trying to make—that the consumers of your work are worse off because you exercised no human judgment in creating it.
  • whateveracct 4mo ago
    these AI companies are high in their own supply and/or they are really trying to shift norms where AI is required to exist (which is good for their financials)
• knicholes 4mo ago
  Everyone is criticizing the number of lines of code and the lack of attention that must certainly have been applied to generate that code and push it into production. What is being ignored is this awesome prompt that is almost certainly better than having no agents.md or plans.md or whatever you've come up with, to add validation steps for committed changes. You're still free to look at your code, the changes, and ask the agent to clean up. Try it. It's really nice.
• drchaim 4mo ago
  But this is almost what we have been doing for the last 3/5 months, isn’t?
  • wilsonnb3 4mo ago
    Article is from February so that tracks
  • fbrncci 4mo ago
    Well to a lot of people this is still a foreign concept.
• angrydev 4mo ago
  Published Feb 11, 2026
  • ukuina 4mo ago
    Might as well be 2025.
• murat124 4mo ago
  The other day I came across to a video showing workers in a e-vape factory. They pick up a bunch of e-vapes from the conveyor belt (each has 6 e-vape think), stick in their mouth and vigorously vape all of them for about 5 seconds, then test the next bunch. Humans reviewing hundreds of lines of change in a PR written by AI is not very different.
  • prakashn27 4mo ago
    Very true. If a PR has 1000 lines I would check only a handful full of them and leave the rest for test suit .
  • Eridrus 4mo ago
    You can do statistical testing of the e-vape line because you have a specific criteria and well defined tolerances that you can define on a per-sample basis and that the factory meets with some acceptable 9s of reliability. PRs are not like this because a single bad PR can be catastrophic for your business in a way that a single bad e-vape cannot. I would also argue that the current output from the AIs when sampled by software engineers regularly doesn't meet the bar of quality we want in our product, hence the need to review every PR and fix a substantial fraction. If you can start to bound the impact of changes and the outputs begin to be generally acceptable unsupervised, such that all you're doing is double checking that nothing has regressed in the factory, then the sampling approach can work.
• varenc 4mo ago
  digression: It's interesting this was submitted to HN over 15 times since it was published in February: https://hn.algolia.com/?dateRange=all&page=0&prefix=false&query=https%3A%2F%2Fopenai.com%2Findex%2Fharness-engineering&sort=byPopularity&type=story https://hn.algolia.com/?dateRange=all&page=0&prefix=false&qu... But this is the only submission that's had any traction. Since the content is nearly the same for all submissions, it highlights how getting to the front page can be a bit random. (Though this is the only one that capitalized 'Leveraged' so maybe that's the secret)
  • swyx 4mo ago
    time of day also matters
    • pramodbiligiri 4mo ago
      (OP here) The truth is much simpler. I mailed the mods and they bumped it up in the second chance pool [1] :) 1. https://news.ycombinator.com/item?id=26998309 https://news.ycombinator.com/item?id=26998309
      • swyx 4mo ago
        ha ok but why do that for a post you didnt write?
        • pramodbiligiri 4mo ago
          Because I thought this is a good blog and more people should get to know. Found it by chance in some other thread: https://news.ycombinator.com/item?id=48414452 https://news.ycombinator.com/item?id=48414452. Did the same with this Cloudflare one the other day: https://news.ycombinator.com/item?id=48276152 https://news.ycombinator.com/item?id=48276152
        • dredmorbius 4mo ago
          I do this somewhat frequently myself. Why? Because good stuff often falls through the cracks, and I like to see people discuss substantive material rather than just the "easy" submissions which tend to attract quick votes. This is one of the best ways to have a positive impact on the HN submission queue. And mods appreciate the suggestions (they can't keep up with the firehose either, and are well aware of the HN submission queue's weaknesses). See: <https://news.ycombinator.com/item?id=46018486 https://news.ycombinator.com/item?id=46018486>, <https://news.ycombinator.com/item?id=26998308 https://news.ycombinator.com/item?id=26998308>.
• faangguyindia 4mo ago
  Codex updates usually appear every few hours (i am not saying this how often it's published) but that's my perception as a user. Often i update codex just to see new update within an hour so. Many times those updates are not properly tested, for example in one update the model selector got completely changed. then next hotfix was pushed which restored original.
  • dawnerd 4mo ago
    Who needs a QA team when you can just test on users and iterate instantly /s
    • frictasolver 4mo ago
      [flagged]
• zatkin 4mo ago
  I worry most about blindspots with this kind of approach. Let's say that this repository goes on for years, at which point the docs folder is several MB in size. Would Codex be able to think outside of the box? Or would the aggregate of the Markdown content fundamentally cover enough ground to prevent it from thinking of novel new approaches to existing problems?
  • esikich 4mo ago
    You tell it to update the docs: not append. I've done the same thing with a readme in the root with links to the docs. After every commit, before the push, I have my agent "update all relevant and related docs, add or remove what's needed" or something to that extent. And it works remarkably well. I also have an append only change log it's supposed to add to. Between that, good commit messages, and comprehensive testing, I've built a homebrew OS and updating it is remarkably smooth. Runs a homebrew FTP and HTTP server and can run Wolfenstein. Working on DOOM right now. Close, but sound has been difficult. https://github.com/ESikich/smallos https://github.com/ESikich/smallos
    • vibcdingenjoyer 4mo ago
      Yep. You’ve got to have it update the docs. After a few sessions, if I forget to request this, opus starts rehashing the same tasks and finds that they are complete - and sometimes still won’t update those docs unless I ask. Another tip is to condense the doc files into the minimal required. Sometimes I’ll end up with 5 to 6 floating around in various states of staleness. Condensing to 2-3 and removing completed tasks seems to help a lot
    • satvikpendem 4mo ago
      Someone else in the comments said to have it make a static website with the info instead with clickable pages and sections so it reads only the content it needs to rather than dumping a long file into context windows. Although I suppose you can have a ToC in the readme too with multiple smaller markdown files as references.
  • therealdrag0 4mo ago
• bronny1989 4mo ago
  why do you have “weeks” to ship what would take “months”?
  • deleted 4mo ago
    [deleted]
  • andai 4mo ago
    > We intentionally chose this constraint so we would build what was necessary to increase engineering velocity by orders of magnitude.
    • robotresearcher 4mo ago
      I guess orders of magnitude ain’t what they used to be.
    • whateveracct 4mo ago
      holy wow. so they really are deluded and are believing fallacies from the mythical man month
• andai 4mo ago
  > To drive a PR to completion, we instruct Codex to review its own changes locally, request additional specific agent reviews both locally and in the cloud, respond to any human or agent given feedback, and iterate in a loop until all agent reviewers are satisfied (effectively this is a Ralph Wiggum Loop ). https://ghuntley.com/loop/ https://ghuntley.com/loop/
  • aniceperson 4mo ago
    ok, but you had 1x token to generate , then more 1x to review locally, 1x for the agent local, 1 x for the cloud. then ???x until all bots are satisfied. You end-up spending at least 5x the amount of tokens for maybe prediction machine to find a discontinuity? I would say a way better approach is 1.123x to generate code + tests + passing analysis tools + human review + 1x "simplify as much as possible", than letting the snake its own tail without boundaries.
• apical_dendrite 4mo ago
  I wonder why we as engineers aren't protesting AI in the same way that artists and people in film and television are. This post should instill the same terror that visual artists feel. If you're a more senior person in tech, this post is effectively saying that a large portion of your skillset is about to become completely worthless. This goes beyond the skills involved in writing the code. Everything that you've learned over years about how to determine whether code is good or bad, and what practices make an engineering team effective is not just obsolete, it's fundamentally counter-productive because it assumes a slow, human-centric process that requires you to actually review and understand the code. Even your ability to mentor junior engineers is now obsolete, because all that experience you've built up is now worthless to them. If this is the approach the industry takes, particularly when combined with a lack of interest in quality from the business (and let's face it, consumers have shown us that they're happy to pay for cheap crap), it's hard to see much of a future for software engineers. You don't need thousands of people with deep technical expertise, you need a handful of manager-types, who will focus on defining product and business requirements and configuring how the AI gets enough context to implement the requirements. Maybe, if we're extremely lucky, there's so much demand for software that total employment doesn't fall off a cliff, but the nature of the work will change so much that many older, more expensive engineers will become unemployable. Those who remain will have to accept that the skills they spend decades developing are now worthless, that younger engineers no longer respect or listen to them, that the business no longer sees them as experts worthy of respect, but old fogies who grew up in a different world. Joe Biden liked to say that a job is more than just a paycheck, it's part of your identity and your sense of self-worth. We're all very used to a certain level of respect (and commensurate remuneration). If you don't think that's true, compare how a software engineer is treated to how a warehouse worker is treated. What happens when we lose that?
  • linsomniac 4mo ago
    >a large portion of your skillset is about to become completely worthless I'm not convinced of that. I watched a video of an architect using AI to create architectural drawings. It became very clear to me that he has a lot of skills and terminology that helped him produce something very specific, in a few minutes. I've been working on some home improvement stuff including a studio/shed and I've struggled to produce even something simple (currently trying to get a conversation packet on the roof trusses to take the the permit department to get started). Even with my high school architecture class. After watching that I wonder how much of what I'm doing with AI that looks easy is because I hae a deep technical knowledge, plus 3 years of heavy work with AI.
• Aperocky 4mo ago
  1 million lines of code aside, I feel like anyone who seriously thought about this would eventually run their own harness. Just like .vimrc and .zshrc, the harness "code" itself can be easy and personal. Provided that it's built on working and existing construct such as tmux.
• shepherdjerred 4mo ago
  This mirrors exactly what I have been doing. - Give Claude/Codex a way to verify its own work (browser, smoke tests, e2e tests, high-fidelity local environment) - Keep all context (issue tracking, docs, ideas, plans, worklogs) in-repo (https://github.com/shepherdjerred/monorepo/tree/main/packages/docs https://github.com/shepherdjerred/monorepo/tree/main/package...) - Give Claude/Codex access to observability (Grafana, Prometheus, Tempo, PagerDuty) - Have Claude/Codex follow good engineering guidelines like fail-fast, type safety, parse at boundaries I haven't yet been able to achieve full autonomy due to cost and CI load on my homelab.
  • para_parolu 4mo ago
    Does it yield good results? I found that instead of docs it’s easier just to ask ai to read code. I feel like this is same as comments in code. Become outdated fast
    • shepherdjerred 4mo ago
      I don't really use "docs" for documentation. I've prompted Claude/Codex to always write a "log" and save it in-repo to track what it did and why. I've found this to be really helpful, e.g. "you did this last week, and now some other thing is happening" or "you tried this approach before to solve alert X but it didn't work" -- except it can discover this itself. https://github.com/shepherdjerred/monorepo/tree/main/packages/docs/logs https://github.com/shepherdjerred/monorepo/tree/main/package... I've also used it to store TODOs and plans. For example I might want to explore some idea and defer it for later, or some weekend have it execute on some tech debt I've put off. One last use case is asking "what did I work on in the last 2-3 weeks, is it healthy, and what additional quality checks can/should I do; is there any follow-up work?"
      • DenisM 4mo ago
        I find that preserving logs that contain errors will confuse future sessions even if the errors were corrected at the time. Do you have that problem? Essentially preserving logs extends the context window with all related problems.
• jonmoore 4mo ago
  This would be much more convincing if the repos, issue trackers, etc. were accessible.
• yurimo 4mo ago
  What I still can't understand is why is massive amount of code generated is a flex? I don't feel that software has gotten a lot better in past 3 years, only sloppier. It's surprising to me that people who know about reward hacking choose a simple objective like lines of code generated as a signal for quality. I'd argue you have to optimize for less lines generated as possible while secondary optimization should be readability for humans. I suspect it's not seen as a problem by providers because more lines generated means more tokens used and hence more billing put out on customers. And if I am working on an existing codebase then isn't a good commit often a negative sum between added and removed lines? I don't want to bloat my codebase but make it more polished and elegant. After reading that I wonder if what they have done could have been accomplished for a far fewer LoC budget.
  • dumbdumb125 4mo ago
    It's a huge flex if the alternative is no code at all. Reward hacking aside, LOC resonates with me in the sense that I've seen 10+ projects to fruition that wouldn't have even begun without an agentic harness and an LLM. It's like the difference between doing stock price predictions with binary "up" or "down" histories and trying to figure out how to normalize actual price histories (basically impossible). The binary work gives a well-defined signal.
  • crdrost 4mo ago
    Yeah so all of personal computing—text editing, SVG antialiasing, etc, fits in 20,000 LOC (VPRI's STEPS project) so a million lines of code is 50 reïnventions of personal computing. BUT: it is unlikely that humans would have solved this problem in 20 kLOC. Sussman said “we really don't know how to compute!” as his talk title and LLMs had to ossify some pre-existing voice as the forever programming habitus and it chose a persona that doesn't know how to program—because we don't —and now we are stuck with it. Claude is our tickets, our implementations, our documentation... And if you tell it “hey the node role should not have those permissions, that should be a service account” it will happily do the right thing, but it has no intrinsic sense of taste and the error message it's trying to clear just says “the node role doesn't have that permission and the system prompt says “keep it short, stupid ” and graybeards might be our last bulwark.
• EnPissant 4mo ago
  > Over the past five months, our team has been running an experiment: building and shipping an internal beta of a software product with 0 lines of manually-written code. This is such a common thing among software engineers nowadays that I was very surprised that OpenAI would open with that line as if it were mind blowing. But then I saw it was published in February and OP is just reposting it to farm karma.
• trytodupe 4mo ago
  [dead]
• charintstr 4mo ago
  I am at a major company that is essentially vibe coding. I’ve shipped about 100k LoC this entire half and am toward top 10% of my team. I find it likely that either A. The code is absolute garbage and is speed for speed sake B. They’re using an internal model that is a generation beyond GPT 5.5 I say this because we’ve attempted to do something similar using the latest gen Claude models and a significantly larger team. The code is probably along the lines of millions LoC but is an absolute mess because of vibing. There’s a price you pay for speed
  • therealdrag0 4mo ago
    I like how they said they were spending 20% of their time addressing slop. Sounds like they’ve tried to automate the slop correction but it’s a good honest reminder. Additionally it’s an internal tool, which is likely much more amenable to slop.
  • iso1337 4mo ago
    Q1 - How much effort did you put into deterministic guardrails like AST linters, etc? I find there’s a ton of slop unless hard guardrails are added, eg step 1 is just around syntax, step 2 is to enforce mental models You still need someone steering direction and have a logically consistent idea of what you actually want to build Q2 - I find that vibe coding really accelerates FE projects because it’s possible to run everything locally and check results For pure distributed infra backend more investments have to be made into the devloop to be able to shift left the feedback loop and decouple it from humans or real deploys
  • sajithdilshan 4mo ago
    I think vibe coding is okay for a small internal tool, dashboard, etc. but it’s definitely a no-go for a production running service.
• IAmGraydon 4mo ago
  Title should probably be marked with (February 2026).
• drivebyhooting 4mo ago
  I wish these breathless blog posts would actually try to be more didactic. For example, actually doing a walkthrough of how to set up these allegedly super powered workflows and concrete demonstrations. I’m not an AI skeptic. Rather I’d don’t want to miss out on any actual super powers.
  • nimonian 4mo ago
    I do quite a lot of what this post describes in a reasonably large project. Here's what works for me: - write gherkin features for new features; update them for enhancements; don't touch them for refactors. Label your PRs with these nouns. - use pre-push hooks for type checks, linting, unit tests, and other quick, scriptable validations. - make a viteperess subsite in your repo, have the agents maintain it - document important principles, architecture, etc. - make a cli command which lists all pages along with the yaml frontmatter description so agents can choose what to read without blowing up the context window. - use ddd and monorepo - write your logic in headless layers, and compose layers into apps. agents navigate layers very successfully. - use zod (or your language equivalent) and contract-first API development; this is my favourite bit tbh, I use orpc - make a single skill called "code" which describes the lifecycle: open a worktree, setup .env to guarantee no conflict with other agents (choose unused ports etc - docker is good here), write or update feature file (this is where you negotiate the spec), implement, validate (e.g. using playwright mcp), pre-push checks, push and wait for review, tear down and fast forward main - testcontainers is great for ensuring multiple agents can run tests that don't conflict Seriously I only have one skill that's it. Everything else is in the docs. I'm feeling very productive like this, in a "making good software" sense not a LoC sense.
    • nullbio 4mo ago
      Can you share your skill please?
      • rednb 4mo ago
        That's a big ask. This kind of harness usually contains plenty of proprietary insights about their business. And also, nowadays, a good harness is a major competitive advantage.
• Frannky 4mo ago
  I started using chatgpt for functions and checking, then for single file changes and checking, now for multiple changes and checking. I am at a point where the only changes I correct are architectural. So it may start to become smarter to learn how to see only the architectural directions while multiple agents work, test, and commit both on unit and against live deployment.
• thelucent 4mo ago
  This might work only if you have “infinite” compute and infinite tokens. As someone that used the $20 plan, this pure agentic approach is impossible to do because I’d hit the limit fast and I would end up with less outcome. What I found that work incredibly well was to provide a human written code as reference, and ask it to extend it. So I scaffold the entire thing, architect it, write few samples code (controllers, services, models, components, database schema, how auth works, etc) so the LLM can have a headstart on their attention (pun intended) I usually wrote a stub with a lot of details on how to implement it. Something like a higher abstraction pseudo code. Then ask the LLM to implement it. When it fails, it is often better to undo the whole changes, adjust the stub so it catches what fails before, and try again. Or, commit the changes, and use a new fresh context and only address what went wrong. - Whenever I tried this agentic from scratch approach, I always end up disappointed; both on the outcome and on the limit that I hit before an hour even passed.
  • zuzululu 4mo ago
    You are not going anywhere with $20 plan Upgrade to $200/month and you should see more usage but even for a hardcore user for me, one can never have enough. I'm still very jealous of those guys that got 200x usage simply by RSVP'ing to openai party
• Waffle2180 4mo ago
  [flagged]
• Yokohiii 4mo ago
  > in an agent-first world casual gaslighting
• ajpaulson 4mo ago
  > The diagram below shows the rule: within each business domain (e.g. App Settings), code can only depend “forward” through a fixed set of layers (Types → Config → Repo → Service → Runtime → UI). Cross-cutting concerns (auth, connectors, telemetry, feature flags) enter through a single explicit interface: Providers. Anything else is disallowed and enforced mechanically. Can anyone give me a simplified explanation of what they’re saying here? Having some trouble understanding.
  • iso1337 4mo ago
    IIUC its just strict separation of concerns Eg UI cannot reach down and directly read config files Configs must be only read by (im assuming) a storage interface layer called repo There’s a strict directionality of dependency Somewhat similar to ports and adaptors but presumably more strictly enforced by deterministic linters
  • shepherdjerred 4mo ago
    They're describing a layered architecture enforced by some script in CI. For example, if you had a `backend`, `common`, and `frontend` package, you would be OK having backend/frontend depending on common, but you wouldn't want common depending on backend/frontend or backend/frontend depending on each other. If you think about JavaScript, there is nothing stopping your dependency graph from becoming spaghetti. It sounds like they built static analysis to enforce rules. Some languages have this built in like Java (Project Jigsaw), Go, and Rust. JavaScript, Python, etc. have no such feature. It's really nothing special -- it has existed before. It just becomes a _lot_ more important with agents since they produce a lot of code, and it is good to have lots of static analysis when heavily utilizing agents. They mention this in the article: > This is the kind of architecture you usually postpone until you have hundreds of engineers. With coding agents, it’s an early prerequisite: the constraints are what allows speed without decay or architectural drift.
  • pramodbiligiri 4mo ago
    As other commenters have clarified, it's about layering, separation of concerns etc. Goes by many names. One such terminology here: https://en.wikipedia.org/wiki/Hexagonal_architecture_(software) https://en.wikipedia.org/wiki/Hexagonal_architecture_(softwa.... DI frameworks use terminology like "Provider": https://en.wikipedia.org/wiki/Dependency_injection#Injectors https://en.wikipedia.org/wiki/Dependency_injection#Injectors
• h4ny 4mo ago
  I'm not an AI skeptic but I'm skeptical of the intent of this article. It makes great claims about agent-first engineering and tries to make a real case based on a real product, with real users, and a real team that's been growing — all without even saying what was built or showing it, just like every other AI hype article.
  • epolanski 4mo ago
    And this thread too is filled with users that "I also have done this or that" but bar one user, nobody followed up with any link to anything.
  • zbrock 4mo ago
    At the time we wrote the article we hadn’t released the product and weren’t ready to talk about it. It was an internal prototype that looked very much like the current Codex app.
    • getnormality 4mo ago
      So, did this internal prototype ultimately end up being used to create/influence a real product, e.g. Codex app?
      • zbrock 4mo ago
        Yep!
        • getnormality 4mo ago
          No detail about this in the article or your comment here, but the voluminous lines of code get a big call-out. Very interesting!
• zuzululu 4mo ago
  I think a lot of people are sleeping on the contents of this article. There is still valuable tidbits I'm going to be applying.
• mgaunard 4mo ago
  Isn't this essentially normal AI usage and what everyone has been doing for 6 months?
• shevy-java 4mo ago
  The world is now agent-first already?
• swyx 4mo ago
  we interviewed Ryan here: https://www.latent.space/p/harness-eng https://www.latent.space/p/harness-eng and he gave a talk version of it in london: https://www.youtube.com/watch?v=am_oeAoUhew https://www.youtube.com/watch?v=am_oeAoUhew
  • pramodbiligiri 4mo ago
    This is a great interview! I had to switch from the transcript to watching it though - the transcript had many mistakes.
    • swyx 4mo ago
      ya sorry we upgraded our thing after this
• bigcat12345678 4mo ago
  This matches quite verbatim for my cursor based agentic repo. There isn't anything that were not already experienced and factored into constructs in the repo. And I also find all of the bits created for an effective agentic engineering project, matches perfectly with the main stream engineering best practices. That has been one of my primary reason to all in on agentic engineering, prior to this, applying best practices is always too costly and conflict with teams daily priority.
• aulin 4mo ago
  Dear OpenAI, the target audience of your blog or at least of this blog post understands English pretty well. Why won't you give them a simple way to disable the shitty ai translation and read the original content? Why translate it at all in the first place? EDIT: found the button, all the way down in the bottom of the page... I hate this so much, give me the original content, I will decide if and when I need translation
• nullbio 4mo ago
  Step 1: Be rich.
• spacebacon 4mo ago
  Leveraging a better way. No last mile. https://github.com/space-bacon/SRT https://github.com/space-bacon/SRT
• simonbarker87 4mo ago
  I’d be interested to know two things: 1. What’s the job satisfaction like day to day being an engineer on this project? How have they adapted to this way of working? 2. How much did it cost? Work is being done whilst the engineers sleep but if that 6 hours overnight task cost $300 and could have been done by a person in 2 hours is it a real saving?
  • epolanski 4mo ago
    > 1. What’s the job satisfaction like day to day being an engineer on this project? How have they adapted to this way of working? The job satisfaction is looking at the bank account every time you feel your job sucks.
• mohsen1 4mo ago
  I've been doing the same experiment in tsz[1] for a while now (the same past five months in fact) and I have come to very similar conclusions. Lots of harness to enforce good architecture splits. Lots of tests and CI. My point of working on tsz is to learn how to do very big projects with AI. Eventually the same workflows and attitude can be leveraged to build customer product apps with UI as well. I see that OpenAI is leveraging automated browser testing and even videos as part of their workflow. I think as models get better this direction for making software would eventually make sense. I don't think we're there yet though. But at least, unlike OpenAI vague claims I can share the output with you to see! Most of the solutions that offer a very high level of automation like Lovable are a bit too optimistic and solutions are not tightly coupled with lots of automated testing. [1] https://github.com/tsz-org/tsz https://github.com/tsz-org/tsz
• eddysir 4mo ago
  [flagged]
• witx 4mo ago
  "Engineering" These people are so delusional it feels like a mental desease by now. I really hope no one gets hurt by all this slop code in the future by these wanna be engineers.
• janpeuker 4mo ago
  I find it so interesting "Agent legibility is the goal" picks up James C. Scott term (without defining it, so I assume that's what they mean) which is _not a good thing_. Legibility is a governance effort to box in life.
  • williamcotton 4mo ago
    Legibility has a time and place, for example, global health data around pandemics.
• Hazemamirrr 4mo ago
  [flagged]
• xyzal 4mo ago
  Given that we can code at 10x speed for at least half a year, one would expect to see at least some pieces of machine-created software with 5 years' worth of equivalent human engineering work. Anyone know some?
• epolanski 4mo ago
  I am not much understanding the naysayers here. I will do a premise: I don't like where software engineering is heading, at all. I have never been unhappier to work in this field since AI came out. And no, it is not possible to opt out of AI, especially when your teammates are all great engineers whose productivity increased a lot without any drops in quality code-wise (in fact the opposite has happened). You need to keep up. But it's tiring and the fun/interesting parts are disappearing. That being said, it's clear that harness engineering is the most important part of our job and that task is going to take increasingly more of our time. And thus having a glimpse of how an AI company handles it is by any means interesting.
• egorferber 4mo ago
  Yep thats true pre grounding is very much worth it, if you just feed the agent a quick environment brief upront instead of making it spam tool calls to figure out where it is, you save a lot of tokens.
• patdoli 4mo ago
  > We tried this early on — used ChatGPT as "project manager" to set up the entire harness before writing any code. After a week it produced 140+ docs of rules, architecture, frameworks. Zero lines of code. When we finally brought in another tool to review, the verdict was: "a perfectly secure empty safe." The harness was immaculate. There was just nothing inside it. > > Harness matters, but if you're not shipping code alongside it, you're just writing fiction.
• deleted 4mo ago
  [deleted]
• zbrock 4mo ago
  Hello! I’m one of the three engineers who write this piece. Happy to answer questions.
  • aabdi 4mo ago
    Very cool article! - are other teams adopting this approach? What’s the blockers if not? - have there been problems where the models alone were not enough to debug and the devs had to fix it themselves? - as the rate of changes has increased with more devs how have you dealt with concurrent writers with merge conflicts? - if there was anything you could change in the approach you started with, what would it be?
    • zbrock 4mo ago
      1. Yes! Many teams internally have adopted a lot of the same practices we outlined in the blog post. Ryan has also been spending time both internally and externally helping companies figure out how to do this in their code bases. 2. Hmm, kind of. There have definitely been issues the models can’t one shot. But we still use Codex to write all the actual code with human guidance. 3. More agents :) Some teams are experimenting with centralized Agent mediated integration queues, others use normal merge queues, many have local Codex threads that monitor CI to resolve and land conflicts or failures. 4. Today’s models and codex app. We started doing all this with gpt-5 and codex-cli. The tools today, 9 months later, are so much better than what we had then.
      • HorizonXP 4mo ago
        Have you built any tooling or products around all of this and deploying it somehow? I’d love to learn more and share notes, because we’ve been doing this too. About 3100+ PRs merged across our 4 person team in 4 months. Impossible without harness engineering, and I agree, the tools are getting even better.
  • s3p 4mo ago
    Were those em dashes you, or GPT
  • 4mo ago
• osigurdson 4mo ago
  They will have to open source it. Otherwise it is impossible for anyone outside of OAI to gain any insights - basically just a Boris at this point.
• mwkaufma 4mo ago
  Another breathless sales pitch selling pickaxes to miners, but where's the gold? Where's the incredible product that the chatbots-talking-to-chatbots over git generating LOC heaps have actually _created_? I just don't see it.
• skulquake 4mo ago
  Thanks for pointing this article out, it really helped me in optimizing a pattern I'm developing to ensure agent context windows don't get bloated over time.
• DenisM 4mo ago
  Notably the feed metrics and traces into the agent. I never thought of doing that.
• advertum 4mo ago
  LOC was never really the point, if anything it's often the opposite. Being able to say the same thing more concisely makes code cheaper to maintain and easier to understand. (Not always: sometimes people are concise just for its own sake, and that's worse.) Either way, that whole debate feels like the past now. As for metrics based on how much was produced or spent, like companies measuring developer performance by tokens used, that's a dead end regardless. Performance should be measured by outcomes: incident/failure rate, SLA, user numbers and feedback, revenue, that kind of thing.
• Pranavsingh431 4mo ago
  [flagged]