9 ms·

The new calculus of AI-based coding

205 points by todsacerdoti 1y ago

Madmallard 1y ago
first the Microsoft guy touting agents now AWS guy doing it ! "My team is no different—we are producing code at 10x of typical high-velocity team. That's not hyperbole - we've actually collected and analyzed the metrics." Rofl "The Cost-Benefit Rebalance" In here he basically just talks about setting up mock dependencies and introducing intermittent failures into them. Mock dependencies have been around for decades, nothing new here. It sounds like this test system you set up is as time consuming as solving the actual problems you're trying to solve, so what time are you saving? "Driving Fast Requires Tighter Feedback Loop" Yes if you're code-vomiting with agents and your test infrastructure isn't rock solid things will fall apart fast, that's obvious. But setting up a rock solid test infrastructure for your system involves basically solving most of the hard problems in the first place. So again, what? What value are you gaining here? "The communication bottleneck" Amazon was doing this when I worked there 12 years ago. We all sat in the same room. "The gains are real - our team's 10x throughput increase isn't theoretical, it's measurable." Show the data and proof. Doubt. Yeah I don't know. This reads like complete nonsense honestly. Paraphrasing: "AI will give us huge gains, and we're already seeing it. But our pipelines and testing will need to be way stronger to withstand the massive increase in velocity!" Velocity to do what? What are you guys even doing? Amazon is firing 30,000 people by the way.
- p1necone 1y ago
  "Our testing needs to be better to handle all this increased velocity" reads to me like a euphemistic way of saying "we've 10x'ed the amount of broken garbage we're producing".
- lispisok 1y ago
  We're back to using LOC as a productivity metric because LLMs are best at cranking out thousands of LOC really fast. Personal experience I had a colleague use Claude Code top create a PR consisting of a dozen files and thousands of line of code for something that could have been done in a couple hundred LOC in a single file.
  - CharlesW 1y ago
    > We're back to using LOC as a productivity metric because LLMs are best at cranking out thousands of LOC really fast. Can you point me to anyone who knows what they're talking about declaring that LOC is the best productivity metric for AI-assisted software development?
    - chipsrafferty 1y ago
      Are you implying that the author of this article doesn't know what they are talking about? Because they basically declared it in the article we just read. Can you point me to where the author of this article gives any proof to the claim of 10x increased productivity other than the screenshot of their git commits, which shows more squares in recent weeks? I know git commits could be net deleting code rather than adding code, but that's still using LOC, or number of commits as a proxy to it, as a metric.
      - CharlesW 1y ago
        > I know git commits could be net deleting code rather than adding code… Yes, I'm also reading that the author believes commit velocity is one reflection of the productivity increases they're seeing, but I assume they're not a moron and has access to many other signals they're not sharing with us. Probably stuff like: https://www.amazon.science/blog/measuring-the-effectiveness-of-software-development-tools-and-practices https://www.amazon.science/blog/measuring-the-effectiveness-...
  - yahoozoo 1y ago
    I had a coworker use Copilot to implement tab indexing through a Material UI DataGrid. The code was a few hundred lines. I showed them a way to do it in literally one line passed in the slot properties.
    - shurane 1y ago
      Man hearing that depresses me. Some people really overcomplicate things.
- blibble 1y ago
  if you've ever had a friend that you knew before, then they went to work at amazon, it's like watching someone get indoctrinated into a cult and this guy didn't survive there for a decade by challenging it
skinnymuch 1y ago
Interesting enough to me though I only skimmed. I switched back to Rails for my side project a month ago and ai coding when doing not too complex stuff has been great. While the old NextJS code base was in shambles. Before I was still doing a good chunk of the NextJS coding. I’m probably going to be directly coding less than 10% of the code base from here on out. I’m now spending time trying to automate things as much as possible, make my workflow better, and see what things can be coded without me in the loop. The stuff I’m talking about is basic CRUD and scraping/crawling. For serious coding, I’d think coding yourself and having ai as your pair programmer is still the way to go.
gachaprize 1y ago
Classic LLM article: 1) Abstract data showing an increase in "productivity" ... CHECK 2) Completely lacking in any information on what was built with that "productivity" ... CHECK Hilarious to read this on the backend of the most widely publicized AWS failure.
- alfalfasprout 1y ago
  Yep. The problem is then leadership sees this and says "oh, we too can expect 10x productivity if everyone uses these tools. We'll force people to use them or else." And guess what happens? Reality doesn't match expectations and everyone ends up miserable. Good engineering orgs should have engineers deciding what tools are appropriate based on what they're trying to do.
- gtsop 1y ago
  [dead]
- tomhow 1y ago
  Please don't post shallow, snarky dismissals on HN. The guidelines ask us to be more thoughtful in the way we respond to things: https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newsguidelines.html
  - Dilettante_ 1y ago
    GP is making very specific points of contention about the article.
    - tomhow 1y ago
      It's the first ever comment by that account, and it seems to contain fulmination and snark, both of which are explicitly against the guidelines, as is curmudgeonliness. Criticism is fine, but it needs to be substantive. If the article isn't fit for HN, people can flag it, and if they really want to, they're welcome to email us at hn@ycombinator.com to point out what's wrong with it and we'll consider further penalties. Comments like the one I replied too just make HN seem mean and miserable and that's definitely something we're trying to avoid.
Animats 1y ago
> Instead, we use an approach where a human and AI agent collaborate to produce the code changes. For our team, every commit has an engineer's name attached to it, and that engineer ultimately needs to review and stand behind the code. We use steering rules to setup constraints for how the AI agent should operate within our codebase, This sounds a lot like Tesla's Fake Self Driving. It self drives right up to the crash, then the user is blamed.
- groby_b 1y ago
  Except here it's made abundantly clear, up front, who has responsibility. There's no pretense that it's fully self driving. And the engineer has the power to modify every bit of that decision. Part of being a mature engineer is knowing when to use which tools, and accepting responsibility for your decisions. It's not that different from collaborating with a junior engineer. This one can just churn out a lot more code, and has occasional flashes of brilliance, and occasional flashes of inanity.
  - happyPersonR 1y ago
    Idk it’s hard to say it’s called “Full Self Driving” and then the CEO says as much.
  - Animats 1y ago
    > Except here it's made abundantly clear, up front, who has responsibility. By the people who are disclaiming it, yes.
exasperaited 1y ago
Absolutely none of that article has ever even so much as brushed past the colloquial definition of "calculus". These guys actually seem rattled now.
- deleted 1y ago
  [deleted]
- photochemsyn 1y ago
  Well, 'calculus' is the kind of marketing word that sounds more impressive than 'arithmetic' and I think 'quantum logic' has gone a bit stale, and 'AI-based' might give more hope to the anxious investor class, as 'AI-assisted' is a bit weak as it means the core developer team isn't going to be cut from the labor costs on the balance sheet, they're just going to be 'assisted' (things like AI-written unit tests that still need some checking). "The Arithmetic of AI-Assisted Coding Looks Marginal" would be the more honest article title.
  - collingreen 1y ago
    "Galaxy-brain pair programming with the next superintelligence"
  - photonthug 1y ago
    Yes, unfortunately a phrase that's used in an attempt to lend gravitas and/or intimidate people. It sort of vaguely indicates "a complex process you wouldn't be interested in and couldn't possibly understand". At the same time it attempts to disarm any accusation of bias in advance by hinting at purely mechanistic procedures. Could be the other way around, but I think marketing-speak is taking cues here from legal-ese and especially the US supreme court, where it's frequently used by the justices. They love to talk about "ethical calculus" and the "calculus of stare decisis" as if they were following any rigorous process or believed in precedent if it's not convenient. New translation from original Latin: "we do what we want and do not intend to explain". Calculus, huh? Show your work and point to a real procedure or STFU
philipp-gayret 1y ago
This is the first time I see "steering rules" mentioned. I do something similar with Claude, curious how it looks for them and how they integrate it with Q/Kiro.
- manmal 1y ago
  Those rules are often ignored by agents. Codex is known to be quite adhering, but it falls back to its own ideas, which run counter to rules I‘ve given it. The longer a session goes on, the more it goes off the rails.
  - CharlesW 1y ago
    Everything related to LLMs is probabilistic, but those rules are also often followed well by agents.
    - manmal 1y ago
      Yes they do, most of the time. Then they don’t. Yesterday, I told codex that it must always run tests by invoking a make target. That target is even configurable w/ parameters, eg to filter by test name. But always, at some point in the session, codex started disregarding that rule and fell back to using the platform native test tool directly. I used strong language to steer it back, but 20% or so of context later, it did that again.
      - Dilettante_ 1y ago
        Once the LLM has made one mistake, it's often best to start a new context. Since its mechanism is to predict the next token of the conversation, it's reasonable to "predict" itself making more mistakes once it has made one.
        manmal 1y ago
        I‘m not sure this is still the case with codex. In this instance, restarting had no strong effect.
whiterook6 1y ago
This reads like "Hey, we're not vibe coding, but when we do, we're careful!" with hints of "AI coding changes the costs associated with writing code, designing features, and refactoring" sprinkles in to stand out.
moron4hire 1y ago
If you are producing real results at 10x then you should be able to show that you are a year ahead of schedule in 5 weeks. Waiting to see anyone show even a month ahead of schedule after 6 months.
- __MatrixMan__ 1y ago
  I've never worked anywhere that knew where they were going well enough that it was even possible to be a month ahead of schedule. By the time a month has elapsed the plan is entirely different. AI can't keep up because its context window is full of yesteryear's wrong ideas about what next month will look like.
  - moron4hire 1y ago
    Yeah, this is the main problem. Writing of code just isn't the bottle neck. It's the discovery of the business case that is the hard part. And if you don't know what it is, you can't prompt your way out of it. We've been having a go around with corporate leadership at my company about "AI is going to solve our problems". Dude, you don't even know what our problems are. How are you going to prompt the AI to analyze a 300 page PDF on budget policy when you can't even tell me how you read a 300 page PDF with your eyes to analyze the budget policy. I'm tempted to give them what they want: just a chatter box they can ask, "analyze this budget policy for me", just so I can see the looks on their faces when it spits out five poorly written paragraphs full of niceties that talk its way around ever doing any analysis. I don't know, maybe I'm too much of a perfectionist. Maybe I'm the problem because I value getting the right answer rather than just spitting out reams of text nobody is ever going to read anyway. Maybe it's better to send the client a bill and hope they are using their own AIs to evaluate the work rather than reading it themselves? Who would ever think we were intentionally engaging in Fraud, Waste, and Abuse if it was the AI that did it?
    - __MatrixMan__ 1y ago
      I've got my own gripes about leadership, but I'm finding that even when its a goal I've set for myself, watching an AI fail at it represents a refinement of what I thought I wanted: I'm not much better than they are. That, or its a discovery of why what I wanted is impossible and it's back to the drawing board. It's nice to not be throwing away code that I'd otherwise have been a perfectionist about (and still thrown away).
reenorap 1y ago
No. The way to code going forward with AI is Test Driven Development. The code itself no longer matters. You give the AI a set of requirements, ie. tests that need to pass, and then let it code whatever way it needs to in order to fulfill those requirements. That's it. The new reality us programmers need to face is that code itself has an exact value of $0. That's because AI can generate it, and with every new iteration of the AI, the internal code will get better. What matters now are the prompts. I always thought TDD was garbage, but now with AI it's the only thing that makes sense. The code itself doesn't matter at all, the only thing that matters is the tests that will prove to the AI that their code is good enough. It can be dogshit code but if it passes all the tests, then it's "good enough". Then, just wait a few months and then rerun the code generation with a new version of the AI and the code will be better. The humans don't need to know what the code actually is. If they find a bug, write a new test and force the AI to rewrite the code to include the new test. I think TDD has really found its future now that AI coding is here to stay. Human code doesn't matter anymore and in fact I would wager that modifying AI generated code is as bad and a burden. We will need to make sure the test cases are accurate and describe what the AI needs to generate, but that's it.
- pcarolan 1y ago
  I mostly agree, but why stop at tests? Shouldn’t it be spec driven development? Then neither the code or the language matter. Wouldn’t user stories and requirements à la bdd (see cucumber) be the right abstraction?
  - reenorap 1y ago
    I don't think you're wrong but I feel like there's a big bridge between the spec and the code. I think the tests are the part that will be able to give the AI enough context to "get it right" quicker. It's sort of like a director telling an AI the high level plot of a movie, vs giving an AI the actual storyboards. The storyboards will better capture the vision of the director vs just a high level plot description, in my opinion.
  - __MatrixMan__ 1y ago
    Maybe one day. I find myself doing plenty of course correction at the test level. Safely zooming out doesn't feel imminent.
brazukadev 1y ago
But here's the critical part: the quality of what you are creating is way lower than you think, just like AI-written blog posts.
- collingreen 1y ago
  Upvoted for dig that is also an accurate and insightful metaphor.
cadamsdotcom 1y ago
"We have real mock versions of all our dependencies!" Congratulations, you invented end-to-end testing. "We have yellow flags when the build breaks!" Congratulations! You invented backpressure. Every team has different needs and path dependencies, so settles on a different interpretation of CI/CD and software eng process. Productizing anything in this space is going to be an uphill battle to yank away teams' hard-earned processes. Productizing process is hard but it's been done before! When paired with a LOT of spruiking it can really progress the field. It's how we got the first CI/CD tools (eg. https://en.wikipedia.org/wiki/CruiseControl https://en.wikipedia.org/wiki/CruiseControl) and testing libraries (eg. pytest) So I wish you luck!
r0x0r007 1y ago
"For me, roughly 80% of the code I commit these days is written by the AI agent" Therefore, it is not commited by you, but by you in the name of AI agent and the holy slop. What to say, I hope that 100x productivity is worth it and you are making tons of money. If this stuff becomes mainstream, I suggest open source developers stop doing the grind part, stop writing and maintaining cool libraries and just leave all to the productivity guys, let's see how far they get. Maybe I've seen too many 1000x hacker news..
- visarga 1y ago
  Just need the feedback to follow suit to be 100x as effective. Tests, docs and rapid loops of guidance with human in the loop. Split your tasks, find the structure that works.
- ChadNauseam 1y ago
  I think it's fine. For example, "I" made this library https://github.com/anchpop/weblocks https://github.com/anchpop/weblocks . It might be more accurate to say that I directed AI to make it, because I didn't write a line of code myself. (And I looked at the code and it is truly terrible.) But I tested that it works, and it does, and it solves my problem perfectly. Yes, it is slop, but this is a leaf node in the abstraction graph, and no one needs to look at it again now that it it written
  - deleted 1y ago
    [deleted]
  - shakna 1y ago
    Most code, though, is not write once and ignore. So it does matter if its crap, because every piece of software is only as good as its least dependency. Fine for just you. Not fine for others, not fine for business, not fine the moment you star count starts moving.
deleted 1y ago
[deleted]
bcrosby95 1y ago
It's amazing that their metrics exactly match the mythical "10x engineer" in productivity boost.
- Naklin 1y ago
  Without disclosing said metrics or any data... This is really something striking to me about all these AI productivity claims. They never provide the methodology and data.
  - quikoa 1y ago
    They rarely even provide the projects either or even the type of project. I'd like to see all these awesome results that are build with AI (preferably not web related).
Madmallard 1y ago
Lots of reasonable criticisms being down-voted here. Are we being AstroTurfed? Is HN falling victim to the AI hype train money too now?
- nl 1y ago
  I'm downvoting most of the criticisms because in general they can be summarized as "all AI is slop". In my experience that simply isn't true. This article attempted to outline a fairly reasonable approach to using AI tooling, and the criticisms hardly seem related to it at all.
StilesCrisis 1y ago
The biggest thing that stood out to me was that they suddenly started working nonstop, even on weekends…? If AI is so great, why can’t they get a single day off in two months?
Madmallard 1y ago
Correct TDD involves solving all the hard problems in the process. What gain does AI give you then?
Lucian6 1y ago
[dead]
zeroq 1y ago
When Karpathy wrote Software 2.0 I was super excited. I naively believed that we'll start building black boxes based on requirements, sets of inputs and outputs, and sudden changes of heart from stakeholders that often happen on a daily basis for many of us and mandates almost complete reimagination of project architecture will simply need another pass of training with new parameters. Instead the mainstream is pushing hard reality where we mass produce a ton of code until it starts to work within guard rails. Does it really work? Is it maintainable? Get out of here. We're moving at 200mph.
- rob_c 1y ago
  How tf else did you honestly expect black-boxes to get built, by self-mangling machine code spit out by a sentient AI god? Karpathy is bullish on everything bleeding edge, and unfortunately it kinda shows when you know the material better than he does. (source, I've been lecturing on all of it for a few years now). I'm not saying this is bad. It's great to see people who are engaging and bullish, it's better than most futurists waving their hands and going "something, something warp drive". But when you take a step back and really ask what is going on behind the scenes, all we have is massive statistical tools performing neato tricks at statistical probability to predict patterns. There's no greater understanding or ability to learn or mimic. YET. The transformer for-instance can't easily learn complex mathematical operations. There's a google paper on "learning" multiplication and I know people working on building networks to "learn" sin/cos from scratch. But given these basic limitations and pretty much, every, single, paper, out of Apple "intelligence" crapping on the buzz. We've pretty much hit a limit beyond being the first company to allow for multi-trillion token parsing (or basic, limited, token parsing memory) for companies to capture and retrieve information.
  - adammarples 1y ago
    Do you really think he knows that little? I mean fair enough you've been lecturing on it, but he was lecturing a decade ago, at Stanford. Then he took a little break to you know, run AI at Tesla...
    - 1y ago
tcherasaro 1y ago
Another day, and another smart person finally discovers the benefits of leveraging AI to write code.
sherinjosephroy 1y ago
[flagged]
ang_cire 1y ago
As a security researcher, I am both salivating at the potential that the proliferation of TDD and other AI-centric "development" brings for me, and scared for IT at the same time. Before we just had code that devs don't know how to build securely. Now we'll have code that the devs don't even know what it's doing internally. Someone found a critical RCE in your code? Good luck learning your own codebase starting now! "Oh, but we'll just ask AI to write it again, and the code will (maybe) be different enough that the exact same vuln won't work anymore!" <- some person who is going to be updating their resume soon. I'm going to repurpose the term, and start calling AI-coding "de-dev".
- zeroq 1y ago
  Just few days ago I spoke with sec guy who was telling me how frustrating it is to validate AI code. The problem is marketing. Cycling industry is akin to audiophiles and will swear on their lives that $15,000 bicycle is the pinnacle of human engineering. This year's bike will go 11% faster than the previous model. But if you read last 10 years of marketing materials and do math it should basically ride itself. There's so much money in AI right now that you can't really expect anyone to say "well, we had hopes, but it doesn't really work the way we expected". Instead you have pitch after pitch, masses parroting CEOs, and everyone wants to get a seat on the hype train. It's easy to dispel audiophiles or carbon enthusiasts but it's not so easy with AI, because no one really knows how it works. OpenAI released a paper in which they stated, sorry for paraphrasing, "we did this, we did that, and we don't know why results were different".
- nosianu 1y ago
  > Now we'll have code that the devs don't even know what it's doing internally. I think that has already been true for some time for large projects continuously updated over a long time, and lots of developers entering and leaving the project throughout the years because nobody who has a choice wants to do that demoralizing job for long (I was one of them in the 1990s, the job was later given to an Indian H1B who could not switch to something better easily, not before putting in a few years of torture to have a better resume, and possibly a greencard). Most famous post here, but I would like to see what e.g. Microsoft's devs would have to say, or Adobe's: https://news.ycombinator.com/item?id=18442941 https://news.ycombinator.com/item?id=18442941 Such code has long been held together by the extensive test suites rather than intimate knowledge of how it all works. The task of the individual developer is to close bug tickets and add features, not to produce an optimal solution, or even refactoring. They long ago gave up on that as taking too long.
zbyforgotp 1y ago
TLDR: ai changes the economic calculus of software development. It makes automated testing more beneficial in comparison to costs. I think he is right.
keeda 1y ago
This article is right, but I think it may underplay the changes that could be coming soon. For instance, as the top comment here about TDD points out, the actual code does not matter anymore. This is an astounding claim! And it has naturally received a lot of objections in the replies. But I think the objections can mostly be overcome with a minor adjustment: You only need to couple TDD with a functional programming style. Functional programming lets you tightly control the context of each coding task, which makes AI models ridiculously good at generating the right code. Given that, if most of your code is tightly-scoped, well-tested components implementing orthogonal functionality, the actual code within those components will not matter. Only glue code becomes important and that too could become much more amenable to extensive integration testing. At that point, even the test code may not matter much, just the test-cases. So as a developer you would only really need to review and tweak the test cases. I call this "Test-Case-Only Development" (TCOD?) The actual code can be completely abstracted away, and your main task becomes design and architecture. It's not obvious this could work, largely because it violates every professional instinct we have. But apparently somebody has even already tried it with some success: https://www.linkedin.com/feed/update/urn:li:activity:7196786658942291969/ https://www.linkedin.com/feed/update/urn:li:activity:7196786... All the downsides that have been mentioned will be true, but also may not matter anymore. E.g. in a large team and large codebase, this will lead to a lot of duplicate code with low cohesion. However, if that code does what it is supposed to and is well-tested, does the duplication matter? DRY was an important principle when the cost of code was high, and so you wanted to have as much leverage as possible via reuse. You also wanted to minimize code because it is a liability (bugs, tech debt, etc.) and testing, which required even more code that still didn't guarantee lack of bugs, was also very expensive. But now that the cost of code is plummeting, that calculus is shifting too. You can churn out code and tests (including even performance tests, which are always an afterthought, if thought of at all) at unimaginable rates. And all this while reducing the dependencies of developers on libraries and frameworks and each other. Fewer dependencies means higher velocity. The overall code "goodput" will likely vastly outweight inefficiences like duplication. Unfortunately, as TFA indicates, there is a huge impedance mismatch with this and the architectures (e.g. most code is OO, not functional), frameworks, and processes we have today. Companies will have to make tough decisions about where they are and where they want to get. I suspect AI-assisted coding taken to its logical conclusion is going to look very different from what we're used to.
rob_c 1y ago
Trust but verify. It's not hard. The corollary being. If you can't (through skill or effort) verify don't trust. If you break this pattern you deserve all the follies that become you as a "professional".
zkmon 1y ago
Is it exciting because work happens at 200mph or is it because you get that much business advantage against your competition? Or is it because, now it allows you to spend only one hour at work per day? To quote Joey from Friends - "400 bucks are gone from my pocket and nobody is getting happier?"
Zanfa 1y ago
IMO the biggest issue with AI code is that writing code is the easiest part of software development. Reviewing code is so much more difficult than writing it, even more so if you're not already intimately familiar with it in the first place. It's like with AI images, where they look plausible at first, but then you start noticing all the little things that are off in the sidelines.
- Dilettante_ 1y ago
  writing code is the easiest part of software development. Reviewing code is so much more difficult than writing it A lot of people say this, and I do not doubt that it is fully true in their real experience. But it is not necessarily the only way for things to be. If more time and effort were put into writing code which is easier to review, the difficulty of writing it would increase and the difficulty of reading it would decrease, flipping that equation. The incentives just aren't like that. It doesn't pay to maximize readability against time spent writing: Not every line will have to be reviewed, and not every line that has to be reviewed will be so complex that readability needs to be perfect to be maintainable.
  - oblio 1y ago
    The problem with AI is that those incentives wrong incentives are taken to 10000x. And regarding "not every line will have to be reviewed, and not every line that has to be reviewed will be so complex that readability needs to be perfect to be maintainable.", the problem with AI is that code becomes basically unknowable. Which is fine if everything that is built is slop, but many things aren't slop. Stuff that touches money, healthcare, personal relationships, etc you know, the things that matter in life, risks all turning into slop, which <will> have real life consequences. We'll start seeing this in a few years.
  - Zanfa 1y ago
    It's not the code itself that makes review difficult. Even the best written code can be difficult to review. The complexity of effective code review arises from the fact that you need to understand the domain to evaluate correctness of both the code itself and the tests covering it.
Simplita 1y ago
[dead]
jcgrillo 1y ago
> For our team, every commit has an engineer's name attached to it, and that engineer ultimately needs to review and stand behind the code. Then they claim (and demonstrate with a picture of a commits/day chart) a team-wide 10x throughput increase. I claim there's got to be a lot of rubber-stamp reviewing going on here. It may help to challenge the "author" to explain things like "why does this lifetime have the scope it does?" or "why did you factor it this way instead of some other way?" e.g. questions which force them to defend the "decisions" they made. I suspect if you're actually doing thorough reviews that the velocity will actually decrease instead of increase using LLMs.
jmull 1y ago
> Over the past three months... [we] have been building something really cool The claim is a fast-moving, high performing team has become a 10x fast moving, high-performing team. That's equivalent to 2-1/2 years of development across a team. Shall we expect the tangible results soon? I'm perfectly willing to accept that AI coding will make us all a lot more productive, but I need to see the results.
- ghm2180 1y ago
  And at that rate we should see a FAANGMULA like company launch in 10x or at least 5x less the time. Right?
  - oblio 1y ago
    Programming was rarely the barrier to building these types of companies. I know software people don't want to accept that, but it's almost always something on the business or administrative/management side of things. Even for the programming bits, if your initial programmers suck (for some reason) but you have money, a great management team would just replace them with better programmers and fix the code mess with their help. So even that isn't a programming problem, it's a management problem. And let's look at Twitter, who had atrocious code early on (fail whale galore), yet managed to make a profitable business due to amazing product market fit, despite management incompetence. Companies just need to pass a code quality bar which is much, much, much lower than the bar programmers set.
- dgemm 1y ago
  > AI coding will make us all I'm willing to believe it will make high-judgement autonomous people more productive, I'm less sure it will scale to everyone. The author is one of the senior-most technical staff at AWS.
- nguyendat 1y ago
  Could you help share little bit more detail about your team experimental result, e.g: new or legacy code base, how do you team measure/control quality. Thanks
Gud 1y ago
People bemoan unrecognisable code bases, maybe that is true. I’ve sure used various LLMs to solve difficult nuts to crack. Problems I have been able to verbalise, but unable to solve. Chances are that if you are using an LLM to mass produce boiler, you are writing too much boiler.
highfrequency 1y ago
> When your throughput increases by an order of magnitude, you're not just writing more code - you're making more decisions. > These aren't just implementation details - they're architectural choices that ripple through the codebase. > The gains are real - our team's 10x throughput increase isn't theoretical, it's measurable. Enjoyed the article and the points it brought up. I do find it uncanny that this article about the merits and challenges of AI coding was likely written by ChatGPT.
deterministic 1y ago
90% of the code I commit is code generated. No AI required.

205 points by todsacerdoti 1y ago

first the Microsoft guy touting agents now AWS guy doing it ! "My team is no different—we are producing code at 10x of typical high-velocity team. That's not hyperbole - we've actually collected and analyzed the metrics." Rofl "The Cost-Benefit Rebalance" In here he basically just talks about setting up mock dependencies and introducing intermittent failures into them. Mock dependencies have been around for decades, nothing new here. It sounds like this test system you set up is as time consuming as solving the actual problems you're trying to solve, so what time are you saving? "Driving Fast Requires Tighter Feedback Loop" Yes if you're code-vomiting with agents and your test infrastructure isn't rock solid things will fall apart fast, that's obvious. But setting up a rock solid test infrastructure for your system involves basically solving most of the hard problems in the first place. So again, what? What value are you gaining here? "The communication bottleneck" Amazon was doing this when I worked there 12 years ago. We all sat in the same room. "The gains are real - our team's 10x throughput increase isn't theoretical, it's measurable." Show the data and proof. Doubt. Yeah I don't know. This reads like complete nonsense honestly. Paraphrasing: "AI will give us huge gains, and we're already seeing it. But our pipelines and testing will need to be way stronger to withstand the massive increase in velocity!" Velocity to do what? What are you guys even doing? Amazon is firing 30,000 people by the way.

p1necone 1y ago
"Our testing needs to be better to handle all this increased velocity" reads to me like a euphemistic way of saying "we've 10x'ed the amount of broken garbage we're producing".
lispisok 1y ago
We're back to using LOC as a productivity metric because LLMs are best at cranking out thousands of LOC really fast. Personal experience I had a colleague use Claude Code top create a PR consisting of a dozen files and thousands of line of code for something that could have been done in a couple hundred LOC in a single file.
- CharlesW 1y ago
  > We're back to using LOC as a productivity metric because LLMs are best at cranking out thousands of LOC really fast. Can you point me to anyone who knows what they're talking about declaring that LOC is the best productivity metric for AI-assisted software development?
  - chipsrafferty 1y ago
    Are you implying that the author of this article doesn't know what they are talking about? Because they basically declared it in the article we just read. Can you point me to where the author of this article gives any proof to the claim of 10x increased productivity other than the screenshot of their git commits, which shows more squares in recent weeks? I know git commits could be net deleting code rather than adding code, but that's still using LOC, or number of commits as a proxy to it, as a metric.
    - CharlesW 1y ago
      > I know git commits could be net deleting code rather than adding code… Yes, I'm also reading that the author believes commit velocity is one reflection of the productivity increases they're seeing, but I assume they're not a moron and has access to many other signals they're not sharing with us. Probably stuff like: https://www.amazon.science/blog/measuring-the-effectiveness-of-software-development-tools-and-practices https://www.amazon.science/blog/measuring-the-effectiveness-...
- yahoozoo 1y ago
  I had a coworker use Copilot to implement tab indexing through a Material UI DataGrid. The code was a few hundred lines. I showed them a way to do it in literally one line passed in the slot properties.
  - shurane 1y ago
    Man hearing that depresses me. Some people really overcomplicate things.
blibble 1y ago
if you've ever had a friend that you knew before, then they went to work at amazon, it's like watching someone get indoctrinated into a cult and this guy didn't survive there for a decade by challenging it

skinnymuch 1y ago

Interesting enough to me though I only skimmed. I switched back to Rails for my side project a month ago and ai coding when doing not too complex stuff has been great. While the old NextJS code base was in shambles. Before I was still doing a good chunk of the NextJS coding. I’m probably going to be directly coding less than 10% of the code base from here on out. I’m now spending time trying to automate things as much as possible, make my workflow better, and see what things can be coded without me in the loop. The stuff I’m talking about is basic CRUD and scraping/crawling. For serious coding, I’d think coding yourself and having ai as your pair programmer is still the way to go.

gachaprize 1y ago

Classic LLM article: 1) Abstract data showing an increase in "productivity" ... CHECK 2) Completely lacking in any information on what was built with that "productivity" ... CHECK Hilarious to read this on the backend of the most widely publicized AWS failure.

alfalfasprout 1y ago
Yep. The problem is then leadership sees this and says "oh, we too can expect 10x productivity if everyone uses these tools. We'll force people to use them or else." And guess what happens? Reality doesn't match expectations and everyone ends up miserable. Good engineering orgs should have engineers deciding what tools are appropriate based on what they're trying to do.
gtsop 1y ago
[dead]
tomhow 1y ago
Please don't post shallow, snarky dismissals on HN. The guidelines ask us to be more thoughtful in the way we respond to things: https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newsguidelines.html
- Dilettante_ 1y ago
  GP is making very specific points of contention about the article.
  - tomhow 1y ago
    It's the first ever comment by that account, and it seems to contain fulmination and snark, both of which are explicitly against the guidelines, as is curmudgeonliness. Criticism is fine, but it needs to be substantive. If the article isn't fit for HN, people can flag it, and if they really want to, they're welcome to email us at hn@ycombinator.com to point out what's wrong with it and we'll consider further penalties. Comments like the one I replied too just make HN seem mean and miserable and that's definitely something we're trying to avoid.

Animats 1y ago

> Instead, we use an approach where a human and AI agent collaborate to produce the code changes. For our team, every commit has an engineer's name attached to it, and that engineer ultimately needs to review and stand behind the code. We use steering rules to setup constraints for how the AI agent should operate within our codebase, This sounds a lot like Tesla's Fake Self Driving. It self drives right up to the crash, then the user is blamed.

groby_b 1y ago
Except here it's made abundantly clear, up front, who has responsibility. There's no pretense that it's fully self driving. And the engineer has the power to modify every bit of that decision. Part of being a mature engineer is knowing when to use which tools, and accepting responsibility for your decisions. It's not that different from collaborating with a junior engineer. This one can just churn out a lot more code, and has occasional flashes of brilliance, and occasional flashes of inanity.
- happyPersonR 1y ago
  Idk it’s hard to say it’s called “Full Self Driving” and then the CEO says as much.
- Animats 1y ago
  > Except here it's made abundantly clear, up front, who has responsibility. By the people who are disclaiming it, yes.

exasperaited 1y ago

Absolutely none of that article has ever even so much as brushed past the colloquial definition of "calculus". These guys actually seem rattled now.

deleted 1y ago
[deleted]
photochemsyn 1y ago
Well, 'calculus' is the kind of marketing word that sounds more impressive than 'arithmetic' and I think 'quantum logic' has gone a bit stale, and 'AI-based' might give more hope to the anxious investor class, as 'AI-assisted' is a bit weak as it means the core developer team isn't going to be cut from the labor costs on the balance sheet, they're just going to be 'assisted' (things like AI-written unit tests that still need some checking). "The Arithmetic of AI-Assisted Coding Looks Marginal" would be the more honest article title.
- collingreen 1y ago
  "Galaxy-brain pair programming with the next superintelligence"
- photonthug 1y ago
  Yes, unfortunately a phrase that's used in an attempt to lend gravitas and/or intimidate people. It sort of vaguely indicates "a complex process you wouldn't be interested in and couldn't possibly understand". At the same time it attempts to disarm any accusation of bias in advance by hinting at purely mechanistic procedures. Could be the other way around, but I think marketing-speak is taking cues here from legal-ese and especially the US supreme court, where it's frequently used by the justices. They love to talk about "ethical calculus" and the "calculus of stare decisis" as if they were following any rigorous process or believed in precedent if it's not convenient. New translation from original Latin: "we do what we want and do not intend to explain". Calculus, huh? Show your work and point to a real procedure or STFU

philipp-gayret 1y ago

This is the first time I see "steering rules" mentioned. I do something similar with Claude, curious how it looks for them and how they integrate it with Q/Kiro.

manmal 1y ago
Those rules are often ignored by agents. Codex is known to be quite adhering, but it falls back to its own ideas, which run counter to rules I‘ve given it. The longer a session goes on, the more it goes off the rails.
- CharlesW 1y ago
  Everything related to LLMs is probabilistic, but those rules are also often followed well by agents.
  - manmal 1y ago
    Yes they do, most of the time. Then they don’t. Yesterday, I told codex that it must always run tests by invoking a make target. That target is even configurable w/ parameters, eg to filter by test name. But always, at some point in the session, codex started disregarding that rule and fell back to using the platform native test tool directly. I used strong language to steer it back, but 20% or so of context later, it did that again.
    - Dilettante_ 1y ago
      Once the LLM has made one mistake, it's often best to start a new context. Since its mechanism is to predict the next token of the conversation, it's reasonable to "predict" itself making more mistakes once it has made one.
      - manmal 1y ago
        I‘m not sure this is still the case with codex. In this instance, restarting had no strong effect.

whiterook6 1y ago

This reads like "Hey, we're not vibe coding, but when we do, we're careful!" with hints of "AI coding changes the costs associated with writing code, designing features, and refactoring" sprinkles in to stand out.

moron4hire 1y ago

If you are producing real results at 10x then you should be able to show that you are a year ahead of schedule in 5 weeks. Waiting to see anyone show even a month ahead of schedule after 6 months.

__MatrixMan__ 1y ago
I've never worked anywhere that knew where they were going well enough that it was even possible to be a month ahead of schedule. By the time a month has elapsed the plan is entirely different. AI can't keep up because its context window is full of yesteryear's wrong ideas about what next month will look like.
- moron4hire 1y ago
  Yeah, this is the main problem. Writing of code just isn't the bottle neck. It's the discovery of the business case that is the hard part. And if you don't know what it is, you can't prompt your way out of it. We've been having a go around with corporate leadership at my company about "AI is going to solve our problems". Dude, you don't even know what our problems are. How are you going to prompt the AI to analyze a 300 page PDF on budget policy when you can't even tell me how you read a 300 page PDF with your eyes to analyze the budget policy. I'm tempted to give them what they want: just a chatter box they can ask, "analyze this budget policy for me", just so I can see the looks on their faces when it spits out five poorly written paragraphs full of niceties that talk its way around ever doing any analysis. I don't know, maybe I'm too much of a perfectionist. Maybe I'm the problem because I value getting the right answer rather than just spitting out reams of text nobody is ever going to read anyway. Maybe it's better to send the client a bill and hope they are using their own AIs to evaluate the work rather than reading it themselves? Who would ever think we were intentionally engaging in Fraud, Waste, and Abuse if it was the AI that did it?
  - __MatrixMan__ 1y ago
    I've got my own gripes about leadership, but I'm finding that even when its a goal I've set for myself, watching an AI fail at it represents a refinement of what I thought I wanted: I'm not much better than they are. That, or its a discovery of why what I wanted is impossible and it's back to the drawing board. It's nice to not be throwing away code that I'd otherwise have been a perfectionist about (and still thrown away).

reenorap 1y ago

No. The way to code going forward with AI is Test Driven Development. The code itself no longer matters. You give the AI a set of requirements, ie. tests that need to pass, and then let it code whatever way it needs to in order to fulfill those requirements. That's it. The new reality us programmers need to face is that code itself has an exact value of $0. That's because AI can generate it, and with every new iteration of the AI, the internal code will get better. What matters now are the prompts. I always thought TDD was garbage, but now with AI it's the only thing that makes sense. The code itself doesn't matter at all, the only thing that matters is the tests that will prove to the AI that their code is good enough. It can be dogshit code but if it passes all the tests, then it's "good enough". Then, just wait a few months and then rerun the code generation with a new version of the AI and the code will be better. The humans don't need to know what the code actually is. If they find a bug, write a new test and force the AI to rewrite the code to include the new test. I think TDD has really found its future now that AI coding is here to stay. Human code doesn't matter anymore and in fact I would wager that modifying AI generated code is as bad and a burden. We will need to make sure the test cases are accurate and describe what the AI needs to generate, but that's it.

pcarolan 1y ago
I mostly agree, but why stop at tests? Shouldn’t it be spec driven development? Then neither the code or the language matter. Wouldn’t user stories and requirements à la bdd (see cucumber) be the right abstraction?
- reenorap 1y ago
  I don't think you're wrong but I feel like there's a big bridge between the spec and the code. I think the tests are the part that will be able to give the AI enough context to "get it right" quicker. It's sort of like a director telling an AI the high level plot of a movie, vs giving an AI the actual storyboards. The storyboards will better capture the vision of the director vs just a high level plot description, in my opinion.
- __MatrixMan__ 1y ago
  Maybe one day. I find myself doing plenty of course correction at the test level. Safely zooming out doesn't feel imminent.

brazukadev 1y ago

But here's the critical part: the quality of what you are creating is way lower than you think, just like AI-written blog posts.

collingreen 1y ago
Upvoted for dig that is also an accurate and insightful metaphor.

cadamsdotcom 1y ago

"We have real mock versions of all our dependencies!" Congratulations, you invented end-to-end testing. "We have yellow flags when the build breaks!" Congratulations! You invented backpressure. Every team has different needs and path dependencies, so settles on a different interpretation of CI/CD and software eng process. Productizing anything in this space is going to be an uphill battle to yank away teams' hard-earned processes. Productizing process is hard but it's been done before! When paired with a LOT of spruiking it can really progress the field. It's how we got the first CI/CD tools (eg. https://en.wikipedia.org/wiki/CruiseControl https://en.wikipedia.org/wiki/CruiseControl) and testing libraries (eg. pytest) So I wish you luck!

r0x0r007 1y ago

"For me, roughly 80% of the code I commit these days is written by the AI agent" Therefore, it is not commited by you, but by you in the name of AI agent and the holy slop. What to say, I hope that 100x productivity is worth it and you are making tons of money. If this stuff becomes mainstream, I suggest open source developers stop doing the grind part, stop writing and maintaining cool libraries and just leave all to the productivity guys, let's see how far they get. Maybe I've seen too many 1000x hacker news..

visarga 1y ago
Just need the feedback to follow suit to be 100x as effective. Tests, docs and rapid loops of guidance with human in the loop. Split your tasks, find the structure that works.
ChadNauseam 1y ago
I think it's fine. For example, "I" made this library https://github.com/anchpop/weblocks https://github.com/anchpop/weblocks . It might be more accurate to say that I directed AI to make it, because I didn't write a line of code myself. (And I looked at the code and it is truly terrible.) But I tested that it works, and it does, and it solves my problem perfectly. Yes, it is slop, but this is a leaf node in the abstraction graph, and no one needs to look at it again now that it it written
- deleted 1y ago
  [deleted]
- shakna 1y ago
  Most code, though, is not write once and ignore. So it does matter if its crap, because every piece of software is only as good as its least dependency. Fine for just you. Not fine for others, not fine for business, not fine the moment you star count starts moving.

deleted 1y ago

[deleted]

bcrosby95 1y ago

It's amazing that their metrics exactly match the mythical "10x engineer" in productivity boost.

Naklin 1y ago
Without disclosing said metrics or any data... This is really something striking to me about all these AI productivity claims. They never provide the methodology and data.
- quikoa 1y ago
  They rarely even provide the projects either or even the type of project. I'd like to see all these awesome results that are build with AI (preferably not web related).

Madmallard 1y ago

Lots of reasonable criticisms being down-voted here. Are we being AstroTurfed? Is HN falling victim to the AI hype train money too now?

nl 1y ago
I'm downvoting most of the criticisms because in general they can be summarized as "all AI is slop". In my experience that simply isn't true. This article attempted to outline a fairly reasonable approach to using AI tooling, and the criticisms hardly seem related to it at all.

StilesCrisis 1y ago

The biggest thing that stood out to me was that they suddenly started working nonstop, even on weekends…? If AI is so great, why can’t they get a single day off in two months?

Madmallard 1y ago

Correct TDD involves solving all the hard problems in the process. What gain does AI give you then?

Lucian6 1y ago

[dead]

zeroq 1y ago

When Karpathy wrote Software 2.0 I was super excited. I naively believed that we'll start building black boxes based on requirements, sets of inputs and outputs, and sudden changes of heart from stakeholders that often happen on a daily basis for many of us and mandates almost complete reimagination of project architecture will simply need another pass of training with new parameters. Instead the mainstream is pushing hard reality where we mass produce a ton of code until it starts to work within guard rails. Does it really work? Is it maintainable? Get out of here. We're moving at 200mph.

rob_c 1y ago
How tf else did you honestly expect black-boxes to get built, by self-mangling machine code spit out by a sentient AI god? Karpathy is bullish on everything bleeding edge, and unfortunately it kinda shows when you know the material better than he does. (source, I've been lecturing on all of it for a few years now). I'm not saying this is bad. It's great to see people who are engaging and bullish, it's better than most futurists waving their hands and going "something, something warp drive". But when you take a step back and really ask what is going on behind the scenes, all we have is massive statistical tools performing neato tricks at statistical probability to predict patterns. There's no greater understanding or ability to learn or mimic. YET. The transformer for-instance can't easily learn complex mathematical operations. There's a google paper on "learning" multiplication and I know people working on building networks to "learn" sin/cos from scratch. But given these basic limitations and pretty much, every, single, paper, out of Apple "intelligence" crapping on the buzz. We've pretty much hit a limit beyond being the first company to allow for multi-trillion token parsing (or basic, limited, token parsing memory) for companies to capture and retrieve information.
- adammarples 1y ago
  Do you really think he knows that little? I mean fair enough you've been lecturing on it, but he was lecturing a decade ago, at Stanford. Then he took a little break to you know, run AI at Tesla...
  - 1y ago

tcherasaro 1y ago

Another day, and another smart person finally discovers the benefits of leveraging AI to write code.

sherinjosephroy 1y ago

[flagged]

ang_cire 1y ago

As a security researcher, I am both salivating at the potential that the proliferation of TDD and other AI-centric "development" brings for me, and scared for IT at the same time. Before we just had code that devs don't know how to build securely. Now we'll have code that the devs don't even know what it's doing internally. Someone found a critical RCE in your code? Good luck learning your own codebase starting now! "Oh, but we'll just ask AI to write it again, and the code will (maybe) be different enough that the exact same vuln won't work anymore!" <- some person who is going to be updating their resume soon. I'm going to repurpose the term, and start calling AI-coding "de-dev".

zeroq 1y ago
Just few days ago I spoke with sec guy who was telling me how frustrating it is to validate AI code. The problem is marketing. Cycling industry is akin to audiophiles and will swear on their lives that $15,000 bicycle is the pinnacle of human engineering. This year's bike will go 11% faster than the previous model. But if you read last 10 years of marketing materials and do math it should basically ride itself. There's so much money in AI right now that you can't really expect anyone to say "well, we had hopes, but it doesn't really work the way we expected". Instead you have pitch after pitch, masses parroting CEOs, and everyone wants to get a seat on the hype train. It's easy to dispel audiophiles or carbon enthusiasts but it's not so easy with AI, because no one really knows how it works. OpenAI released a paper in which they stated, sorry for paraphrasing, "we did this, we did that, and we don't know why results were different".
nosianu 1y ago
> Now we'll have code that the devs don't even know what it's doing internally. I think that has already been true for some time for large projects continuously updated over a long time, and lots of developers entering and leaving the project throughout the years because nobody who has a choice wants to do that demoralizing job for long (I was one of them in the 1990s, the job was later given to an Indian H1B who could not switch to something better easily, not before putting in a few years of torture to have a better resume, and possibly a greencard). Most famous post here, but I would like to see what e.g. Microsoft's devs would have to say, or Adobe's: https://news.ycombinator.com/item?id=18442941 https://news.ycombinator.com/item?id=18442941 Such code has long been held together by the extensive test suites rather than intimate knowledge of how it all works. The task of the individual developer is to close bug tickets and add features, not to produce an optimal solution, or even refactoring. They long ago gave up on that as taking too long.

zbyforgotp 1y ago

TLDR: ai changes the economic calculus of software development. It makes automated testing more beneficial in comparison to costs. I think he is right.

keeda 1y ago

This article is right, but I think it may underplay the changes that could be coming soon. For instance, as the top comment here about TDD points out, the actual code does not matter anymore. This is an astounding claim! And it has naturally received a lot of objections in the replies. But I think the objections can mostly be overcome with a minor adjustment: You only need to couple TDD with a functional programming style. Functional programming lets you tightly control the context of each coding task, which makes AI models ridiculously good at generating the right code. Given that, if most of your code is tightly-scoped, well-tested components implementing orthogonal functionality, the actual code within those components will not matter. Only glue code becomes important and that too could become much more amenable to extensive integration testing. At that point, even the test code may not matter much, just the test-cases. So as a developer you would only really need to review and tweak the test cases. I call this "Test-Case-Only Development" (TCOD?) The actual code can be completely abstracted away, and your main task becomes design and architecture. It's not obvious this could work, largely because it violates every professional instinct we have. But apparently somebody has even already tried it with some success: https://www.linkedin.com/feed/update/urn:li:activity:7196786658942291969/ https://www.linkedin.com/feed/update/urn:li:activity:7196786... All the downsides that have been mentioned will be true, but also may not matter anymore. E.g. in a large team and large codebase, this will lead to a lot of duplicate code with low cohesion. However, if that code does what it is supposed to and is well-tested, does the duplication matter? DRY was an important principle when the cost of code was high, and so you wanted to have as much leverage as possible via reuse. You also wanted to minimize code because it is a liability (bugs, tech debt, etc.) and testing, which required even more code that still didn't guarantee lack of bugs, was also very expensive. But now that the cost of code is plummeting, that calculus is shifting too. You can churn out code and tests (including even performance tests, which are always an afterthought, if thought of at all) at unimaginable rates. And all this while reducing the dependencies of developers on libraries and frameworks and each other. Fewer dependencies means higher velocity. The overall code "goodput" will likely vastly outweight inefficiences like duplication. Unfortunately, as TFA indicates, there is a huge impedance mismatch with this and the architectures (e.g. most code is OO, not functional), frameworks, and processes we have today. Companies will have to make tough decisions about where they are and where they want to get. I suspect AI-assisted coding taken to its logical conclusion is going to look very different from what we're used to.

rob_c 1y ago

Trust but verify. It's not hard. The corollary being. If you can't (through skill or effort) verify don't trust. If you break this pattern you deserve all the follies that become you as a "professional".

zkmon 1y ago

Is it exciting because work happens at 200mph or is it because you get that much business advantage against your competition? Or is it because, now it allows you to spend only one hour at work per day? To quote Joey from Friends - "400 bucks are gone from my pocket and nobody is getting happier?"

Zanfa 1y ago

IMO the biggest issue with AI code is that writing code is the easiest part of software development. Reviewing code is so much more difficult than writing it, even more so if you're not already intimately familiar with it in the first place. It's like with AI images, where they look plausible at first, but then you start noticing all the little things that are off in the sidelines.

Dilettante_ 1y ago
writing code is the easiest part of software development. Reviewing code is so much more difficult than writing it A lot of people say this, and I do not doubt that it is fully true in their real experience. But it is not necessarily the only way for things to be. If more time and effort were put into writing code which is easier to review, the difficulty of writing it would increase and the difficulty of reading it would decrease, flipping that equation. The incentives just aren't like that. It doesn't pay to maximize readability against time spent writing: Not every line will have to be reviewed, and not every line that has to be reviewed will be so complex that readability needs to be perfect to be maintainable.
- oblio 1y ago
  The problem with AI is that those incentives wrong incentives are taken to 10000x. And regarding "not every line will have to be reviewed, and not every line that has to be reviewed will be so complex that readability needs to be perfect to be maintainable.", the problem with AI is that code becomes basically unknowable. Which is fine if everything that is built is slop, but many things aren't slop. Stuff that touches money, healthcare, personal relationships, etc you know, the things that matter in life, risks all turning into slop, which <will> have real life consequences. We'll start seeing this in a few years.
- Zanfa 1y ago
  It's not the code itself that makes review difficult. Even the best written code can be difficult to review. The complexity of effective code review arises from the fact that you need to understand the domain to evaluate correctness of both the code itself and the tests covering it.

Simplita 1y ago

[dead]

jcgrillo 1y ago

> For our team, every commit has an engineer's name attached to it, and that engineer ultimately needs to review and stand behind the code. Then they claim (and demonstrate with a picture of a commits/day chart) a team-wide 10x throughput increase. I claim there's got to be a lot of rubber-stamp reviewing going on here. It may help to challenge the "author" to explain things like "why does this lifetime have the scope it does?" or "why did you factor it this way instead of some other way?" e.g. questions which force them to defend the "decisions" they made. I suspect if you're actually doing thorough reviews that the velocity will actually decrease instead of increase using LLMs.

jmull 1y ago

> Over the past three months... [we] have been building something really cool The claim is a fast-moving, high performing team has become a 10x fast moving, high-performing team. That's equivalent to 2-1/2 years of development across a team. Shall we expect the tangible results soon? I'm perfectly willing to accept that AI coding will make us all a lot more productive, but I need to see the results.

ghm2180 1y ago
And at that rate we should see a FAANGMULA like company launch in 10x or at least 5x less the time. Right?
- oblio 1y ago
  Programming was rarely the barrier to building these types of companies. I know software people don't want to accept that, but it's almost always something on the business or administrative/management side of things. Even for the programming bits, if your initial programmers suck (for some reason) but you have money, a great management team would just replace them with better programmers and fix the code mess with their help. So even that isn't a programming problem, it's a management problem. And let's look at Twitter, who had atrocious code early on (fail whale galore), yet managed to make a profitable business due to amazing product market fit, despite management incompetence. Companies just need to pass a code quality bar which is much, much, much lower than the bar programmers set.
dgemm 1y ago
> AI coding will make us all I'm willing to believe it will make high-judgement autonomous people more productive, I'm less sure it will scale to everyone. The author is one of the senior-most technical staff at AWS.
nguyendat 1y ago
Could you help share little bit more detail about your team experimental result, e.g: new or legacy code base, how do you team measure/control quality. Thanks

Gud 1y ago

People bemoan unrecognisable code bases, maybe that is true. I’ve sure used various LLMs to solve difficult nuts to crack. Problems I have been able to verbalise, but unable to solve. Chances are that if you are using an LLM to mass produce boiler, you are writing too much boiler.

highfrequency 1y ago

> When your throughput increases by an order of magnitude, you're not just writing more code - you're making more decisions. > These aren't just implementation details - they're architectural choices that ripple through the codebase. > The gains are real - our team's 10x throughput increase isn't theoretical, it's measurable. Enjoyed the article and the points it brought up. I do find it uncanny that this article about the merits and challenges of AI coding was likely written by ChatGPT.

deterministic 1y ago

90% of the code I commit is code generated. No AI required.