4 ms·
I found myself agreeing with quite a lot of this article. I'm a pretty huge proponent for AI-assisted development, but I've never found those 10x claims convin
by simonw 1y ago
I found myself agreeing with quite a lot of this article.
I'm a pretty huge proponent for AI-assisted development, but I've never found those 10x claims convincing. I've estimated that LLMs make me 2-5x more productive on the parts of my job which involve typing code into a computer, which is itself a small portion of that I do as a software engineer.
That's not too far from this article's assumptions. From the article:
> I wouldn't be surprised to learn AI helps many engineers do certain tasks 20-50% faster, but the nature of software bottlenecks mean this doesn't translate to a 20% productivity increase and certainly not a 10x increase.
I think that's an under-estimation - I suspect engineers that really know how to use this stuff effectively will get more than a 0.2x increase - but I do think all of the other stuff involved in building software makes the 10x thing unrealistic in most cases.
- hyperbovine 1y agoThis is (I think) a reference to the 10x engineer, another myth of which I have always been highly dubious (https://www.simplethread.com/the-10x-programmer-myth/ https://www.simplethread.com/the-10x-programmer-myth/).
- Izkata 1y ago10x sounds nice which is probably why it stuck, but it came from actual research which found the difference was larger than 10x - but also they were measuring between best and worst, not best and average as it's used nowadays. https://www.construx.com/blog/productivity-variations-among-software-developers-and-teams-the-origin-of-10x/ https://www.construx.com/blog/productivity-variations-among-...
- mgfist 1y agoAll of this is hard to quantify. How much better than the average engineer is John Carmack, or Rob Pike or Linus? I consider myself average-ish and I don't think there's any world in which I could do what those guys did no matter how much time you gave me (especially without the hindsight knowledge of the creations). So I'd say they're all infinitely better than me.
- SketchySeaBeast 1y agoI guess that makes Newton a 10x scientist. Really puts in perspective how utterly unrealistic it is to be looking to hire exclusively 10x programmers - the true 10x'ers are legends, not just regular devs who type a bit faster.
- dolebirchwood 1y agoIt would be more sensible if the "10x" moniker was dropped altogether, and we just went back to calling these people what they've always been: "geniuses". Then there might be more realistic expectations of only finding them among 1% of the population.
- zepolen 1y agoNewton was a 1000x
- jirach08 1y agoI'd argue he was 9.81x
- SketchySeaBeast 1y agoI think it's more important to measure his changing contribution as a function of time.
- coffeebeqn 1y agoAnd how much better are they than your average engineer when plopped into a mediocre organization where they aren’t the political and technical top dog? I would guess they would all quit within a week.
- bigfishrunning 1y agoGood engineers don't stay in mediocre organizations, mediocre ones do. Do you think these "top dogs" were at the top of their game from day one? They all learned, just like everyone else; talent just gave them a higher ceiling.
- zahlman 1y ago>but also they were measuring between best and worst, not best and average as it's used nowadays. Depending on the environment, I can imagine the worst devs being net negative.
- smrtinsert 1y agoAnd they get promoted too. Multiple times I've seen people get promoted for decisions that doom the company years later. Considering all the various departments and people that go into supporting these net negative engineers, more people are net negative than they think.
- logicchains 1y agoYou don't believe Fabrice Bellard exists?
- gopher_space 1y agoHe doesn't believe there are hundreds of Fabrice Bellard clones who think working at your company wouldn't be a waste of their time. The myth might be that thinking about 10X is useful in any sense. You can't plan around one gracing you with their presence and you won't be able to retain them when they do. Thinking about it personally, a 10X label means I'm supposedly the smartest person in the room and that I'm earning 1/10th what I should be. Both of those are huge negatives.
- AdrianB1 1y agoIt highly depends on the circumstances. In over 30 years in the industry I met 3 people that were many times more productive than everyone else around them, even more than 10 times. What does this translate to? Well, there are some extraordinary people around, very rare and you cannot count on finding some and, when you find them, it is almost impossible to retain them because management and HR never agree to pay them enough to stay around.
- coltonv 1y agoThanks for the comment Simon! This is honestly the first one I've read where it feels like someone actually read the article. I'm totally open to the idea that some people, especially those working on the languages/tools that LLMs are good at, are indeed getting a 2x improvement in certain parts of their job.
- simonw 1y agoSomething I have realized about Hacked News is that most of the comments on any given article are from people who are responding to the headline without actually clicking through and reading it! This is particularly true for headlines like this one which stand alone as statements.
- pmnord 1y agoThis is a truism across the entire web
- coltonv 1y agoPerhaps that's my fault for making the title almost clickbaity. My goal was to get people who felt anxious about AI turning them into dinosaurs not feel like they are missing some secret sauce, so hopefully the reach this is getting contributes that. Again, appreciate your thoughts, I have a huge amount of respect for your work. I hope you have a good one!
- steveklabnik 1y agoNah, it’s the way news aggregators have been we ever since they have existed.
- simonw 1y agoIf you hadn't made the title clickbaity you probably wouldn't have hit the homepage!
- tptacek 1y ago
- mavamaarten 1y agoYeah. I just need to babysit it too much. Take copilot, it gives good suggestions and blows me away sometimes with a block of code which is exactly what I'd type. But actively letting it code (at least with gpt4.1 or gpt4o) just doesn't work well enough for me. Half of the time it doesn't even compile, and after fixing that it's just not really correctly working either. I'd expect it to work like a very junior programmer, but it works like a very drunk senior programmer that isn't listening to you very well at all.
- ffsm8 1y agoTo date, I've not been able to effectively use Copilot in any projects. The suggestions were always unusably bad. The /fix were always obviously and straight up false unless it was a super silly issue. Claude Code with Opus model on the other hand was mind-blowing to me and made me change my mind on almost everything wrt my opinion of LLMs for coding. You still need to grow the skill of how to build the context and formulate the prompt, but the buildin execution loop is a complete game changer and I didn't realize that until I actually used it effectively on a toy project myself. MCP in particular was another thing I always thought was massively over hyped, until I actually started to use some in the same toy project. Frankly, the building blocks already exist at this point to make a vast majority of all jobs redundant (and I'm thinking about all grunt work office jobs, not coding in particular). The tooling still need to be created, so I'm not seeing a short term realization (<2 yrs), but medium term (5+yrs)? You should expect most companies to let people go at staggering numbers, with only small amounts of highly skilled people left to administer the agents
- necovek 1y agoThanks for this perspective, but I am a bit confused by some of your takes: you used "Claude Code with Opus model" in "the same toy project" with great success, which led you to conclude that this will "make a vast majority of all jobs redundant". Toy project viability does not connect with making people redundant in the process (ever, really) — at least not for me. Care to elaborate where do you draw the optimism from?
- mgfist 1y agoMy biggest takeaway from using AI is that (1) for my day job, it doesn't make me super productive with creation, but it does help with discovery, learning, getting myself unstuck, and writing tedious code. (2) however, the biggest unlock is it makes working on side projects __immensely__ easier. Before AI I was always too tired to spend significant time on side projects. Now, I can see my ideas come to life (albeit with shittier code), with much less mental effort. I also get to improve my AI engineering skills without the constraint of deadlines, data privacy, tool constraints etc..
- bluefirebrand 1y ago> (1) for my day job, it doesn't make me super productive with creation, but it does help with discovery, learning, getting myself unstuck, and writing tedious code I hear this take a lot but does it really make that much of an improvement over what we already had with search engines, online documentation and online Q&A sites?
- j45 1y agoYes, it can.
- simonw 1y agoThose things don't really help with getting unstuck, especially if the reason you are struck is that there tedious code that you anticipate writing and don't want to deal with.
- orwin 1y agoExactly. My two worst roadblocks are the beginning of a new feature when I procrastinate way too much (I'm a bit afraid of choosing a design/architecture and committing to it) and towards the end when I have to fix small regressions and write tests, and I procrastinate because I just don't want to. AI solved the second roadblock 100% of the time, and help with design decisions enough to be useful (Claude4 at least). The code in the middle is a plus, but tbh I often do it myself (unless it's frontend code).
- 1y ago
- thomascgalvin 1y agoMy experience with GenAI is that it's a significant improvement to Stack Overflow, and generally as capable as someone hired right out of college. If I'm using it to remember the syntax or library for something I used to know how to do, it's great. If I'm using it to explore something I haven't done before, it makes me faster, but sometimes it lies to me. Which was also true of Stack Overflow. But when I ask it to so something fairly complex on it's own, it usually tips over. I've tried a bunch of tests with a bunch of models, and it never quite gets it right. Sometimes it's minor stuff that I can fix if I bang on it long enough, and sometimes it's a steaming pile that I end up tossing in the garbage. For example, I've asked it to code me a web-based calculator, or a 3D model of the solar system using WebGL, and none of the models I've tried have been able to do either.
- doc_manhat_3 1y agoI think even claims of 2-5x are highly suspect. It would imply that if your team is using AI then all else equal they accomplish 2-5 times as much in a quarter. I don't know about you but I'm certainly not seeing this and most people on my team use AI. [And to those saying we're using it wrong... well I can't argue with something that's not falsifiable]
- __loam 1y agoThere's a bunch of open source projects where we can observe that not happening in real time.
- coffeebeqn 1y agoMy company is all in on LLMs and honestly the improvement seems to be like 0.9x to 1.2x depending on the project. None of them are moving at break neck speed and many projects are just as bogged down by complexity as ever. Pretty big (3000+) company with a large mature codebase in multiple languages. For god knows how much money spent on it
- rightbyte 1y agoDo you mean -10 to +20% or 90 to 120%? I am not allowed to use LLMs at work for work code so I can't tell what claims are real. Just my 80s game reimplementations of Snake and Asteroids.
- yen223 1y agoOnce a company gets to a certain size, they no longer optimise for product development, instead they optimise for risk mitigation. A lot of processes will be put in place with the sole purpose of slowing down code development.
- paulcole 1y ago> but I've never found those 10x claims convincing Who's making these claims?
- WD-42 1y agoThere was a YC video just a few months ago where a bunch of jergoffs sat in a circle and talked about engineers being 10 to 100x as effective as before. Im sure google will bring it up.
- paulcole 1y ago> Im sure google will bring it up. It didn’t.
- WD-42 1y agoFound it https://youtu.be/IACHfKmZMr8 https://youtu.be/IACHfKmZMr8
- paulcole 1y agoSo this whole discussion started because some guy got triggered by a thinly veiled advertisement on a podcast run by a VC firm? I’ll admit this is not helping the case of “but people are saying…”
- WD-42 1y agoYou asked who’s making these silly claims, I provided one example of YC partners doing it. Not sure who got triggered or what advertising you are talking about, but there you go.
- paulcole 1y agoWhat you showed me is the equivalent of somebody saying, “People are claiming my life would be better if I drank more Pepsi” and the underlying evidence turned out to be a Pepsi commercial. You understand what YC is right?
- mkozlows 1y agoThe thing that I keep wondering about: If the coding part is 2-5x more productive for you, but the stuff around the coding doesn't change... at some point, it'll have to, right? The cost/benefit of a lot of practices (this article talks about code review, which is a big one) changes a lot if coding becomes significantly easier relative to other tasks.
- coffeebeqn 1y agoWhy would it have to? Most of the job is trying to understand what people really want and that’s not any easier than it was 10 or 30 years ago
- simonw 1y agoYes, absolutely. Code used to be more expensive to write, which meant that a lot of features weren't sensible to build - the incremental value they provided wasn't worth the implementation effort. Now when I'm designing software there are all sorts of things where I'm much less likely to think "nah, that will take too long to type the code for".
- cleak 1y agoI’ve found I do get small bursts of 10x productivity when trying to prototype an idea - much of the research on frameworks and such just goes away. Of course that’s usually followed by struggling to make a seemingly small change for an hour or two. It seems like the 10x number is just classic engineers underestimating tasks - making estimates based on peak productivity that never materializes. I have found for myself it helps motivate me, resulting in net productivity gain from that alone. Even when it generates bad ideas, it can get me out of a rut and give me a bias towards action. It also keeps me from procrastinating on icky legacy codebases.
- sureglymop 1y agoI do also believe that those who are often looked at or referred to as 10x engineers will maybe only see a marginal productivity increase. The smartest programmer I know is so impressive mainly for two reasons: first, he seems to have just an otherworldly memory and seems to kind of have absolutely every little feature and detail of the programming languages he uses memorized. Second, his real power is really in cognitive ability, or the ability to always quickly and creatively come up with the smartest and most efficient yet elegant and clean solution to any given problem. Of course somewhat opinionated but in a good way. Funnily he often wouldn't know the academic/common name for some algorithm he arrived at but it just happened to be what made sense to him and he arrived at it independently. Like a talented musician with perfect pitch who can't read notation or doesn't know theory yet is 10x more talented than someone who has studied it all. When I pair program with him, it's evident that the current iteration of AI tools is not as quick or as sharp. You could arrive at similar solutions but you would have to iterate for a very long time. It would actually slow that person down significantly. However, there is such a big spectrum of ability in this field that I could actually see this increasing for example my productivity by 10x. My background/profession is not in software engineering but when I do it in my free time the perfectionist tendencies make me work very slowly. So for me these AI tools are actually cool for generating the first crappy proof of concepts for my side projects/ideas, just to get something working quickly.
- skadamou 1y agoI like the quip that AI raises the floor not the ceiling. I think it helps the bottom 20% perform more like the middle 50% but doesn't do much for people at the top.
- eldaisfish 1y agoI add to that analogy. AI raises the floor but some of the floor tiles fall away, unpredictably.
- jes5199 1y agomaybe, but I find that it makes it much faster to do things that _I already know how to do_, and can only slowly, ploddingly get me to places that I don't already have a strong mental model for, as I have to discover mistakes the hard way
- tmaly 1y agoI wonder if a better metric would be developer happiness? Instead of being 2x or 5x more productive, what if we looked at what a developer enjoyed doing and figured out how to use AI for everything else?
- WD-42 1y agoIf the 10x thing was true we’d all be on iOS 24 and the typescript Go rewrite would have been done months ago. But of course that’s ridiculous.
- 9rx 1y ago> and the typescript Go rewrite would have been done months ago. 10x is intended to symbolize a multiplier. As Microsoft fired that guy, 10 × 0 is still 0.
- fulafel 1y agoSpeed of shipping software and pace of writing code are different things. Shipping software like iOS has a <50% component of programming so Amdahl's law caps the end-to-end improvement rather low, assuming other parts of the process stay the same.
- GeoAtreides 1y ago> 0.2x increase 1.2x increase
- didibus 1y ago> engineers that really know how to use this stuff effectively I guess this is still the "caveat" that can keep the hype hopes going. But I've found at a team velocity level, with our teams, where everyone is actively using agentic coding like Claude Code on the daily, we actually didn't see an increase in team velocity yet. I'm curious to hear anecdotal from other teams, has your team seen velocity increase since it adopted agentic AI?
- jan_g 1y agoSame here. I have a colleague that is completely enamored with these agents. Uses them for everything he can, not just coding. Commit messages, opening PRs, Linear tickets, etc. Basically, he uses agents for everything he can. But the productivity gain is just not there. He's about as fast or rather as slow as he was before. And to a degree I think this goes for the whole team. It's the oxymoron of AI: more code, more documentation, more text, more of everything generated than ever, but the effect is that this means more complexity, more PRs to review, more bugs, more stuff to know and understand, ... We are all still learning how to use these agents effectively. And the particular developer's effect can and does multiply as everything else with GenAI. Was he a bit sloppy before, not covering various edge-cases and used quick-and-dirty shortcuts? Then this remains true for the code he produces using agents. And to those, who claim that "by using more agents I will gain 10x productivity" I say please read a certain book about how just adding developers to a project makes it even more delayed. The resemblance of team/project leadership -> developers dynamic is truly uncanny.
- nicknow 1y agoCompletely agree. What will happen is over time this will become the new baseline for developing software. It will mean we can deliver software faster. Maybe more so than other advances, but it won't fundamentally change the fact that software takes real effort and that effort will not go away, since that effort is much more than just coding this or that function. I could create a huge list of things that have made developing and deploying quality software easier: linters, static type checkers, code formatters, hot reload, intelligent code completion, distributed version control (i.e., Git), unit testing frameworks, inference schema tools, code from schema, etc. I'm sure others can add dozens of items to that list. And yet there seems to be an unending amount of software to be built, limited only by the people available to build it and an organizations funding to hire those people. In my personal work, I've found AI-assisted development to make me faster (not sure I have a good estimate for how much faster.) What I've also found is that it makes it much easier to tackle novel problems within an existing solution base. And I believe this is likely to be a big part of the dev productivity gain. Just an example, lets say we want to use the strangler pattern as part of our modernization approach for a legacy enterprise app that has seen better days. Unless you have some senior devs who are both experienced with that pattern AND experienced with your code base, it can take a lot of trial and error to figure out how to make it work. (As you said, most of our work isn't actually typing code.) This is where an AI/LLM tool can go to work on understanding the code base and understanding the pattern to create a reference implementation approach and tests. That can save a team of devs many weeks of trial & error (and stress) not to mention guidance on where they will run into roadblocks deep into the code base. And, in my opinion, this is where a huge portion of the AI-assisted dev savings will come from - not so much writing the code (although that's helpful) but helping devs get to the details of a solution much faster. It's that googling has always gotten us to generic references and AI gets us those references fit for our solution.
- baxtr 1y agoThe problem is, if you don’t achieve 10x the venture case won’t work out properly.
- SkyPuncher 1y agoI've basically come to the same 2x to 5x conclusion as you. Problem is that "5x productivity" is really only a small portion of my actual job. The hardest part of my job is actually understanding the problem space and making sure we're applying the correct solution. Actual coding is probably about 30% of my job. That means, I'm only looking at something like 30% productivity gain by being 5x as effective at coding.
- QuantumGood 1y agoIt lets me try things I couldn't commit the time to in the past, like quickly cobbling together a keystroke macro. I can also put together the outline of a plan in a few minutes. So much more can be 'touched' upon usefully.
- emseetech 1y ago| I think that's an under-estimation I'm not sure it is and I'll take it a step further: Over the course of development, efficiency gains trend towards zero. AI has a better case for increasing surface area (what an engineer is capable of working on) and effectiveness, but efficiency is a mirage.
- jprokay13 1y agoAt first I thought becoming “10x” meant outputting 10x as much code. Now that I’m using Claude more as an expensive rubber duck, I’m hoping that I spend more time defining the fundamentals correctly that will lead to a large improvement in outcomes in the long run.
- giantg2 1y agoI completely agree. I saw the claims about 30% increase in dev productivity a while ago and thought how is that possible when most of my job consists of meetings, SARs, threat modeling, etc.
- EGreg 1y ago“10x? That’s crazy talk. I’m only 5x more productive, let’s be accurate sirs”
- Waterluvian 1y ago> I've estimated that LLMs make me 2-5x more productive on the parts of my job which involve typing code into a computer, which is itself a small portion of that I do as a software engineer. This feels exactly right and is what I’ve thought since this all began. But it also makes me think maybe there are those that A.I. helps 10x, but more because that code input is actually a very large part of their job. Some coders aren’t doing much design or engineering, just assembly.
- simonw 1y agoYeah, I hadn't thought about that. If you really are a programmer who gets all of their work assigned to them as detailed specifications maybe you are seeing a 10x boost. I don't think I've encountered programmer like that in my own career, but I guess they might exist somewhere!
- astrange 1y agoPersonally I've found it's very good at writing support tools / shell scripts. I mostly use it to parse the output of other tools that don't have machine-readable output yet. Claude Code (which is apparently the best in general) isn't very good at reviewing existing large projects IME, because it doesn't want to load a lot of text into its context. If you ask it to review an existing project it'll search for keywords instead of just loading an entire file. That and it really wants to please you, so if you imply you own a project it'll be a lot more positive than it may deserve.
- belter 1y agoLooking forward to those 20x most productive days out of an LLMs. And what are those most productive days? The ones when you can simplify and delete hundreds of lines of code... :-)
- edanm 1y agoI agree. I'm a big fan/proponent of AI assisted development (though nowhere near your amount of experience with it). And I think that 2x-10x speed up can be true, depending on what you mean exactly and what your task is exactly. This article thinks that most people who say 10x productivity are claiming 10x speedup on end-to-end delivering features. If that's indeed what someone is saying, they're most of the time quite simply wrong (or lying). But I think some people (like me) aren't claiming that. Of course the end to end product process includes a lot more work than just the pure coding aspect, and indeed none of those other parts are getting a 10x speedup right now. That said, there are a few cases where this 10x end-to-end is possible. E.g. when working alone, especially on new things but not only - you're skipping a lot of this overhead. That's why smaller teams, even solo teams, are suddenly super interesting - because they are getting a bigger speedup comparatively speaking, and possibly enough of one to be able to rival larger teams.
- jolt42 1y agoProgrammers are notoriously bad about making estimates. Sure it sped something up 10x, but did you consider those 10 tries using AI that didn't pan out? You're not even breaking even, you are losing time.
- moron4hire 1y agoIf 10x could be believed, we're long enough into having AI-coding assist that any such company that had gone all in would be head and shoulders above their competitors by now. And we're not seeing that at all. The companies whose software I use that did announce big AI initiatives 6 months ago, if they really had gotten 10x productivity gain, that'd be 60 months—5 years—worth of "productivity". And yet somehow all of their software has gotten worse.
- motorest 1y ago> I've estimated that LLMs make me 2-5x more productive on the parts of my job which involve typing code into a computer, which is itself a small portion of that I do as a software engineer. I think that the key realization is that there are tasks where LLMs excel and might even buy you 10x productivity, whereas some tasks their contribution might even be net negative. LLM are largely excellent at writing and refactoring unit tests, mainly because their context is very limited (i.e., write a method in a class that calls this specific method of this specific class a specific way and check the output) and their output is very repetitive (i.e., write isolated methods in standalone classes without output that are not called anywhere). They also seem helpful when prompted to add logging. LLMs are also effective in creating greenfield projects, serving as glorified template engines. But when lightly pressed on specific tasks like implementing a cross-domain feature... Their output starts to be at best a big ball of mud.
- buu700 1y agoI don't doubt that some people are mistaken or dishonest in their self-reports as the article asserts, but my personal experience at least is a firm counterexample. I've been heavily leaning on AI for an engagement that would otherwise have been impossible for me to deliver to the same parameters and under the same constraints. Without AI, I simply wouldn't have been able to fit the project into my schedule, and would have turned it down. Instead, not only did I accept and fit it into my schedule, I was able to deliver on all stretch goals, put in much more polish and automated testing than originally planned, and accommodate a reasonable amount of scope creep. With AI, I'm now finding myself evaluating other projects to fit into my schedule going forward that I couldn't have considered otherwise. I'm not going to specifically claim that I'm an "AI 10x engineer", because I don't have hard metrics to back that up, but I'd guesstimate that I've experienced a ballpark 10x speedup for the first 80% of the project and maybe 3 - 5x+ thereafter depending on the specific task. That being said, there was one instance where I realized halfway through typing a short prompt that it would have been faster to make those particular changes by hand, so I also understand where some people's skepticism is coming from if their impression is shaped by experiences like that. I believe the discrepancy we're seeing across the industry is that prompt-based engineering and traditional software engineering are overlapping but distinct skill sets. Speaking for myself, prompt-based engineering has come naturally due to strong written communication skills, strong code review skills (e.g. participating in security audits), and otherwise being what I'd describe as a strong "jack of all trades, master of some" in software development across the stack. On the other hand, for example, I could easily see someone who's super 1337 at programming high-performance algorithms and mid at most everything else finding that AI insufficiently enhances their core competency while also being difficult to effectively manage for anything outside of that. As to how I actually approach this: * Gemini Pro is essentially my senior engineer. I use Gemini to perform codebase-wide analyses, write documentation, and prepare detailed sprint plans with granular todo lists. Particularly for early stages of the project or major new features, I'll spend a several hours at a time meta-prompting and meta-meta-prompting with Gemini just to get a collection of prompts, documents, and JSON todo lists that encapsulate all of my technical requirements and feedback loops. This is actually harder than manual programming because I don't get the "break" of performing out all the trivial and boilerplate parts of coding; my prompts here are much more information-dense than code. * Claude Sonnet is my coding agent. For Gemini-assisted sprints, I'll fire Claude off with a series of pre-programmed prompts and let it run for hours overnight. For smaller things, I'll pair program with Claude directly and multitask while it codes, or if I really need a break I'll take breaks in between prompting. * More recently, Grok 4 through the Grok chat service is my Stack Overflow. I can't rave enough about it. Asking it questions and/or pasting in code diffs for feedback gets incredible results. Sometimes I'll just act as a middleman pasting things back and forth between Grok and Claude/Gemini while multitasking on other things, and find that they've collaboratively resolved the issue. Occasionally, I've landed on the correct solution on my own within the 2 - 3 minutes it took for Grok to respond, but even then the second opinion was useful validation. o3 is good at this too, but Grok 4 has been on another level in my experience; its information is usually up to date, and its answers are usually either correct or at least on the right track. * I've heard from other comments here (possibly from you, Simon, though I'm not sure) that o3 is great at calling out anti-patterns in Claude output, e.g. its obnoxious tendency to default to keeping old internal APIs and marking them as "legacy" or "for backwards compatibility" instead of just removing them and fixing the resulting build errors. I'll be giving this a shot during tech debt cleanup. As you can see, my process is very different from vibe coding. Vibe coding is fine for prototyping, on for non-engineers with no other options, but it's not how I would advise anyone to build a serious product for critical use cases. One neat thing I was able to do, with a couple days' notice, was add a script to generate a super polished product walkthrough slide deck with a total of like 80 pages of screenshots and captions covering different user stories, with each story having its own zoomed out overview of a diagram of thumbnails linking to the actual slides. It looked way better than any other product overview deck I've put together by hand in the past, with the bonus that we've regenerated it on demand any time an up-to-date deck showing the latest iteration of the product was needed. This honestly could be a pretty useful product in itself. Without AI, we would've been stuck putting together a much worse deck by hand, and it would've gotten stale immediately. (I've been in the position of having to give disclaimers about product materials being outdated when sharing them, and it's not fun.) Anyway, I don't know if any of this will convince anyone to take my word for it, but hopefully some of my techniques can at least be helpful to someone. The only real metric I have to share offhand is that the project has over 4000 (largely non-trivial) commits made substantially solo across 2.5 months on a part-time schedule juggled with other commitments, two vacations, and time spent on aspects of the engagement other than development. I realize that's a bit vague, but I promise that it's a fairly complex project which I feel pretty confident I wouldn't have been capable of delivering in the same form on the same schedule without AI. The founders and other stakeholders have been extremely satisfied with the end result. I'd post it here for you all to judge, but unfortunately it's currently in a soft launch status that we don't want a lot of attention on just yet.
- Cthulhu_ 1y agoThe other thing is that I don't believe software developers actually "do their best" when writing the code itself, that is, optimize the speed of writing code. Nor do they need to; they know writing the code doesn't take up time, waiting for CI and a code review and that iteration cycle does. And does an AI agent doing a code review actually reduce that time too? I have doubts. Caveat, I haven't seen it in practice yet.