3 ms·

Most software is not one off little utilities/scripts, greenfield small projects, etc... That's where LLMs excel, when you don't have much context and can regu

Most software is not one off little utilities/scripts, greenfield small projects, etc... That's where LLMs excel, when you don't have much context and can regurgitate solutions. It's less to do with junior/senior/etc.. and more to do with the types of problems you are tackling.

spiderfarmer 2y ago
Most software is simple HTML.
- mhh__ 2y ago
  This isn't where the leverage is though
  - spiderfarmer 2y ago
    No, but AI will replace a lot of web developers.
refulgentis 2y ago
This is a 30KLOC 6 platform flutter app that's, in this user story, doing VOIP audio, running 3 audio models on-device, including in your browser. A near-replica of the Google Assistant audio pipeline, except all on-device. It's a real system, not kindergarten "look at the React Claude Artifacts did, the button makes a POST request!" The 1500 loc websocket / session management code it refactored and tested touches on nearly every part of the system (i.e. persisting messages, placing search requests, placing network requests to run a chat flow) Also, it's worth saying this bit a bit louder: the "just throwing files in" I mention is key. With that, the quality you observed being in reverse is the distinction: with o1 thinking, and whatever Sonnet's magic is, there's a higher payoff from working a larger codebase. For example, here, it knew exactly what to do for the web because it already saw the patterns iOS/Android/macOS shared. The bend I saw in the curve came from being ultra lazy one night and seeing what would happen if it just had all the darn files.
- achierius 2y ago
  I've definitely noticed the opposite on larger codebases. It's able to do magical things on smaller ones but really starts to fall apart as I scale up.
- noch 2y ago
  > This is a 30KLOC 6 platform flutter app […] It's a real system, not kindergarten "look at the React Claude Artifacts did, the button makes a POST request!" This is powerful and significant but I think we need to ground ourselves on what a skilled programmer means when he talks about solving problems. That is, honestly ask: What is the level of skill a programmer requires to build what you've described? Mostly, imo, the difficulties in building it are in platform and API details not in any fundamental engineering problem that has to be solved. What makes web and Android programming so annoying is all the abstractions and frameworks and cruft that you end up having to navigate. Once you've navigated it, you haven't really solved anything, you've just dealt with obstacles other programmers have put in your way. The solutions are mostly boilerplate-like and the code I write is glue. I think the definition of "junior engineer" or "simple app" will be defined by what LLMs can produce and so, in a way, unfortunately, the goal posts and skill ceiling will keep shifting. On the other, hand, say we watch a presentation by the Lead Programmer at Naughty Dog, "Parallelizing the Naughty Dog Engine Using Fibers"[^0] and ask the same questions: what level of skill is required to solve the problems he's describing (solutions worth millions of dollars because his product has to sell that much to have good ROI): "I have a million LOC game engine for which I need to make a scheduler with no memory management for multithreaded job synchronization for the PS4." A lot of these guys, if you've talked to them, are often frustrated that LLMs simply can't help them make headway with, or debug, these hard problems where novel hardware-constrained solutions are needed. --- [^0]: https://www.youtube.com/watch?v=HIVBhKj7gQU https://www.youtube.com/watch?v=HIVBhKj7gQU
  - refulgentis 2y ago
    It's been pretty hard, but if you reduce it to "Were you using a framework, or writing one that needs to push the absolute limits of performance?"... ...I guess the first?... ...But not really? I'm not writing GPU kernels or operating system task schedulers, but I am going to some pretty significant lengths to be running ex. local LLM, embedding model, Whisper, model for voice activity detection, model for speaker counting, syncing state with 3 web sockets. Simultaneously. In this case, Android and iOS are no valhalla of vapid-stackoverflow-copy-pasta-with-no-hardware-constraints, as you might imagine. And the novelty is, 6 years ago, I would have targeted iOS and prayed. Now I'm on every platform at top-tier speeds. All that boring tedious scribe-like stuff that 90% of us spend 80% of our time on, is gone. I'm not sure there's very many people at all who get to solve novel hardware-constrained problems these days, I'm quite pleased to brush shoulders with someone who brushes shoulders with them :) Thus, smacks more of no-true-scotsman than something I can chew on. Productivity gains are productivity gains, and these are no small productivity gain in a highly demanding situation.
dheera 2y ago
I think most software outside of the few Silicon Valleys of the world is in fact a bunch of dirty hacks put together. I fully believe recursive application of current SOTA LLMs plus some deployment framework can replace most software engineers who work in the cornfields.
riwsky 2y ago
No true Botsman would replace a junior eng!

spiderfarmer 2y ago

Most software is simple HTML.

mhh__ 2y ago
This isn't where the leverage is though
- spiderfarmer 2y ago
  No, but AI will replace a lot of web developers.

refulgentis 2y ago

This is a 30KLOC 6 platform flutter app that's, in this user story, doing VOIP audio, running 3 audio models on-device, including in your browser. A near-replica of the Google Assistant audio pipeline, except all on-device. It's a real system, not kindergarten "look at the React Claude Artifacts did, the button makes a POST request!" The 1500 loc websocket / session management code it refactored and tested touches on nearly every part of the system (i.e. persisting messages, placing search requests, placing network requests to run a chat flow) Also, it's worth saying this bit a bit louder: the "just throwing files in" I mention is key. With that, the quality you observed being in reverse is the distinction: with o1 thinking, and whatever Sonnet's magic is, there's a higher payoff from working a larger codebase. For example, here, it knew exactly what to do for the web because it already saw the patterns iOS/Android/macOS shared. The bend I saw in the curve came from being ultra lazy one night and seeing what would happen if it just had all the darn files.

achierius 2y ago
I've definitely noticed the opposite on larger codebases. It's able to do magical things on smaller ones but really starts to fall apart as I scale up.
noch 2y ago
> This is a 30KLOC 6 platform flutter app […] It's a real system, not kindergarten "look at the React Claude Artifacts did, the button makes a POST request!" This is powerful and significant but I think we need to ground ourselves on what a skilled programmer means when he talks about solving problems. That is, honestly ask: What is the level of skill a programmer requires to build what you've described? Mostly, imo, the difficulties in building it are in platform and API details not in any fundamental engineering problem that has to be solved. What makes web and Android programming so annoying is all the abstractions and frameworks and cruft that you end up having to navigate. Once you've navigated it, you haven't really solved anything, you've just dealt with obstacles other programmers have put in your way. The solutions are mostly boilerplate-like and the code I write is glue. I think the definition of "junior engineer" or "simple app" will be defined by what LLMs can produce and so, in a way, unfortunately, the goal posts and skill ceiling will keep shifting. On the other, hand, say we watch a presentation by the Lead Programmer at Naughty Dog, "Parallelizing the Naughty Dog Engine Using Fibers"[^0] and ask the same questions: what level of skill is required to solve the problems he's describing (solutions worth millions of dollars because his product has to sell that much to have good ROI): "I have a million LOC game engine for which I need to make a scheduler with no memory management for multithreaded job synchronization for the PS4." A lot of these guys, if you've talked to them, are often frustrated that LLMs simply can't help them make headway with, or debug, these hard problems where novel hardware-constrained solutions are needed. --- [^0]: https://www.youtube.com/watch?v=HIVBhKj7gQU https://www.youtube.com/watch?v=HIVBhKj7gQU
- refulgentis 2y ago
  It's been pretty hard, but if you reduce it to "Were you using a framework, or writing one that needs to push the absolute limits of performance?"... ...I guess the first?... ...But not really? I'm not writing GPU kernels or operating system task schedulers, but I am going to some pretty significant lengths to be running ex. local LLM, embedding model, Whisper, model for voice activity detection, model for speaker counting, syncing state with 3 web sockets. Simultaneously. In this case, Android and iOS are no valhalla of vapid-stackoverflow-copy-pasta-with-no-hardware-constraints, as you might imagine. And the novelty is, 6 years ago, I would have targeted iOS and prayed. Now I'm on every platform at top-tier speeds. All that boring tedious scribe-like stuff that 90% of us spend 80% of our time on, is gone. I'm not sure there's very many people at all who get to solve novel hardware-constrained problems these days, I'm quite pleased to brush shoulders with someone who brushes shoulders with them :) Thus, smacks more of no-true-scotsman than something I can chew on. Productivity gains are productivity gains, and these are no small productivity gain in a highly demanding situation.

dheera 2y ago

I think most software outside of the few Silicon Valleys of the world is in fact a bunch of dirty hacks put together. I fully believe recursive application of current SOTA LLMs plus some deployment framework can replace most software engineers who work in the cornfields.

riwsky 2y ago

No true Botsman would replace a junior eng!