5 ms·

> you have [ ... ]. then you have [ ... ] Those are groups defined by something other than actual LLM usage, which makes them both not particularly interesting

> you have [ ... ]. then you have [ ... ] Those are groups defined by something other than actual LLM usage, which makes them both not particularly interesting. What is interesting: You have people who've tried using LLMs to generate code and found it utterly useless. Then you have people who've tried using LLMs to generate code and believe that it has worked very well for them.

0x500x79 1y ago
I think this is an easy thing to wrap my mind around (since I have been in both camps): AI can generate lots of code very quickly. AI does not generate code that follows taste and or best practices. So in cases where the task is small, easily plannable, within the training corpus, or for a project that doesn't have high stakes it can produce something workable quickly. In larger projects or something that needs maintainability for the future code generation can fall apart or produce subpar results.
- logicchains 1y ago
  Recent LLMs are fairly good at following instructions, so a lot of the difference comes down to the level of detail and quality of the instructions given. Written communication is a skill for which there's a huge amount of variance among developers, so it's not surprising that different developers get very different results. The relative quality of the LLM's output is determined primarily by the written communication skills of the individuals instructing it.
  - NoOn3 1y ago
    It seems to me If you know all these instructions clearly, then you know everything, and it's easy for you to write the code yourself, and you don't need an LLM.
    - logicchains 1y ago
      The amount of typing it takes to describe a solution in English text is often less than the amount of typing needed to actually implement it in code, especially after accounting for boilerplate and unit tests. Not to mention the time spent waiting for the compiler and test harness to run. As a concrete example, the HTTP2.0 spec is way fewer chars long than any HTTP2.0 server implementation, and the C spec is way fewer chars long than any compliant C compiler. The C++ spec is way, way fewer chars long than any compliant C++ compiler.
      - no_wizard 1y ago
        >The amount of typing it takes to describe a solution in English text is often less than the amount of typing needed to actually implement it in code I don't find this to be true. I find describing a solution in English well to be slower than describing the problem in code (IE, by writing tests first) and having that be the structured data that the LLM uses to generate code. Its far faster, from the results I'm seeing plus my own personal experience, to write clear tests which benefit from being a form of structured data that the LLM can analyze. Its the guidance we have given to our engineers at my day job and it has made working with these tools dramatically easier. In some cases, I have found LLM performance to be subpar enough that it is indeed, faster to write it myself. If it has to hold many different pieces of information together, it starts to falter.
      - svantana 1y ago
        I don't think it's so clear-cut. The C spec I found is 4MB and the tcc compiler source code is 1.8MB. It might need some more code to be fully compliant, but it may still be smaller than 4MB. I think the main reason why code bases are much larger is because they contain stuff not covered by the spec (optimization, vendor-specific stuff, etc etc). Personally I'd rather write a compiler than a specification, but to each their own.
      - steveBK123 1y ago
        Use shorter variable names
  - mattwad 1y ago
    Yea this! How many devs say "it doesn't do what i expect" did not try to write up a plan of action before it just YOLO'd some new features? We have to learn to use this new tool, but how to do that is still changing all the time.
    - norm_namillu 1y ago
      > We have to learn to use this new tool, but how to do that is still changing all the time. so we need to program in natural language now but targeting some subset of it that is no learnable? and this is better how?
  - no_wizard 1y ago
    Corpus also matters. I know Rust developers who aren't getting very good results even with high quality prompts. On the other hand, I helped integrate Cursor as a staff engineer at my current job for all our developers (many hundreds), who primarily work in JavaScript / TypeScript, and even middling prompts will get results that only require refactoring, assuming the LLM doesn't need a ton of context for the code generation (e.g. greenfield or independent features). Our general approach and guidance has been that developers need to write the tests first and have Cursor use that as a basis for what code to generate. This helps prevent atrophy and over time we've find thats where developers add the most value with these tools. I know plenty of developers want to do it the other way (have AI generate the tests) but we've had more issues with that approach. We discourage AI generating everything and having a human edit the output, as it tends to be slower than our chosen approach and more likely to have issues. That said, LLMs still struggle if they need to hold alot of context. For instance, if you have a bunch of files that it needs to understand to also generate code that is worthwhile, particularly if you want it to re-use code.
    - logicchains 1y ago
      >Corpus also matters. I know Rust developers who aren't getting very good results even with high quality prompts. Which model were they using, out of interest? I've gotten decent results for Rust from Gemini 2.5 Pro. Its first attempt will often be disgusting (cloning and other inefficiencies everywhere), but it can be prompted to optimise that afterwards. It also helps a lot to think ahead about lifetimes and explicitly tell it how to structure them, if there might be anything tricky lifetime-wise.
      - no_wizard 1y ago
        No idea. I do know they all have access to Cursor and tried different models, even the more expensive options. What you're describing though, having to go through that elaborate detail really drives to my point though, and I think shows a weakness in these tools that is a hidden cost to scaling their productivity benefits. What I can tell you though both from observation and experience, is that because the corpus for TypeScript / JavaScript is infinitely larger as it stands today, even Gemini 2.5 Pro will 'get to correct' faster even with middling prompt(s) vs for a language like Rust.
  - dasil003 1y ago
    Yes, and also writing for an LLM to consume is its own skill with nuances that are in flux as models and tooling improves.
- fhd2 1y ago
  That about sums it up from my experience as well. But as parent said, such takes unfortunately don't get a lot of eyeballs :(
  - fragmede 1y ago
    That was four short paragraphs with basically no details - there's basically nothing for eyeballs to see. If I wrote a history of the world that consisted of "Some people lived and died, some of them were bad, I guess", how many copies do you think I'd sell? Any? What's interesting is the details, and a post giving actual detail of building some app, and the benefits and shortcomings of a specific tool would be of great interest to many. If I ask an LLM to save the date to the database, and then ask it to save the time, do I get two variables in two columns? Does that make sense for my app? Do I have to ask it to refactor? Is it able to do that successfully? Does it drop tables during program initialization? How does the latest model do with designing the entire program? How often does it hallucinate for this set of libraries and prompts? There's quite a bit of variance! If it hallucinated libraries and APIs left and right it would be far less useful. Some people don't even get hallucinations because their prompts are so we'll trod. There are all sorts of interesting details to be learned and shared about these new tools that would get a ton of eyeballs.
- fmbb 1y ago
  LLMs also don’t always generate code that works. And you don’t always know or understand what the product owner wants you to build. Writing code faster is very rarely what you need.
- brazzy 1y ago
  I've just seen that change happen in the a pet project within less than 10 hours of work. I tried vibe-coding something for my own use, your classic "scratch your own itch" project. The first MVP was a one-shot success, really impressive. But as the code grew with every added feature, progress soon slowed down. The AI claimed to have fixed a bug when it didn't. It switched chest several timea back and forth between using a library function and rolling its own implementation, each time claiming to have "simplified" the code and made it "more reliable". With every change, it claimed to have "improved" the code, even when it just added a bunch of shamelessly duplicated shit. One effect I am sure AI will have is to massively excarbate the phenomenon of people who quickly produce a large amount of shitty, unmaintainable code that fulfills half the requirements, and then leave the mess behind for another greenfield project.
  - AaronAPU 1y ago
    We’re so unaccustomed to working with non-deterministic computer tech that rather than acknowledge they are hit-or-miss, everyone just picks one side and goes all-in on it. Which sounds an awful lot like politics.
    - skydhash 1y ago
      > We’re so unaccustomed to working with non-deterministic computer tech The whole history of computer history is about making computing deterministic, after we found out that having generative anything (Recursion, Context-Free grammar,...) is a double-edged sword. So for any known set of inputs, you want the output to be finite and non-zero, and all items having the correct properties.
      - AaronAPU 1y ago
        All of physics was like that until QM
    - norm_namillu 1y ago
      "non-deterministic" is a weird way to say "mangles data 1/3 to 1/2 the time"
- michaelhoney 1y ago
  You're not wrong, in July 2025. But it will get better, and it will not stop getting better when it reaches equal-to-the-best-human level.
  - norm_namillu 1y ago
    yeah and if my airplane keeps accelerating at the rate it does when taking off, it would reach the speed of light in a few months. "this software doesn't work but if we 10x the resources in r&d a couple more times surely it will" is quite the argument to be seen making in public
  - abalashov 1y ago
    No, but I think you're wrong. My child grew 3 inches in the last year alone, according to his latest physical. If I were to adopt your extrapolation methods, he'll soon not only be the tallest human alive, but the tallest structure on the planet.
jandrese 1y ago
I think there is a strong case that experienced developers can not be replaced by AI anytime soon. Where the danger lies is for junior developers fresh out of college. How are they supposed to become experienced developers if an AI can do the grunt work they normally get assigned? It really doesn't help that AI companies are hype machines full of salesmen trying to hype up their product so you buy it. There have been a lot of amazing AI "success stories" that don't hold up under scrutiny.
- elktown 1y ago
  > Where the danger lies is for junior developers [...] if an AI can do the grunt work they normally get assigned? Just because "well, maybe true for junior devs!" is a compromise over "AI will make all programmers obsolete!" it doesn't make it reasonable. It's still an extraordinary claim.
qsort 1y ago
And then you have research that says they're both full of shit. The article is perhaps a bit shallow, but it's spiritually correct: there's a lot of uncertainty and people who claim they figured it all out are mostly spewing nonsense.
- rapind 1y ago
  I mean it's working for me. It's definitely not living up to the hype IMO (market is way overvaluing), but I wouldn't want to work without it any more, which is actually saying a lot. This take is as a senior (greybeard) developer using paid claude-code in the terminal only (I use plain vim for my own code). I'm running my own business while wearing all of the hats, so I'm not as worried about becoming obsolete, but I also have zero motivation to get other businesses using it (I'm not invested into any AI companies)! To be honest, I could probably even be using it better, but I haven't spent much time yak shaving over my setup beyond zen mcp. That being said, I think anyone looking to invest into established AI companies right now is in for a rude awakening. I think it'll be a commodity / utility like cloud computing with tons of competition and not a ton of differentiating features. That's just my opinion though, and I could be horrendously wrong!
  - qsort 1y ago
    I'm also seeing benefits, and I'm on pretty much the same stack as you. The METR paper I'm citing was misquoted a lot (N was small, and it wasn't a favorable setup for AI tools), but the most important finding was that it's very easy to fool ourselves about productivity benefits. Am I going to cancel my Anthropic subscription? Certainly not, but I'm not going to pretend like my setup is The One True Way. The plural of anecdote isn't data. Nobody has this figured out, nobody.
deleted 1y ago
[deleted]
victorbjorklund 1y ago
Or are those people biased by what they want to be true because of their current situation (the dev who dont wanna change how they work and therefore wants AI to not work or the non-technical person who dont wanna learn to code or be dependent on a developer and therefore want it to work)
setr 1y ago
I think the problem with this one is that LLMs are somehow both unreasonably effective as well as unreasonably ineffective. Letting the code editor llms do their suggestions, I’ve gotten a ton of useless boilerplate garbage suggestions, but periodically, a couple times of day, it suggests blocks of code that are far more complete, comprehensive and correct than it has any right to be. Whether you’ll get the high IQ or low IQ LLM on the next suggestion is a crapshoot; how much consideration you give to either outcome (focus on the random instances of brilliance, or the constant stream of bullshit) drives the final perception.
- abalashov 1y ago
  > Whether you’ll get the high IQ or low IQ LLM on the next suggestion is a crapshoot I have heard theories from people--whose ideas I don't ordinarily consider to be in the realm of purely uninformed speculation--that this can vary depending on system-wide GPU load, and is throttled accordingly based on real-time demand by the major LLM providers. Whether it's true, I have no idea.
  - emp17344 1y ago
    Sounds like conspiratorial thinking to come to terms with the fact that LLMs are fundamentally non-deterministic text predictors. You can get dramatically different responses to the same question, and that’s just how the system works.
    - abalashov 1y ago
      That was my reaction as well, but I wondered.
- awfulneutral 1y ago
  That has been my experience too. I recently turned it all off because I decided the amount of times it takes me 5x longer to accomplish something, in addition to the subtle increase of bugs, is not currently worth it. I guess I'll try it again in a few months.

0x500x79 1y ago

I think this is an easy thing to wrap my mind around (since I have been in both camps): AI can generate lots of code very quickly. AI does not generate code that follows taste and or best practices. So in cases where the task is small, easily plannable, within the training corpus, or for a project that doesn't have high stakes it can produce something workable quickly. In larger projects or something that needs maintainability for the future code generation can fall apart or produce subpar results.

logicchains 1y ago
Recent LLMs are fairly good at following instructions, so a lot of the difference comes down to the level of detail and quality of the instructions given. Written communication is a skill for which there's a huge amount of variance among developers, so it's not surprising that different developers get very different results. The relative quality of the LLM's output is determined primarily by the written communication skills of the individuals instructing it.
- NoOn3 1y ago
  It seems to me If you know all these instructions clearly, then you know everything, and it's easy for you to write the code yourself, and you don't need an LLM.
  - logicchains 1y ago
    The amount of typing it takes to describe a solution in English text is often less than the amount of typing needed to actually implement it in code, especially after accounting for boilerplate and unit tests. Not to mention the time spent waiting for the compiler and test harness to run. As a concrete example, the HTTP2.0 spec is way fewer chars long than any HTTP2.0 server implementation, and the C spec is way fewer chars long than any compliant C compiler. The C++ spec is way, way fewer chars long than any compliant C++ compiler.
    - no_wizard 1y ago
      >The amount of typing it takes to describe a solution in English text is often less than the amount of typing needed to actually implement it in code I don't find this to be true. I find describing a solution in English well to be slower than describing the problem in code (IE, by writing tests first) and having that be the structured data that the LLM uses to generate code. Its far faster, from the results I'm seeing plus my own personal experience, to write clear tests which benefit from being a form of structured data that the LLM can analyze. Its the guidance we have given to our engineers at my day job and it has made working with these tools dramatically easier. In some cases, I have found LLM performance to be subpar enough that it is indeed, faster to write it myself. If it has to hold many different pieces of information together, it starts to falter.
    - svantana 1y ago
      I don't think it's so clear-cut. The C spec I found is 4MB and the tcc compiler source code is 1.8MB. It might need some more code to be fully compliant, but it may still be smaller than 4MB. I think the main reason why code bases are much larger is because they contain stuff not covered by the spec (optimization, vendor-specific stuff, etc etc). Personally I'd rather write a compiler than a specification, but to each their own.
    - steveBK123 1y ago
      Use shorter variable names
- mattwad 1y ago
  Yea this! How many devs say "it doesn't do what i expect" did not try to write up a plan of action before it just YOLO'd some new features? We have to learn to use this new tool, but how to do that is still changing all the time.
  - norm_namillu 1y ago
    > We have to learn to use this new tool, but how to do that is still changing all the time. so we need to program in natural language now but targeting some subset of it that is no learnable? and this is better how?
- no_wizard 1y ago
  Corpus also matters. I know Rust developers who aren't getting very good results even with high quality prompts. On the other hand, I helped integrate Cursor as a staff engineer at my current job for all our developers (many hundreds), who primarily work in JavaScript / TypeScript, and even middling prompts will get results that only require refactoring, assuming the LLM doesn't need a ton of context for the code generation (e.g. greenfield or independent features). Our general approach and guidance has been that developers need to write the tests first and have Cursor use that as a basis for what code to generate. This helps prevent atrophy and over time we've find thats where developers add the most value with these tools. I know plenty of developers want to do it the other way (have AI generate the tests) but we've had more issues with that approach. We discourage AI generating everything and having a human edit the output, as it tends to be slower than our chosen approach and more likely to have issues. That said, LLMs still struggle if they need to hold alot of context. For instance, if you have a bunch of files that it needs to understand to also generate code that is worthwhile, particularly if you want it to re-use code.
  - logicchains 1y ago
    >Corpus also matters. I know Rust developers who aren't getting very good results even with high quality prompts. Which model were they using, out of interest? I've gotten decent results for Rust from Gemini 2.5 Pro. Its first attempt will often be disgusting (cloning and other inefficiencies everywhere), but it can be prompted to optimise that afterwards. It also helps a lot to think ahead about lifetimes and explicitly tell it how to structure them, if there might be anything tricky lifetime-wise.
    - no_wizard 1y ago
      No idea. I do know they all have access to Cursor and tried different models, even the more expensive options. What you're describing though, having to go through that elaborate detail really drives to my point though, and I think shows a weakness in these tools that is a hidden cost to scaling their productivity benefits. What I can tell you though both from observation and experience, is that because the corpus for TypeScript / JavaScript is infinitely larger as it stands today, even Gemini 2.5 Pro will 'get to correct' faster even with middling prompt(s) vs for a language like Rust.
- dasil003 1y ago
  Yes, and also writing for an LLM to consume is its own skill with nuances that are in flux as models and tooling improves.
fhd2 1y ago
That about sums it up from my experience as well. But as parent said, such takes unfortunately don't get a lot of eyeballs :(
- fragmede 1y ago
  That was four short paragraphs with basically no details - there's basically nothing for eyeballs to see. If I wrote a history of the world that consisted of "Some people lived and died, some of them were bad, I guess", how many copies do you think I'd sell? Any? What's interesting is the details, and a post giving actual detail of building some app, and the benefits and shortcomings of a specific tool would be of great interest to many. If I ask an LLM to save the date to the database, and then ask it to save the time, do I get two variables in two columns? Does that make sense for my app? Do I have to ask it to refactor? Is it able to do that successfully? Does it drop tables during program initialization? How does the latest model do with designing the entire program? How often does it hallucinate for this set of libraries and prompts? There's quite a bit of variance! If it hallucinated libraries and APIs left and right it would be far less useful. Some people don't even get hallucinations because their prompts are so we'll trod. There are all sorts of interesting details to be learned and shared about these new tools that would get a ton of eyeballs.
fmbb 1y ago
LLMs also don’t always generate code that works. And you don’t always know or understand what the product owner wants you to build. Writing code faster is very rarely what you need.
brazzy 1y ago
I've just seen that change happen in the a pet project within less than 10 hours of work. I tried vibe-coding something for my own use, your classic "scratch your own itch" project. The first MVP was a one-shot success, really impressive. But as the code grew with every added feature, progress soon slowed down. The AI claimed to have fixed a bug when it didn't. It switched chest several timea back and forth between using a library function and rolling its own implementation, each time claiming to have "simplified" the code and made it "more reliable". With every change, it claimed to have "improved" the code, even when it just added a bunch of shamelessly duplicated shit. One effect I am sure AI will have is to massively excarbate the phenomenon of people who quickly produce a large amount of shitty, unmaintainable code that fulfills half the requirements, and then leave the mess behind for another greenfield project.
- AaronAPU 1y ago
  We’re so unaccustomed to working with non-deterministic computer tech that rather than acknowledge they are hit-or-miss, everyone just picks one side and goes all-in on it. Which sounds an awful lot like politics.
  - skydhash 1y ago
    > We’re so unaccustomed to working with non-deterministic computer tech The whole history of computer history is about making computing deterministic, after we found out that having generative anything (Recursion, Context-Free grammar,...) is a double-edged sword. So for any known set of inputs, you want the output to be finite and non-zero, and all items having the correct properties.
    - AaronAPU 1y ago
      All of physics was like that until QM
  - norm_namillu 1y ago
    "non-deterministic" is a weird way to say "mangles data 1/3 to 1/2 the time"
michaelhoney 1y ago
You're not wrong, in July 2025. But it will get better, and it will not stop getting better when it reaches equal-to-the-best-human level.
- norm_namillu 1y ago
  yeah and if my airplane keeps accelerating at the rate it does when taking off, it would reach the speed of light in a few months. "this software doesn't work but if we 10x the resources in r&d a couple more times surely it will" is quite the argument to be seen making in public
- abalashov 1y ago
  No, but I think you're wrong. My child grew 3 inches in the last year alone, according to his latest physical. If I were to adopt your extrapolation methods, he'll soon not only be the tallest human alive, but the tallest structure on the planet.

jandrese 1y ago

I think there is a strong case that experienced developers can not be replaced by AI anytime soon. Where the danger lies is for junior developers fresh out of college. How are they supposed to become experienced developers if an AI can do the grunt work they normally get assigned? It really doesn't help that AI companies are hype machines full of salesmen trying to hype up their product so you buy it. There have been a lot of amazing AI "success stories" that don't hold up under scrutiny.

elktown 1y ago
> Where the danger lies is for junior developers [...] if an AI can do the grunt work they normally get assigned? Just because "well, maybe true for junior devs!" is a compromise over "AI will make all programmers obsolete!" it doesn't make it reasonable. It's still an extraordinary claim.

qsort 1y ago

And then you have research that says they're both full of shit. The article is perhaps a bit shallow, but it's spiritually correct: there's a lot of uncertainty and people who claim they figured it all out are mostly spewing nonsense.

rapind 1y ago
I mean it's working for me. It's definitely not living up to the hype IMO (market is way overvaluing), but I wouldn't want to work without it any more, which is actually saying a lot. This take is as a senior (greybeard) developer using paid claude-code in the terminal only (I use plain vim for my own code). I'm running my own business while wearing all of the hats, so I'm not as worried about becoming obsolete, but I also have zero motivation to get other businesses using it (I'm not invested into any AI companies)! To be honest, I could probably even be using it better, but I haven't spent much time yak shaving over my setup beyond zen mcp. That being said, I think anyone looking to invest into established AI companies right now is in for a rude awakening. I think it'll be a commodity / utility like cloud computing with tons of competition and not a ton of differentiating features. That's just my opinion though, and I could be horrendously wrong!
- qsort 1y ago
  I'm also seeing benefits, and I'm on pretty much the same stack as you. The METR paper I'm citing was misquoted a lot (N was small, and it wasn't a favorable setup for AI tools), but the most important finding was that it's very easy to fool ourselves about productivity benefits. Am I going to cancel my Anthropic subscription? Certainly not, but I'm not going to pretend like my setup is The One True Way. The plural of anecdote isn't data. Nobody has this figured out, nobody.

deleted 1y ago

[deleted]

victorbjorklund 1y ago

Or are those people biased by what they want to be true because of their current situation (the dev who dont wanna change how they work and therefore wants AI to not work or the non-technical person who dont wanna learn to code or be dependent on a developer and therefore want it to work)

setr 1y ago

I think the problem with this one is that LLMs are somehow both unreasonably effective as well as unreasonably ineffective. Letting the code editor llms do their suggestions, I’ve gotten a ton of useless boilerplate garbage suggestions, but periodically, a couple times of day, it suggests blocks of code that are far more complete, comprehensive and correct than it has any right to be. Whether you’ll get the high IQ or low IQ LLM on the next suggestion is a crapshoot; how much consideration you give to either outcome (focus on the random instances of brilliance, or the constant stream of bullshit) drives the final perception.

abalashov 1y ago
> Whether you’ll get the high IQ or low IQ LLM on the next suggestion is a crapshoot I have heard theories from people--whose ideas I don't ordinarily consider to be in the realm of purely uninformed speculation--that this can vary depending on system-wide GPU load, and is throttled accordingly based on real-time demand by the major LLM providers. Whether it's true, I have no idea.
- emp17344 1y ago
  Sounds like conspiratorial thinking to come to terms with the fact that LLMs are fundamentally non-deterministic text predictors. You can get dramatically different responses to the same question, and that’s just how the system works.
  - abalashov 1y ago
    That was my reaction as well, but I wondered.
awfulneutral 1y ago
That has been my experience too. I recently turned it all off because I decided the amount of times it takes me 5x longer to accomplish something, in addition to the subtle increase of bugs, is not currently worth it. I guess I'll try it again in a few months.