6 ms·

Hearing people on tech twitter say that LLMs always produce better code than they do by hand was pretty enlightening for me. LLMs can produce better code for l

Hearing people on tech twitter say that LLMs always produce better code than they do by hand was pretty enlightening for me. LLMs can produce better code for languages and domains I’m not proficient in, at a much faster rate, but damn it’s rare I look at LLM output and don’t spot something I’d do measurably better. These things are average text generation machines. Yes you can improve the output quality by writing a good prompt that activates the right weights, getting you higher quality output. But if you’re seeing output that is consistently better than what you produce by hand, you’re probably just below average at programming. And yes, it matters sometimes. Look at the number of software bugs we’re all subjected to. And let’s not forget that code is a liability. Utilizing code that was “cheap” to generate has a cost, which I’m sure will be the subject of much conversation in the near future.

ch4s3 9mo ago
In my view they’re great for rough drafts, iterating on ideas, throwaway code, and pushing into areas I haven’t become proficient in yet. I think in a lot of cases they write ok enough tests.
christophilus 9mo ago
Claude Code is way better than I am at rummaging through Git history, handling merge conflicts, renaming things, writing SQL queries whose syntax I always forget (window functions and the like). But yeah. If I give it a big, non-specific task, it generates a lot of mediocre (or worse) code.
- throwawayffffas 9mo ago
  That's funny that's all the things I don't trust it to do. I actually use it the other way around, give it a big non-specific task, see if it works, specify better, retry, throw away 60% - 90% of the generated code, fix bugs in a bunch of places and out comes an implemented feature.
  - bryanlarsen 9mo ago
    Agreed. Claude is horrible at munging git history and can destroy the thing I depend on to fix Claude's messes. I always do my git rebasing by hand. The first iteration of Claude code is usually a big over-coded mess, but it's pretty good at iterating to clean it up, given proper instruction.
    - rectang 9mo ago
      I give the agent the following standing instructions: "Make the smallest possible change. Do not refactor existing code unless I explicitly ask." That directive cut down considerably on the amount of extra changes I had to review. When it gets it right, the changes are close to the right size now. The agent still tries to do too much, typically suggesting three tangents for every interaction.
jimbo1167 9mo ago
How are you judging that you'd write "better" code? More maintainable? More efficient? Does it produce bugs in the underlying code it's generating? Genuinely curious where you see the current gaps.
- jordwest 9mo ago
  For me the biggest gaps in LLM code are: - it adds superfluous logic that is assumed but isn’t necessary - as a result the code is more complex, verbose, harder to follow - it doesn’t quite match the domain because it makes a bunch of assumptions that aren’t true in this particular domain They’re things that can often be missed in a first pass look at the code but end up adding a lot of accidental complexity that bites you later. When reading an unfamiliar code base we tend to assume that a certain bit of logic is there for a good reason, and that helps you understand what the system is trying to do. With generative codebases we can’t really assume that anymore unless the code has been thoroughly audited/reviewed/rewritten, at which point I find it’s easier to just write the code myself.
  - CapsAdmin 9mo ago
    This has been my experience as well. But, these are things we developers care about. Coding aside, LLM's aren't very good at following nice practices in general unless explicitly prompted to. For example if you ask an LLM to create an error modal box from scratch, will it also implement the ability to select the text, or being able to ctrl c to copy the text, or perhaps a copy message button? Maybe this is a bad example, but they usually don't do things like this unless you explicitly ask them to. I don't personally care too much about this, but I think it's noteworthy in the context of lay people using LLM's to vibe code.
  - zahlman 9mo ago
    I've seen a lot of examples where it fails to take advantage of previous work and rewrites functionality from scratch.
godzillabrennus 9mo ago
As someone who: 1.) Has a brain that is not wired to think like a computer and write code. 2.) Falls asleep at the keyboard while writing code for more than an hour or two. 3.) Has a lot of passion for sticking with an idea and making it happen, even if that means writing code and knowing the code is crap. So, in short, LLMs write better code than I do. I'm not alone.
- djaouen 9mo ago
  You are defective. But, fear not! So is the rest of humanity!
iwontberude 9mo ago
The most prolific coders are also more competent than average. Their proliferations are what have trained these models. These models are trained on incredibly successful projects written by masters of their fields. This is usually where I find the most pushback is that the most competent SWEs see it as theft and also useless to them since they have already spend years honing skills to work relentlessly and efficiently towards solutions -- sometimes at great expense.
- habinero 9mo ago
  > The most prolific coders are also more competent than average This is absolutely not true lol, as anyone who's worked with a fabled 10X engineer will tell you. It's like saying the best civil engineer is the one that builds the most bridges. The best code looks real boring.
  - iwontberude 9mo ago
    I've worked with a 10x engineer and indeed they were significantly more competent than the rest of the team in their execution and design. They've seen so many problems and had a chance to discard bad patterns and adopt/try out new ones.
    - habinero 9mo ago
      Nah, just better at marketing and coming up with plausible things that make them sound good. You learn to spot that type after a while. There are a rare few people who do get a lot done, but usually it's because they work 80 hour weeks.
- nitwit005 9mo ago
  I'd assume most of the code visible on the web leans amateur. A huge portion of github repos seem to be from students these days. You'll see GitHub's Education page listing "5 million students" (https://github.com/education https://github.com/education), which I assume is an under-estimate, as that's only the formal program.
habinero 9mo ago
I saw someone refer to it as future digital asbestos.
- mrguyorama 9mo ago
  Asbestos is a miracle material with a serious and hard to really avoid downside. It's so good that we are genuinely left with crappy options to replace it, and people have died in fires that could have been saved with the right application of asbestos. Current AI hype is closer to the Radium craze back during the discovery of radioactivity. Yes it's a neat new thing that will have some interesting uses. No don't put it in everything and especially not in your food what are you doing oh my god!
wild_egg 9mo ago
After a certain experience level though, I think most of us get to the point of knowing what that difference in quality actually matters. Some seniors love to bikeshed PRs all day because they can do it better but generally that activity has zero actual value. Sometimes it matters, often it doesn't. Stop with the "I could do this better by hand" and ask "is it worth the extra 4 hours to do this by hand, or is this actually good enough to meet the goals?"
- Arch-TK 9mo ago
  "actually good enough to meet the goals?" There's "okay for now" and then there's "this is so crap that if we set our bar this low we'll be knee deep in tech debt in a month". A lot of LLM output in the specific areas _I_ work in is firmly in that latter category and many times just doesn't work.
  - gbnwl 9mo ago
    So I can tell you don’t use these tools, or at least much, because at the speed of development with them you’ll be knee deep in tech debt in a day, not a month, but as a corollary can have the same agentic coding tools undergo the equivalent of weeks of addressing tech debt the next day. Well, I think this applies to greenfield AI-first oriented projects that work this way from the get go and with few humans in the loop (human to human communication definitely becomes the rate limiting step). But I imagine that’s not the nature of your work.
    - daveguy 9mo ago
      I think you missed the your parent post's phrase "in the specific areas _I_ work in" ... LLMs are a lot better at crud and boilerplate than novel hardware interfaces and a bunch of other domains.
      - gbnwl 9mo ago
        But why would it take a month to generate significant tech debt in novel domains, it would accrue even faster then right? The main idea I wanted to get across is that iteration speed is much faster so what's "tech debt" in the first pass, can be addressed much faster in future passes, which will happen on the order of days rather than sprints in the older paradigm. Yes the first iterations will have a bunch of issues but if you keep your hands on the controller you can get things to a decent state quickly. I think one of the biggest gaps I see in devs using these tools is what they do after the first pass. Also, even for novel domains, using tools like deep research and the ability of these tools to straight up search through the internet, including public repos during the planning phase (you should be planning first before implementing right? You're not just opening a window and asking in a few sentences for a vaguely defined final product I hope) is a huge level up. If there are repos, papers, articles, etc of your novel domain out there, there's a path to a successful research -> plan -> implement -> iterate path out there imo, especially when you get better at giving the tools ways to evaluate their own results, rather than going back and forth yourself for hours telling them "no, this part is wrong, no now this part is wrong, etc etc"
abighamb 9mo ago
This has been my experience as well. It's let me apply my general knowledge across domains, and do things in tech stacks or languages I don't know well. But that has also cost me hours debugging a solution I don't quite understand. When working in my core stack though it's a nice force multiplier for routine changes.
- logicallee 9mo ago
  >When working in my core stack though it's a nice force multiplier for routine changes. what's your core stack?
CapsAdmin 9mo ago
I've been playing with vibe coding a lot lately and I think in most cases, the current SOTA LLM's don't produce code that I'd be satisfied with. I kind of feel like LLM's are really really good at hacking on a messy and fragile structure, because they can "keep track many things in their head" BUT An LLM can write a PNG decoder that works in whatever language I choose in one or a few shots. I can do that too, but it will take me longer than a minute! (and I might learn something about the png format that might be useful later..) Also, us engineers can talk about code quality all day, but does this really matter to non-engineers? Maybe objectively it does, but can we convince them that it does?
- raddan 9mo ago
  I tried vibe coding a BMP decoder not too long ago with the rationale being “what’s simpler than BMP?” What I got was an absolute mess that did not work at all. Perhaps this was because, in retrospect, BMP is not actually all that simple, a fact that I discovered when I did write a BMP decoder by hand. But I spent equal time vibe coding and real coding. At the end of the real coding session, I understood BMP, which I see as a benefit unto itself. This is perhaps a bit cynical but my hot take on vibe coders is that they place little value on understanding things.
  - gbnwl 9mo ago
    Mind explaining the process you tried? As someone who’s generally not had any issue getting LLMs to sort out my side projects (ofc with my active involvement as well), I really wonder what people who report these results are trying. Did you just open a chat with claude code and try to get a single context window to one shot it?
  - zahlman 9mo ago
    Just out of curiousity (as someone fairly familiar with the BMP spec, and also PNG incidentally): what did you find to be the trickiest/most complex aspects?
    - raddan 9mo ago
paodealho 9mo ago
I worked for a relatively large company (around 400 employees there are programmers). The people who embraced LLM-generated code clearly share one trait: they are feature pushers who love to say "yes!" to management. You see, management is always right, and these programmers are always so eager to put their requirements, however incomplete, into a Copilot session and open a pull request as fast as possible. The worst case I remember happened a few months ago when a staff (!) engineer gave a presentation about benchmarks they had done between Java and Kotlin concurrency tools and how to write concurrent code. There was a very large and strange difference in performance favoring Kotlin that didn't make sense. When I dug into their code, it was clear everything had been generated by a LLM (lots of comments with emojis, for example) and the Java code was just wrong. The competent programmers I've seen there use LLMs to generate some shell scripts, small python automations or to explore ideas. Most of the time they are unimpressed by these tools.
throwawayffffas 9mo ago
> Hearing people on tech twitter say that LLMs always produce better code than they do by hand was pretty enlightening for me. That's hilarious LLM code is always very bad. It's only merit is it occasionally works. > LLMs can produce better code for languages and domains I’m not proficient in. I am sure that's not true.
- ambicapter 9mo ago
  It seems true by construction. If you're not proficient in a language than the bar for "better than you" is necessarily lower.
  - throwawayffffas 9mo ago
    If you are an experienced developer in general your code will in general be better than an LLMs even in languages you are not proficient. The LLM code might be more in keeping with the languages conventions but that does not make the code automatically superior. LLMs typically produce bad code in other dimensions.
- caycep 9mo ago
  I think it says more about who's still on tech twitter vs. anything about the llm....
nitwit005 9mo ago
It'd be rather surprising if you could train an AI on a bunch of average code, and somehow get code that's always above average. Where did the improvement come from? We should feed the output code back in to get even better code.
- zahlman 9mo ago
  AI generally can improve through reinforcement learning, but this requires it to be able to compare its output to some form of metric. There aren't a lot of people I'd trust to RLHF for code quality, and anything more automated than that is destined to collapse due to Goodhart's Law.
bdangubic 9mo ago
> if you’re seeing output that is consistently better than what you produce by hand, you’re probably just below average at programming even though this statement does not mathematically / statistically make sense - vast majority of SWEs are “below average.” therein lies the crux of this debate. I’ve been coding since the 90’s and: - LLM output is better than mine from the 90’s - LLM output is better than mine from early 2000’s - LLM output is worse than any of mine from 2010 onward - LLM output (in the right hands) is better than 90% of human-written code I have seen (and I’ve seen a lot)
- xyzsparetimexyz 9mo ago
  That's not how averages work
kokanee 9mo ago
> These things are average text generation machines. Funny... seems like about half of devs think AI writes good code, and half think it doesn't. When you consider that it is designed to replicate average output, that makes a lot of sense. So, as insulting as OP's idea is, it would make sense that below-average devs are getting gains by using AI, and above-average devs aren't. In theory, this situation should raise the average output quality, but only if the training corpus isn't poisoned with AI output. I have an anecdote that doesn't mean much on its own, but supports OP's thesis: there are two former coworkers in my linkedin feed who are heavy AI evangelists, and have drifted over the years from software engineering into senior business development roles at AI startups. Both of them are unquestionably in the top 5 worst coders I have ever worked with in 15 years, one of them having been fired for code quality and testing practices. Their coding ability, transition to less technical roles, and extremely vocal support for the power of vibe coding definitely would align with OP's uncharitable character evaluation.
- NomDePlum 9mo ago
  > it would make sense that below-average devs are getting gains by using AI They are certainly opening more PRs. Being the gate and last safety check on the PRs is certainly driving me in the opposite direction.
rytill 9mo ago
LLMs are not "average text generation machines" once they have context. LLMs learn a distribution. The moment you start the prompt with "You are an interactive CLI tool that helps users with software engineering at the level of a veteran expert" you have biased the LLM such that the tokens it produces are from a very non-average part of the distribution it's modeling.
- jason_oster 9mo ago
  True, but nuanced. The model does not react to "you are an experienced programmer" kinds of prompts. It reacts to being given relevant information that needs to be reflected in the output. See examples in https://arxiv.org/abs/2305.14688 https://arxiv.org/abs/2305.14688; They certainly do say things like "You are a physicist specialized in atomic structure ...", but the important point is that the rest of the "expert persona" prompt _calls attention to key details_ that improves the response. The hint about electromagnetic forces in the expert persona prompt is what tipped off the model to mention it in the output. Bringing attention to key details is what makes this work. A great tip for anyone who wants to micromanage code with an LLM is to include precise details about what they wish to micromanage: say "store it in a hash map keyed by unsigned integers" instead of letting the model decide which data structure to use.
- xyzsparetimexyz 9mo ago
  Speak english
moron4hire 9mo ago
If firing up old coal plants and skyrocketing RAM prices and $5000 consumer GPUs and violating millions of developers' copyrights and occasionally coaxing someone into killing themselves is the cost of Brian Who Got Peter Principled Into Middle Management getting to enjoy programming again instead of blaming his kids for why he watches football and drinks all weekend instead of cultivating a hobby, I guess we have no choice but to oblige him his little treat.
- xyzsparetimexyz 9mo ago
  Well said

ch4s3 9mo ago

In my view they’re great for rough drafts, iterating on ideas, throwaway code, and pushing into areas I haven’t become proficient in yet. I think in a lot of cases they write ok enough tests.

christophilus 9mo ago

Claude Code is way better than I am at rummaging through Git history, handling merge conflicts, renaming things, writing SQL queries whose syntax I always forget (window functions and the like). But yeah. If I give it a big, non-specific task, it generates a lot of mediocre (or worse) code.

throwawayffffas 9mo ago
That's funny that's all the things I don't trust it to do. I actually use it the other way around, give it a big non-specific task, see if it works, specify better, retry, throw away 60% - 90% of the generated code, fix bugs in a bunch of places and out comes an implemented feature.
- bryanlarsen 9mo ago
  Agreed. Claude is horrible at munging git history and can destroy the thing I depend on to fix Claude's messes. I always do my git rebasing by hand. The first iteration of Claude code is usually a big over-coded mess, but it's pretty good at iterating to clean it up, given proper instruction.
  - rectang 9mo ago
    I give the agent the following standing instructions: "Make the smallest possible change. Do not refactor existing code unless I explicitly ask." That directive cut down considerably on the amount of extra changes I had to review. When it gets it right, the changes are close to the right size now. The agent still tries to do too much, typically suggesting three tangents for every interaction.

jimbo1167 9mo ago

How are you judging that you'd write "better" code? More maintainable? More efficient? Does it produce bugs in the underlying code it's generating? Genuinely curious where you see the current gaps.

jordwest 9mo ago
For me the biggest gaps in LLM code are: - it adds superfluous logic that is assumed but isn’t necessary - as a result the code is more complex, verbose, harder to follow - it doesn’t quite match the domain because it makes a bunch of assumptions that aren’t true in this particular domain They’re things that can often be missed in a first pass look at the code but end up adding a lot of accidental complexity that bites you later. When reading an unfamiliar code base we tend to assume that a certain bit of logic is there for a good reason, and that helps you understand what the system is trying to do. With generative codebases we can’t really assume that anymore unless the code has been thoroughly audited/reviewed/rewritten, at which point I find it’s easier to just write the code myself.
- CapsAdmin 9mo ago
  This has been my experience as well. But, these are things we developers care about. Coding aside, LLM's aren't very good at following nice practices in general unless explicitly prompted to. For example if you ask an LLM to create an error modal box from scratch, will it also implement the ability to select the text, or being able to ctrl c to copy the text, or perhaps a copy message button? Maybe this is a bad example, but they usually don't do things like this unless you explicitly ask them to. I don't personally care too much about this, but I think it's noteworthy in the context of lay people using LLM's to vibe code.
- zahlman 9mo ago
  I've seen a lot of examples where it fails to take advantage of previous work and rewrites functionality from scratch.

godzillabrennus 9mo ago

As someone who: 1.) Has a brain that is not wired to think like a computer and write code. 2.) Falls asleep at the keyboard while writing code for more than an hour or two. 3.) Has a lot of passion for sticking with an idea and making it happen, even if that means writing code and knowing the code is crap. So, in short, LLMs write better code than I do. I'm not alone.

djaouen 9mo ago
You are defective. But, fear not! So is the rest of humanity!

iwontberude 9mo ago

The most prolific coders are also more competent than average. Their proliferations are what have trained these models. These models are trained on incredibly successful projects written by masters of their fields. This is usually where I find the most pushback is that the most competent SWEs see it as theft and also useless to them since they have already spend years honing skills to work relentlessly and efficiently towards solutions -- sometimes at great expense.

habinero 9mo ago
> The most prolific coders are also more competent than average This is absolutely not true lol, as anyone who's worked with a fabled 10X engineer will tell you. It's like saying the best civil engineer is the one that builds the most bridges. The best code looks real boring.
- iwontberude 9mo ago
  I've worked with a 10x engineer and indeed they were significantly more competent than the rest of the team in their execution and design. They've seen so many problems and had a chance to discard bad patterns and adopt/try out new ones.
  - habinero 9mo ago
    Nah, just better at marketing and coming up with plausible things that make them sound good. You learn to spot that type after a while. There are a rare few people who do get a lot done, but usually it's because they work 80 hour weeks.
nitwit005 9mo ago
I'd assume most of the code visible on the web leans amateur. A huge portion of github repos seem to be from students these days. You'll see GitHub's Education page listing "5 million students" (https://github.com/education https://github.com/education), which I assume is an under-estimate, as that's only the formal program.

habinero 9mo ago

I saw someone refer to it as future digital asbestos.

mrguyorama 9mo ago
Asbestos is a miracle material with a serious and hard to really avoid downside. It's so good that we are genuinely left with crappy options to replace it, and people have died in fires that could have been saved with the right application of asbestos. Current AI hype is closer to the Radium craze back during the discovery of radioactivity. Yes it's a neat new thing that will have some interesting uses. No don't put it in everything and especially not in your food what are you doing oh my god!

wild_egg 9mo ago

After a certain experience level though, I think most of us get to the point of knowing what that difference in quality actually matters. Some seniors love to bikeshed PRs all day because they can do it better but generally that activity has zero actual value. Sometimes it matters, often it doesn't. Stop with the "I could do this better by hand" and ask "is it worth the extra 4 hours to do this by hand, or is this actually good enough to meet the goals?"

Arch-TK 9mo ago
"actually good enough to meet the goals?" There's "okay for now" and then there's "this is so crap that if we set our bar this low we'll be knee deep in tech debt in a month". A lot of LLM output in the specific areas _I_ work in is firmly in that latter category and many times just doesn't work.
- gbnwl 9mo ago
  So I can tell you don’t use these tools, or at least much, because at the speed of development with them you’ll be knee deep in tech debt in a day, not a month, but as a corollary can have the same agentic coding tools undergo the equivalent of weeks of addressing tech debt the next day. Well, I think this applies to greenfield AI-first oriented projects that work this way from the get go and with few humans in the loop (human to human communication definitely becomes the rate limiting step). But I imagine that’s not the nature of your work.
  - daveguy 9mo ago
    I think you missed the your parent post's phrase "in the specific areas _I_ work in" ... LLMs are a lot better at crud and boilerplate than novel hardware interfaces and a bunch of other domains.
    - gbnwl 9mo ago
      But why would it take a month to generate significant tech debt in novel domains, it would accrue even faster then right? The main idea I wanted to get across is that iteration speed is much faster so what's "tech debt" in the first pass, can be addressed much faster in future passes, which will happen on the order of days rather than sprints in the older paradigm. Yes the first iterations will have a bunch of issues but if you keep your hands on the controller you can get things to a decent state quickly. I think one of the biggest gaps I see in devs using these tools is what they do after the first pass. Also, even for novel domains, using tools like deep research and the ability of these tools to straight up search through the internet, including public repos during the planning phase (you should be planning first before implementing right? You're not just opening a window and asking in a few sentences for a vaguely defined final product I hope) is a huge level up. If there are repos, papers, articles, etc of your novel domain out there, there's a path to a successful research -> plan -> implement -> iterate path out there imo, especially when you get better at giving the tools ways to evaluate their own results, rather than going back and forth yourself for hours telling them "no, this part is wrong, no now this part is wrong, etc etc"

abighamb 9mo ago

This has been my experience as well. It's let me apply my general knowledge across domains, and do things in tech stacks or languages I don't know well. But that has also cost me hours debugging a solution I don't quite understand. When working in my core stack though it's a nice force multiplier for routine changes.

logicallee 9mo ago
>When working in my core stack though it's a nice force multiplier for routine changes. what's your core stack?

CapsAdmin 9mo ago

I've been playing with vibe coding a lot lately and I think in most cases, the current SOTA LLM's don't produce code that I'd be satisfied with. I kind of feel like LLM's are really really good at hacking on a messy and fragile structure, because they can "keep track many things in their head" BUT An LLM can write a PNG decoder that works in whatever language I choose in one or a few shots. I can do that too, but it will take me longer than a minute! (and I might learn something about the png format that might be useful later..) Also, us engineers can talk about code quality all day, but does this really matter to non-engineers? Maybe objectively it does, but can we convince them that it does?

raddan 9mo ago
I tried vibe coding a BMP decoder not too long ago with the rationale being “what’s simpler than BMP?” What I got was an absolute mess that did not work at all. Perhaps this was because, in retrospect, BMP is not actually all that simple, a fact that I discovered when I did write a BMP decoder by hand. But I spent equal time vibe coding and real coding. At the end of the real coding session, I understood BMP, which I see as a benefit unto itself. This is perhaps a bit cynical but my hot take on vibe coders is that they place little value on understanding things.
- gbnwl 9mo ago
  Mind explaining the process you tried? As someone who’s generally not had any issue getting LLMs to sort out my side projects (ofc with my active involvement as well), I really wonder what people who report these results are trying. Did you just open a chat with claude code and try to get a single context window to one shot it?
- zahlman 9mo ago
  Just out of curiousity (as someone fairly familiar with the BMP spec, and also PNG incidentally): what did you find to be the trickiest/most complex aspects?
  - raddan 9mo ago

paodealho 9mo ago

I worked for a relatively large company (around 400 employees there are programmers). The people who embraced LLM-generated code clearly share one trait: they are feature pushers who love to say "yes!" to management. You see, management is always right, and these programmers are always so eager to put their requirements, however incomplete, into a Copilot session and open a pull request as fast as possible. The worst case I remember happened a few months ago when a staff (!) engineer gave a presentation about benchmarks they had done between Java and Kotlin concurrency tools and how to write concurrent code. There was a very large and strange difference in performance favoring Kotlin that didn't make sense. When I dug into their code, it was clear everything had been generated by a LLM (lots of comments with emojis, for example) and the Java code was just wrong. The competent programmers I've seen there use LLMs to generate some shell scripts, small python automations or to explore ideas. Most of the time they are unimpressed by these tools.

throwawayffffas 9mo ago

> Hearing people on tech twitter say that LLMs always produce better code than they do by hand was pretty enlightening for me. That's hilarious LLM code is always very bad. It's only merit is it occasionally works. > LLMs can produce better code for languages and domains I’m not proficient in. I am sure that's not true.

ambicapter 9mo ago
It seems true by construction. If you're not proficient in a language than the bar for "better than you" is necessarily lower.
- throwawayffffas 9mo ago
  If you are an experienced developer in general your code will in general be better than an LLMs even in languages you are not proficient. The LLM code might be more in keeping with the languages conventions but that does not make the code automatically superior. LLMs typically produce bad code in other dimensions.
caycep 9mo ago
I think it says more about who's still on tech twitter vs. anything about the llm....

nitwit005 9mo ago

It'd be rather surprising if you could train an AI on a bunch of average code, and somehow get code that's always above average. Where did the improvement come from? We should feed the output code back in to get even better code.

zahlman 9mo ago
AI generally can improve through reinforcement learning, but this requires it to be able to compare its output to some form of metric. There aren't a lot of people I'd trust to RLHF for code quality, and anything more automated than that is destined to collapse due to Goodhart's Law.

bdangubic 9mo ago

> if you’re seeing output that is consistently better than what you produce by hand, you’re probably just below average at programming even though this statement does not mathematically / statistically make sense - vast majority of SWEs are “below average.” therein lies the crux of this debate. I’ve been coding since the 90’s and: - LLM output is better than mine from the 90’s - LLM output is better than mine from early 2000’s - LLM output is worse than any of mine from 2010 onward - LLM output (in the right hands) is better than 90% of human-written code I have seen (and I’ve seen a lot)

xyzsparetimexyz 9mo ago
That's not how averages work

kokanee 9mo ago

> These things are average text generation machines. Funny... seems like about half of devs think AI writes good code, and half think it doesn't. When you consider that it is designed to replicate average output, that makes a lot of sense. So, as insulting as OP's idea is, it would make sense that below-average devs are getting gains by using AI, and above-average devs aren't. In theory, this situation should raise the average output quality, but only if the training corpus isn't poisoned with AI output. I have an anecdote that doesn't mean much on its own, but supports OP's thesis: there are two former coworkers in my linkedin feed who are heavy AI evangelists, and have drifted over the years from software engineering into senior business development roles at AI startups. Both of them are unquestionably in the top 5 worst coders I have ever worked with in 15 years, one of them having been fired for code quality and testing practices. Their coding ability, transition to less technical roles, and extremely vocal support for the power of vibe coding definitely would align with OP's uncharitable character evaluation.

NomDePlum 9mo ago
> it would make sense that below-average devs are getting gains by using AI They are certainly opening more PRs. Being the gate and last safety check on the PRs is certainly driving me in the opposite direction.

rytill 9mo ago

LLMs are not "average text generation machines" once they have context. LLMs learn a distribution. The moment you start the prompt with "You are an interactive CLI tool that helps users with software engineering at the level of a veteran expert" you have biased the LLM such that the tokens it produces are from a very non-average part of the distribution it's modeling.

jason_oster 9mo ago
True, but nuanced. The model does not react to "you are an experienced programmer" kinds of prompts. It reacts to being given relevant information that needs to be reflected in the output. See examples in https://arxiv.org/abs/2305.14688 https://arxiv.org/abs/2305.14688; They certainly do say things like "You are a physicist specialized in atomic structure ...", but the important point is that the rest of the "expert persona" prompt _calls attention to key details_ that improves the response. The hint about electromagnetic forces in the expert persona prompt is what tipped off the model to mention it in the output. Bringing attention to key details is what makes this work. A great tip for anyone who wants to micromanage code with an LLM is to include precise details about what they wish to micromanage: say "store it in a hash map keyed by unsigned integers" instead of letting the model decide which data structure to use.
xyzsparetimexyz 9mo ago
Speak english

moron4hire 9mo ago

If firing up old coal plants and skyrocketing RAM prices and $5000 consumer GPUs and violating millions of developers' copyrights and occasionally coaxing someone into killing themselves is the cost of Brian Who Got Peter Principled Into Middle Management getting to enjoy programming again instead of blaming his kids for why he watches football and drinks all weekend instead of cultivating a hobby, I guess we have no choice but to oblige him his little treat.

xyzsparetimexyz 9mo ago
Well said