6 ms·

Well yeah. And because when an expert looks at the code chatgpt produces, the flaws are more obvious. It programs with the skill of the median programmer on Git

by josephg 11mo ago

Well yeah. And because when an expert looks at the code chatgpt produces, the flaws are more obvious. It programs with the skill of the median programmer on GitHub. For beginners and people who do cookie cutter work, this can be incredible because it writes the same or better code they could write, fast and for free. But for experts, the code it produces is consistently worse than what we can do. At best my pride demands I fix all its flaws before shipping. More commonly, it’s a waste of time to ask it to help, and I need to code the solution from scratch myself anyway. I use it for throwaway prototypes and demos. And whenever I’m thrust into a language I don’t know that well, or to help me debug weird issues outside my area of expertise. But when I go deep on a problem, it’s often worse than useless.

ethbr1 11mo ago
This is why AI is the perfect management Rorschach test. To management (out of IC roles for long enough to lose their technical expertise), it looks perfect! To ICs, the flaws are apparent! So inevitably management greenlights new AI projects* and behaviors, and then everyone is in the 'This was my idea, so it can't fail' CYA scenario. * Add in a dash of management consulting advice here, and note that management consultants' core product was already literally 'something that looks plausible enough to make execs spend money on it'
panarky 11mo ago
> It programs with the skill of the median programmer on GitHub This is a common intuition but it's provably false. The fact that LLMs are trained on a corpus does not mean their output represents the median skill level of the corpus. Eighteen months ago GPT-4 was outperforming 85% of human participants in coding contests. And people who participate in coding contests are already well above the median skill level on Github. And capability has gone way up in the last 18 months.
- sethherr 11mo ago
  Trying to figure out how to align this with my experiences (which match the parents’ comment), and I have an idea: Coding contests are not like my job at all. My job is taking fuzzy human things and making code that solves it. Frankly AI isn’t good at closing open issues on open source projects either.
- incrudible 11mo ago
  > The fact that LLMs are trained on a corpus does not mean their output represents the median skill level of the corpus. It does, by default. Try asking ChatGPT to implement quicksort in JavaScript, the result will be dogshit. Of course it can do better if you guide it, but that implies you recognize dogshit, or at least that you use some sort of prompting technique that will veer it off the beaten path.
  - keeda 11mo ago
    I asked the free version of ChatGPT to implement quicksort in JS. I can't really see much wrong with it, but maybe I'm missing something? (Ugh, I just can't get HN to format code right... pastebin here: https://pastebin.com/tjaibW1x https://pastebin.com/tjaibW1x) ---- function quickSortInPlace(arr, left = 0, right = arr.length - 1) { if (left < right) { const pivotIndex = partition(arr, left, right); quickSortInPlace(arr, left, pivotIndex - 1); quickSortInPlace(arr, pivotIndex + 1, right); } return arr; } function partition(arr, left, right) { const pivot = arr[right]; let i = left; for (let j = left; j < right; j++) { if (arr[j] < pivot) { [arr[i], arr[j]] = [arr[j], arr[i]]; i++; } } [arr[i], arr[right]] = [arr[right], arr[i]]; // Move pivot into place return i; }
    - josephg 11mo ago
      This is exactly the level of code I've come to expect from chatgpt. Its about the level of code I'd want from a smart CS student. But I'd hope to never use this in production: - It always uses the last item as a pivot, which will give it pathological O(n^2) performance if the list is sorted. Passing an already sorted list to a sort function is a very common case. Good quicksort implementations will use a random pivot, or at least the middle pivot so re-sorting lists is fast. - If you pass already sorted data, the recursive call to quickSortInPlace will take up stack space proportional to the size of the array. So if you pass a large sorted array, not only will the function take n^2 time, it might also generate a stack overflow and crash. - This code: ... = [arr[j], arr[i]]; Creates an array and immediately destructures it. This is - or at least used to be - quite slow. I'd avoid doing that in the body of quicksort's inner loop. - There's no way to pass a custom comparator, which is essential in real code. I just tried in firefox: // Sort an array of 1 million sorted elements arr = Array(1e6).fill(0).map((_, i) => i) console.time('x') quickSortInPlace(arr) console.timeEnd('x') My computer ran for about a minute then the javascript virtual machine crashed: Uncaught InternalError: too much recursion This is about the quality of quicksort implementation I'd expect to see in a CS class, or in a random package in npm. If someone on my team committed this, I'd tell them to go rewrite it properly. (Or just use a standard library function - which wouldn't have these flaws.)
- usefulcat 11mo ago
  The best argument I've yet heard against the effectiveness of AI tools for SW dev is the absence of an explosion of shovelware over the past 1-2 years. https://mikelovesrobots.substack.com/p/wheres-the-shovelware-why-ai-coding https://mikelovesrobots.substack.com/p/wheres-the-shovelware... Basically, if the tools are even half as good as some proponents claim, wouldn't you expect at least a significant increase in simple games on Steam or apps in app stores over that time frame? But we're not seeing that.
  - Miraste 11mo ago
    Interesting approach. I can think of one more explanation the author didn't consider: what if software development time wasn't the bottleneck to what he analyzed? The chart for Google Play app submissions, for example, goes down because Google made it much more difficult to publish apps on their store in ways unrelated to software quality. In that case, it wouldn't matter whether AI tools could write a billion production-ready apps, because the limiting factor is Google's submission requirements.
    - parineum 11mo ago
      There are other charts besides Google play. Particularly insightful id the steam chart as steam is already full of shovelware and, in my experience, many developers wish they were making games but the pay is bad. GitHub repos is pretty interesting too but it could be that people just aren't committing this stuff. Showing zero increase is unexpected though.
  - zeckalpha 11mo ago
    Interesting, I would make the exact opposite conclusion from the same data: if AI coding was that bad, we'd see more crapware.
  - aydyn 11mo ago
    Are you sure we aren't seeing an increase in steam games? Charts I'm looking at show a mild exponential around 2024 https://www.statista.com/statistics/552623/number-games-released-steam/ https://www.statista.com/statistics/552623/number-games-rele... Also theres probably a bottleneck in manual review time.
- bpicolo 11mo ago
  Algorithmic coding contests are not an equivalent skillset to professional software development
- voidhorse 11mo ago
  Coding Contest != Software Engineering Or even solving problems that business need to solve, generally speaking. This complete misunderstand of what software engineering even is is the major reason so many engineers are fed up with the clueless leaders foisting AI tools upon their orgs because they apparently lack the critical reasoning skills to be able to distinguish marketing speak from reality.
- josephg 11mo ago
  I don't think this disproves my claim, for several reasons. First, I don't know where those human participants came from, but if you pick people off the street or from a college campus, they aren't going to be the world's best programmers. On the other hand, github users are on average more skilled than the average CS student. Even students and beginners who use github usually don't have much code there. If the LLMs are weighted to treat every line of code about same, they'd pick up more lines of code from prolific developers (who are often more experienced) than they would from beginners. Also in a coding contest, you're under time pressure. Even when your code works, its often ugly and thrown together. On github, the only code I check in is code that solves whatever problem I set out to solve. I suspect everyone writes better code on github than we do in programming competitions. I suspect if you gave the competitors functionally unlimited time to do the programming competition, many more would outperform GPT-4. Programming contests also usually require that you write a fully self contained program which has been very well specified. The program usually doesn't need any error handling, or need to be maintained. (And if it does need error handling, the cases are all fully specified in the problem description). Relatively speaking, LLMs are pretty good at these kind of problems - where I want some throwaway code that'll work today and get deleted tomorrow. But most software I write isn't like that. And LLMs struggle to write maintainable software in large projects. Most problems aren't so well specified. And for most code, you end up spending more effort maintaining the code over its lifetime than it takes to write in the first place. Chatgpt usually writes code that is a headache to maintain. It doesn't write or use local utility functions. It doesn't factor its code well. The code is often overly verbose. It often writes code that's very poorly optimized. Or the code contains quite obvious bugs for unexpected input - like overflow errors or boundary conditions. And the code it produces very rarely handles errors correctly. None of these problems really matter in programming competitions. But it does matter a lot more when writing real software. These problems make LLMs much less useful at work.
- astrange 11mo ago
  Chess AI trained at specific human levels performs better than any humans at those levels, because the random mistakes get averaged out. https://www.maiachess.com https://www.maiachess.com
torginus 11mo ago
In my experience (with ChatGPT 5.1 as of late) is that the AI follows a problem->solution internal logic and doesn't think and try to structure its code. If you ask for an endpoint to a CRUD API, it'll make one. If you ask for 5, it'll repeat the same code 5 times and modify it for the use case. A dev wouldn't do this, they would try to figure out the common parts of code, pull them out into helpers, and try to make as little duplicated code as possible. I feel like the AI has a strong bias towards adding things, and not removing them. The most obviously wrong thing is with CSS - when I try to do some styling, it gets 90% of the way there, but there's almost always something that's not quite right. Then I tell the AI to fix a style, since that div is getting clipped or not correctly centered etc. It almost always keeps adding properties, and after 2-3 tries and an incredibly bloated style, I delete the thing and take a step back and think logically about how to properly lay this out with flexbox.
- BurningFrog 11mo ago
  Bad code is only really bad if it needs to be maintained. If your AI reliably generates working code from a detailed prompt, the prompt is now the source that needs to be maintained. There is no important reason to even look at the generated code
  - AlexandrB 11mo ago
    I think this might be plausible in the future, but it needs a lot more tooling. For starters you need to be able to run the prompt through the exact same model so you can reproduce a "build".
    - josephg 11mo ago
      Even the exact same model isn't enough. There are several sources of nondeterminism in LLMs. These would all need to be squashed or seeded - which as far as I know isn't a feature that openai / anthropic / etc provide.
    - BurningFrog 11mo ago
      OK, then the current models aren't as good as I thought/hoped. I guess one thing it means is that we still need extensive test suites. I suppose an LLM can write those too.
malfist 11mo ago
By volume the vast majority of code on github is students. Think about that when you average github for ai
- pseudalopex 11mo ago
  > By volume the vast majority of code on github is students. Who determined this? How?

by josephg 11mo ago

ethbr1 11mo ago

This is why AI is the perfect management Rorschach test. To management (out of IC roles for long enough to lose their technical expertise), it looks perfect! To ICs, the flaws are apparent! So inevitably management greenlights new AI projects* and behaviors, and then everyone is in the 'This was my idea, so it can't fail' CYA scenario. * Add in a dash of management consulting advice here, and note that management consultants' core product was already literally 'something that looks plausible enough to make execs spend money on it'

panarky 11mo ago

> It programs with the skill of the median programmer on GitHub This is a common intuition but it's provably false. The fact that LLMs are trained on a corpus does not mean their output represents the median skill level of the corpus. Eighteen months ago GPT-4 was outperforming 85% of human participants in coding contests. And people who participate in coding contests are already well above the median skill level on Github. And capability has gone way up in the last 18 months.

sethherr 11mo ago
Trying to figure out how to align this with my experiences (which match the parents’ comment), and I have an idea: Coding contests are not like my job at all. My job is taking fuzzy human things and making code that solves it. Frankly AI isn’t good at closing open issues on open source projects either.
incrudible 11mo ago
> The fact that LLMs are trained on a corpus does not mean their output represents the median skill level of the corpus. It does, by default. Try asking ChatGPT to implement quicksort in JavaScript, the result will be dogshit. Of course it can do better if you guide it, but that implies you recognize dogshit, or at least that you use some sort of prompting technique that will veer it off the beaten path.
- keeda 11mo ago
  I asked the free version of ChatGPT to implement quicksort in JS. I can't really see much wrong with it, but maybe I'm missing something? (Ugh, I just can't get HN to format code right... pastebin here: https://pastebin.com/tjaibW1x https://pastebin.com/tjaibW1x) ---- function quickSortInPlace(arr, left = 0, right = arr.length - 1) { if (left < right) { const pivotIndex = partition(arr, left, right); quickSortInPlace(arr, left, pivotIndex - 1); quickSortInPlace(arr, pivotIndex + 1, right); } return arr; } function partition(arr, left, right) { const pivot = arr[right]; let i = left; for (let j = left; j < right; j++) { if (arr[j] < pivot) { [arr[i], arr[j]] = [arr[j], arr[i]]; i++; } } [arr[i], arr[right]] = [arr[right], arr[i]]; // Move pivot into place return i; }
  - josephg 11mo ago
    This is exactly the level of code I've come to expect from chatgpt. Its about the level of code I'd want from a smart CS student. But I'd hope to never use this in production: - It always uses the last item as a pivot, which will give it pathological O(n^2) performance if the list is sorted. Passing an already sorted list to a sort function is a very common case. Good quicksort implementations will use a random pivot, or at least the middle pivot so re-sorting lists is fast. - If you pass already sorted data, the recursive call to quickSortInPlace will take up stack space proportional to the size of the array. So if you pass a large sorted array, not only will the function take n^2 time, it might also generate a stack overflow and crash. - This code: ... = [arr[j], arr[i]]; Creates an array and immediately destructures it. This is - or at least used to be - quite slow. I'd avoid doing that in the body of quicksort's inner loop. - There's no way to pass a custom comparator, which is essential in real code. I just tried in firefox: // Sort an array of 1 million sorted elements arr = Array(1e6).fill(0).map((_, i) => i) console.time('x') quickSortInPlace(arr) console.timeEnd('x') My computer ran for about a minute then the javascript virtual machine crashed: Uncaught InternalError: too much recursion This is about the quality of quicksort implementation I'd expect to see in a CS class, or in a random package in npm. If someone on my team committed this, I'd tell them to go rewrite it properly. (Or just use a standard library function - which wouldn't have these flaws.)
usefulcat 11mo ago
The best argument I've yet heard against the effectiveness of AI tools for SW dev is the absence of an explosion of shovelware over the past 1-2 years. https://mikelovesrobots.substack.com/p/wheres-the-shovelware-why-ai-coding https://mikelovesrobots.substack.com/p/wheres-the-shovelware... Basically, if the tools are even half as good as some proponents claim, wouldn't you expect at least a significant increase in simple games on Steam or apps in app stores over that time frame? But we're not seeing that.
- Miraste 11mo ago
  Interesting approach. I can think of one more explanation the author didn't consider: what if software development time wasn't the bottleneck to what he analyzed? The chart for Google Play app submissions, for example, goes down because Google made it much more difficult to publish apps on their store in ways unrelated to software quality. In that case, it wouldn't matter whether AI tools could write a billion production-ready apps, because the limiting factor is Google's submission requirements.
  - parineum 11mo ago
    There are other charts besides Google play. Particularly insightful id the steam chart as steam is already full of shovelware and, in my experience, many developers wish they were making games but the pay is bad. GitHub repos is pretty interesting too but it could be that people just aren't committing this stuff. Showing zero increase is unexpected though.
- zeckalpha 11mo ago
  Interesting, I would make the exact opposite conclusion from the same data: if AI coding was that bad, we'd see more crapware.
- aydyn 11mo ago
  Are you sure we aren't seeing an increase in steam games? Charts I'm looking at show a mild exponential around 2024 https://www.statista.com/statistics/552623/number-games-released-steam/ https://www.statista.com/statistics/552623/number-games-rele... Also theres probably a bottleneck in manual review time.
bpicolo 11mo ago
Algorithmic coding contests are not an equivalent skillset to professional software development
voidhorse 11mo ago
Coding Contest != Software Engineering Or even solving problems that business need to solve, generally speaking. This complete misunderstand of what software engineering even is is the major reason so many engineers are fed up with the clueless leaders foisting AI tools upon their orgs because they apparently lack the critical reasoning skills to be able to distinguish marketing speak from reality.
josephg 11mo ago
I don't think this disproves my claim, for several reasons. First, I don't know where those human participants came from, but if you pick people off the street or from a college campus, they aren't going to be the world's best programmers. On the other hand, github users are on average more skilled than the average CS student. Even students and beginners who use github usually don't have much code there. If the LLMs are weighted to treat every line of code about same, they'd pick up more lines of code from prolific developers (who are often more experienced) than they would from beginners. Also in a coding contest, you're under time pressure. Even when your code works, its often ugly and thrown together. On github, the only code I check in is code that solves whatever problem I set out to solve. I suspect everyone writes better code on github than we do in programming competitions. I suspect if you gave the competitors functionally unlimited time to do the programming competition, many more would outperform GPT-4. Programming contests also usually require that you write a fully self contained program which has been very well specified. The program usually doesn't need any error handling, or need to be maintained. (And if it does need error handling, the cases are all fully specified in the problem description). Relatively speaking, LLMs are pretty good at these kind of problems - where I want some throwaway code that'll work today and get deleted tomorrow. But most software I write isn't like that. And LLMs struggle to write maintainable software in large projects. Most problems aren't so well specified. And for most code, you end up spending more effort maintaining the code over its lifetime than it takes to write in the first place. Chatgpt usually writes code that is a headache to maintain. It doesn't write or use local utility functions. It doesn't factor its code well. The code is often overly verbose. It often writes code that's very poorly optimized. Or the code contains quite obvious bugs for unexpected input - like overflow errors or boundary conditions. And the code it produces very rarely handles errors correctly. None of these problems really matter in programming competitions. But it does matter a lot more when writing real software. These problems make LLMs much less useful at work.
astrange 11mo ago
Chess AI trained at specific human levels performs better than any humans at those levels, because the random mistakes get averaged out. https://www.maiachess.com https://www.maiachess.com

torginus 11mo ago

In my experience (with ChatGPT 5.1 as of late) is that the AI follows a problem->solution internal logic and doesn't think and try to structure its code. If you ask for an endpoint to a CRUD API, it'll make one. If you ask for 5, it'll repeat the same code 5 times and modify it for the use case. A dev wouldn't do this, they would try to figure out the common parts of code, pull them out into helpers, and try to make as little duplicated code as possible. I feel like the AI has a strong bias towards adding things, and not removing them. The most obviously wrong thing is with CSS - when I try to do some styling, it gets 90% of the way there, but there's almost always something that's not quite right. Then I tell the AI to fix a style, since that div is getting clipped or not correctly centered etc. It almost always keeps adding properties, and after 2-3 tries and an incredibly bloated style, I delete the thing and take a step back and think logically about how to properly lay this out with flexbox.

BurningFrog 11mo ago
Bad code is only really bad if it needs to be maintained. If your AI reliably generates working code from a detailed prompt, the prompt is now the source that needs to be maintained. There is no important reason to even look at the generated code
- AlexandrB 11mo ago
  I think this might be plausible in the future, but it needs a lot more tooling. For starters you need to be able to run the prompt through the exact same model so you can reproduce a "build".
  - josephg 11mo ago
    Even the exact same model isn't enough. There are several sources of nondeterminism in LLMs. These would all need to be squashed or seeded - which as far as I know isn't a feature that openai / anthropic / etc provide.
  - BurningFrog 11mo ago
    OK, then the current models aren't as good as I thought/hoped. I guess one thing it means is that we still need extensive test suites. I suppose an LLM can write those too.

malfist 11mo ago

By volume the vast majority of code on github is students. Think about that when you average github for ai

pseudalopex 11mo ago
> By volume the vast majority of code on github is students. Who determined this? How?