2 ms·

> different developers on HN have very different jobs and skill levels. Definitely this. When I use AIs for web development they do an ok job most of the time.

by IshKebab 10mo ago

> different developers on HN have very different jobs and skill levels. Definitely this. When I use AIs for web development they do an ok job most of the time. Definitely on par with a junior dev. For anything outside of that they're still pretty bad. Not useless by any stretch, but it's still a fantasy to think you could replace even a good junior dev with AI in most domains. I am slightly worried for my job... but only because AI will keep improving and there is a chance it will be as good as me one day. Today it's not a threat at all.

ryandrake 10mo ago
Yea, LLMs produce results on par with what I would expect out of a solid junior developer. They take direction, their models act as the “do the research” part, and they output lots of code: code that has to be carefully scrutinized and refined. They are like very ambitious interns who never get tired and want to please, but often just produce crap that has to be totally redone or refactored heavily in order to go into production. If you think LLMs are “better programmers than you,” well, I have some disappointing news for you that might take you a while to accept.
- monsieurbanana 10mo ago
  > LLMs produce results on par with what I would expect out of a solid junior developer This is a common take but it hasn't been my experience. LLMs produce results that vary from expert all the way to slightly better than markov chains. The average result might be equal to a junior developer, and the worst case doesn't happen that often, but the fact that it happens from time to time makes it completely unreliable for a lot of tasks. Junior developers are much more consistent. Sure, you will find the occasional developer that would delete the test file rather than fixing the tests, but either they will learn their lesson after seeing your wth face or you can fire them. Can't do that with llms.
  - jvanderbot 10mo ago
    I think any further discussion about quality just needs to have the following metadata: - Language - Total LOC - Subject matter expertise required - Total dependency chain - Subjective score (audited randomly) And we can start doing some analysis. Otherwise we're pissing into ten kinds of winds. My own subjective experience is earth shattering at webapps in html and css (because I'm terrible and slow at it), and annoyingly good but a bit wrong usually in planning and optimization in rust and horribly lost at systems design or debugging a reasonably large rust system.
    - monsieurbanana 10mo ago
      I agree in that these discussions (this whole hn thread tbh) are seriously lacking in concrete examples to be more than holy wars 3.0. Besides one point: junior developers can learn from their egregious mistakes, llms can't no matter how strongly worded you are in their system prompt. In a functional work environment, you will build trust with your coworkers little by little. The pale equivalent in LLMs is improving system prompts and writing more and more ai directives that might or might not be followed.
      - embedding-shape 10mo ago
        > Besides one point: junior developers can learn from their egregious mistakes, llms can't no matter how strongly worded you are in their system prompt. I think if you set off an LLM to do something, and it does a "egregious mistake" in the implementation, and then you adjust the system prompt to explicitly guard against that or go towards a different implementation and you restart from scratch again yet it does the exact same "egregious mistake", then you need to try a different model/tool than the one you've tried that with. It's common with smaller models, or bigger models that are heavily quanitized that they aren't great at following system/developer prompts, but that really shouldn't happen with the available SOTA models, I haven't had something ignored like that in years by now.
        jvanderbot 10mo ago
        And honestly this is precisely why I don't fear unemployment, but I do fear less employment overall. I can learn and get better and use LLMs as a tool. So there's still a "me" there steering. Eventually this might not be the case. But if automating things has taught me anything, it's that removing the person is usually such a long tail cost that it's cheaper to keep someone in the loop. But is this like steel production or piloting (few highly trained experts are in the loop) or more like warehouse work (lots of automation removed any skills like driving or inventory work etc).
  - AnimalMuppet 10mo ago
    I can in fact fire an LLM. It's even easier than firing a junior developer. Or rather, it's more like a contractor. If I don't like the job they did, I don't give them the next job.
anthonypasq 10mo ago
you say this as if web development isnt 90% of software.

ryandrake 10mo ago

Yea, LLMs produce results on par with what I would expect out of a solid junior developer. They take direction, their models act as the “do the research” part, and they output lots of code: code that has to be carefully scrutinized and refined. They are like very ambitious interns who never get tired and want to please, but often just produce crap that has to be totally redone or refactored heavily in order to go into production. If you think LLMs are “better programmers than you,” well, I have some disappointing news for you that might take you a while to accept.

monsieurbanana 10mo ago
> LLMs produce results on par with what I would expect out of a solid junior developer This is a common take but it hasn't been my experience. LLMs produce results that vary from expert all the way to slightly better than markov chains. The average result might be equal to a junior developer, and the worst case doesn't happen that often, but the fact that it happens from time to time makes it completely unreliable for a lot of tasks. Junior developers are much more consistent. Sure, you will find the occasional developer that would delete the test file rather than fixing the tests, but either they will learn their lesson after seeing your wth face or you can fire them. Can't do that with llms.
- jvanderbot 10mo ago
  I think any further discussion about quality just needs to have the following metadata: - Language - Total LOC - Subject matter expertise required - Total dependency chain - Subjective score (audited randomly) And we can start doing some analysis. Otherwise we're pissing into ten kinds of winds. My own subjective experience is earth shattering at webapps in html and css (because I'm terrible and slow at it), and annoyingly good but a bit wrong usually in planning and optimization in rust and horribly lost at systems design or debugging a reasonably large rust system.
  - monsieurbanana 10mo ago
    I agree in that these discussions (this whole hn thread tbh) are seriously lacking in concrete examples to be more than holy wars 3.0. Besides one point: junior developers can learn from their egregious mistakes, llms can't no matter how strongly worded you are in their system prompt. In a functional work environment, you will build trust with your coworkers little by little. The pale equivalent in LLMs is improving system prompts and writing more and more ai directives that might or might not be followed.
    - embedding-shape 10mo ago
      > Besides one point: junior developers can learn from their egregious mistakes, llms can't no matter how strongly worded you are in their system prompt. I think if you set off an LLM to do something, and it does a "egregious mistake" in the implementation, and then you adjust the system prompt to explicitly guard against that or go towards a different implementation and you restart from scratch again yet it does the exact same "egregious mistake", then you need to try a different model/tool than the one you've tried that with. It's common with smaller models, or bigger models that are heavily quanitized that they aren't great at following system/developer prompts, but that really shouldn't happen with the available SOTA models, I haven't had something ignored like that in years by now.
      - jvanderbot 10mo ago
        And honestly this is precisely why I don't fear unemployment, but I do fear less employment overall. I can learn and get better and use LLMs as a tool. So there's still a "me" there steering. Eventually this might not be the case. But if automating things has taught me anything, it's that removing the person is usually such a long tail cost that it's cheaper to keep someone in the loop. But is this like steel production or piloting (few highly trained experts are in the loop) or more like warehouse work (lots of automation removed any skills like driving or inventory work etc).
- AnimalMuppet 10mo ago
  I can in fact fire an LLM. It's even easier than firing a junior developer. Or rather, it's more like a contractor. If I don't like the job they did, I don't give them the next job.

anthonypasq 10mo ago

you say this as if web development isnt 90% of software.