3 ms·

By what criteria? In several professional capacities, including coding, LLM's are far superior to an average human already.

by epups 3y ago

slowmovintarget 3y ago
No they're not. They're superior at specific tests when prompted. But asking for a complete application nearly always fails, when an average human that knows programming should be able to make one that works.
- CamperBob2 3y ago
  But asking for a complete application nearly always fails, when an average human that knows programming should be able to make one that works. "This talking dog is a dumbass. The risotto recipe he gave me sucked, and the C++ code he wrote is full of security holes. I don't see what all the hype is about." Hint: ML will get better. Humans will not. ("Slow moving target," indeed.) What we're learning is just how many aspects of our vaunted 'intelligence' are really just features of our languages and the graphs they encode.
  - riffraff 3y ago
    This might be true but it's irrelevant to the claim "LLMs are already better than humans".
    - CamperBob2 3y ago
      You are fixated on the current state of the art, when the first couple of time derivatives are what actually matter. The assertion in question is already true in a limited sense, and it's only going to go in one direction from here.
    - ben_w 3y ago
      Google Translate, which IIRC is also a Transformer model like GPT these days, knows more natural languages than I can remember the names of, to a higher standard than I know my best non-native language, and I moved to Germany 5 years ago. GPT-3 knows most programming languages better than I do, even though I literally learned to read with the Commodore 64 user manual back in the 80s and haven't stopped being a nerd since; and while GPT code isn't always correct or even compilable, it's not like I don't still make mistakes that cause compilation to fail a few times a day, and there's a reason we all insist on testing code rather than just assuming it will work when a dev stops typing. Again, I acknowledge GPT-3.5 (and I assume 4 but have not used it) is not perfect: like others I'd call 3.5 a "junior developer" (and it isn't even that good in every subject!); yet, despite that, the only domains where I can regularly beat it at those where my perception of reality is fundamentally different from its perception (how words sound and how numbers are composed, so it's relatively bad at arithmetic and rhyme), or where it has been forced to be bad (ask it to write about conflict, my experience is everyone reconciles and lives happily ever after). But in most cases it would take me years to get as good as it already is. And there are more of those subjects than I can remember the names of, too.
- p-e-w 3y ago
  > when an average human that knows programming should be able to make one that works. Very few people who "know programming" are actually capable of creating a complete application that serves a specific purpose. Many professional programmers struggle to implement basic algorithms like prime number testing. Coding AIs absolutely outperform the average programming professional, because the average programming professional can barely program. The software industry's demand for programmers is far too great for every coding position to be filled with top-class full stack engineers.
  - shreyshnaccount 3y ago
    Who is this average programmer you're talking about? We did primality testing in literally the first sem at uni
    - pedrosorio 3y ago
      Take a look at this old, commonly referenced article, might be before your time: https://blog.codinghorror.com/why-cant-programmers-program/ https://blog.codinghorror.com/why-cant-programmers-program/
    - TeMPOraL 3y ago
      > We did primality testing in literally the first sem at uni There's a lot of things you did during university classes. I bet you don't even remember half of them, and of the half you do, you couldn't actually do most of them from memory right now. The advantage LLMs have over programmers is that they've seen much more code than any human ever would, remember pretty much all of it - not necessarily verbatim, but also as learned concepts, and that this knowledge does not decay. It's all equally accessible. Whether you ask LLM for basic or optimized primality testing algorithms, or how to use some obscure part of Microsoft's DCOM that approximately nobody on the planet wrote new code for in the last decade - they'll perform equally well. Not perfect, but not worse than an average software dev that does remember the things they're asked to work with.
      - shreyshnaccount 3y ago
- qup 3y ago
  The LLM knows it doesn't work, too, we just don't run it long enough to let it try it out. It would try, get an error, and know how to handle the error. https://github.com/drifting-in-space/botsh https://github.com/drifting-in-space/botsh Check out some agents that probably can already do more than you knew about.

By what criteria? In several professional capacities, including coding, LLM's are far superior to an average human already.

by epups 3y ago

By what criteria? In several professional capacities, including coding, LLM's are far superior to an average human already.

slowmovintarget 3y ago

No they're not. They're superior at specific tests when prompted. But asking for a complete application nearly always fails, when an average human that knows programming should be able to make one that works.

CamperBob2 3y ago
But asking for a complete application nearly always fails, when an average human that knows programming should be able to make one that works. "This talking dog is a dumbass. The risotto recipe he gave me sucked, and the C++ code he wrote is full of security holes. I don't see what all the hype is about." Hint: ML will get better. Humans will not. ("Slow moving target," indeed.) What we're learning is just how many aspects of our vaunted 'intelligence' are really just features of our languages and the graphs they encode.
- riffraff 3y ago
  This might be true but it's irrelevant to the claim "LLMs are already better than humans".
  - CamperBob2 3y ago
    You are fixated on the current state of the art, when the first couple of time derivatives are what actually matter. The assertion in question is already true in a limited sense, and it's only going to go in one direction from here.
  - ben_w 3y ago
    Google Translate, which IIRC is also a Transformer model like GPT these days, knows more natural languages than I can remember the names of, to a higher standard than I know my best non-native language, and I moved to Germany 5 years ago. GPT-3 knows most programming languages better than I do, even though I literally learned to read with the Commodore 64 user manual back in the 80s and haven't stopped being a nerd since; and while GPT code isn't always correct or even compilable, it's not like I don't still make mistakes that cause compilation to fail a few times a day, and there's a reason we all insist on testing code rather than just assuming it will work when a dev stops typing. Again, I acknowledge GPT-3.5 (and I assume 4 but have not used it) is not perfect: like others I'd call 3.5 a "junior developer" (and it isn't even that good in every subject!); yet, despite that, the only domains where I can regularly beat it at those where my perception of reality is fundamentally different from its perception (how words sound and how numbers are composed, so it's relatively bad at arithmetic and rhyme), or where it has been forced to be bad (ask it to write about conflict, my experience is everyone reconciles and lives happily ever after). But in most cases it would take me years to get as good as it already is. And there are more of those subjects than I can remember the names of, too.
p-e-w 3y ago
> when an average human that knows programming should be able to make one that works. Very few people who "know programming" are actually capable of creating a complete application that serves a specific purpose. Many professional programmers struggle to implement basic algorithms like prime number testing. Coding AIs absolutely outperform the average programming professional, because the average programming professional can barely program. The software industry's demand for programmers is far too great for every coding position to be filled with top-class full stack engineers.
- shreyshnaccount 3y ago
  Who is this average programmer you're talking about? We did primality testing in literally the first sem at uni
  - pedrosorio 3y ago
    Take a look at this old, commonly referenced article, might be before your time: https://blog.codinghorror.com/why-cant-programmers-program/ https://blog.codinghorror.com/why-cant-programmers-program/
  - TeMPOraL 3y ago
    > We did primality testing in literally the first sem at uni There's a lot of things you did during university classes. I bet you don't even remember half of them, and of the half you do, you couldn't actually do most of them from memory right now. The advantage LLMs have over programmers is that they've seen much more code than any human ever would, remember pretty much all of it - not necessarily verbatim, but also as learned concepts, and that this knowledge does not decay. It's all equally accessible. Whether you ask LLM for basic or optimized primality testing algorithms, or how to use some obscure part of Microsoft's DCOM that approximately nobody on the planet wrote new code for in the last decade - they'll perform equally well. Not perfect, but not worse than an average software dev that does remember the things they're asked to work with.
    - shreyshnaccount 3y ago
qup 3y ago
The LLM knows it doesn't work, too, we just don't run it long enough to let it try it out. It would try, get an error, and know how to handle the error. https://github.com/drifting-in-space/botsh https://github.com/drifting-in-space/botsh Check out some agents that probably can already do more than you knew about.