3 ms·

You have to put it in context: How good is the human alternative? I have interviewed hundreds of engineers across the entire skill spectrum. I think GPT4 right

You have to put it in context: How good is the human alternative? I have interviewed hundreds of engineers across the entire skill spectrum. I think GPT4 right now is about on-par with a mid level developer. It makes a lot of mistakes, especially around counting, and usually can notice and fix them if they're pointed out. Humans do this all the time -- how often do you get a compiler error from something silly? It often misapplies interfaces on the first go-around. Again, this is just like humans. We make mistakes, notice them, fix them. If you simulate this by telling GPT that its code produced an error it will often correct itself. The absolutely killer feature of GPT4 is that it has these skills in every subject. It's fluent in kernel operations. Databases. Networking. Various UX frameworks. Any language. It's definitely not perfect. But, if the alternative is hiring a mid-level human engineer, GPT4 is a really compelling alternative.

xtagon 3y ago
A mid-level human engineer can iteratively fix the mistakes they start making, instead of that just being the final result. Can GPT-4 do that?
- throwaway09223 3y ago
  Yes, like I said if you give it the results, eg "That code resulted in this error. What's wrong?" it will more often than not correct its mistake. Hooking it up to automatically run the code in question and examine the output is a trivial undertaking - many folks have already done this.
  - jhugo 3y ago
    My experience has usually been that it will apologize and then produce another snippet with a different flaw. When that is pointed out, it will usually go back to the original snippet with the original flaw. It sometimes also insists that it has “run the test suite and they all pass”, just like a human programmer that is trying to fake it until they make it I guess?
- tomjakubowski 3y ago
  I wonder if anyone has done a feedback loop with GPT and compiler errors to train this sort of thing
  - aldarisbm 3y ago
    they have, with langchain
- kubrickslair 3y ago
  It does it pretty well for Python with code interpreter, especially for quant/ ML heavy code.
bcrosby95 3y ago
I would peg gpt as between junior and mid. Mids (around 5-10 years of experience) tend to produce overly complex code for the problem at hand.
- saberience 3y ago
  Dude, chatgpt as a junior or mid level programmer? Have you actually met junior or mid level programmers? I’ve got 20 years of experience in the tech industry and chatgpt is far far better than any mid level engineer I’ve ever met and I include FAANG engineers too. Chatgpt can answer any leetcode programming question in essentially every programming language in existence in less than 20 seconds. Chatgpt can analyze classes for errors written in every language in seconds. Chatgpt helped me write animation, ai, sound handling code in c++ and helped me build a full working game, the total code generation time for chatgpt “thinking” was on the order of one hour. I don’t know any senior engineer who could help me (a c++ newbie) write a whole working game engine with all the necessary systems in less than one hour of “thinking time”. I’m starting to think that all the “naysayers” of chatgpt simply used it once or twice, saw one error and dismissed the whole thing. Or they’re simply afraid for their own skills and avoid chatgpt as the alternative scares them too much.
  - YeGoblynQueenne 3y ago
    Can you share a link to the game you made with ChatGPT?
  - bcrosby95 3y ago
    Yes, we employ them regularly. And yes, I use both ChatGPT and copilot regularly. In my experience, if you compare the unedited output of ChatGPT to the unedited output of someone with a couple years of experience in the given domain, ChatGPT will have more subtle bugs than the human provided output. Hence my between junior & mid assessment. It doesn't mean that it's useless though. Having a tool at that level of experience for every domain in the world that can cook up code near instantaneously is damn useful. And I assume it'll only get better.
Yodel0914 3y ago
> I think GPT4 right now is about on-par with a mid level developer. At writing code, that might be a slight exaggeration. At explaining code it's at least at mid level, possibly better.
- saberience 3y ago
  How many mid level developers do you know that understand the syntax and apis of every single programming language in existence? I’ve used chatgpt recently to build an entire c++ game with sound, keyboard and mouse control, ai, etc, having never used c++ before. I’ve also never built a game before. I’ve worked with a lot mid level developers and I’ve never seen any that can write me a a c++ file giving me working code for a collision system in 30 seconds. Let me know if you find one though.

xtagon 3y ago

A mid-level human engineer can iteratively fix the mistakes they start making, instead of that just being the final result. Can GPT-4 do that?

throwaway09223 3y ago
Yes, like I said if you give it the results, eg "That code resulted in this error. What's wrong?" it will more often than not correct its mistake. Hooking it up to automatically run the code in question and examine the output is a trivial undertaking - many folks have already done this.
- jhugo 3y ago
  My experience has usually been that it will apologize and then produce another snippet with a different flaw. When that is pointed out, it will usually go back to the original snippet with the original flaw. It sometimes also insists that it has “run the test suite and they all pass”, just like a human programmer that is trying to fake it until they make it I guess?
tomjakubowski 3y ago
I wonder if anyone has done a feedback loop with GPT and compiler errors to train this sort of thing
- aldarisbm 3y ago
  they have, with langchain
kubrickslair 3y ago
It does it pretty well for Python with code interpreter, especially for quant/ ML heavy code.

bcrosby95 3y ago

I would peg gpt as between junior and mid. Mids (around 5-10 years of experience) tend to produce overly complex code for the problem at hand.

saberience 3y ago
Dude, chatgpt as a junior or mid level programmer? Have you actually met junior or mid level programmers? I’ve got 20 years of experience in the tech industry and chatgpt is far far better than any mid level engineer I’ve ever met and I include FAANG engineers too. Chatgpt can answer any leetcode programming question in essentially every programming language in existence in less than 20 seconds. Chatgpt can analyze classes for errors written in every language in seconds. Chatgpt helped me write animation, ai, sound handling code in c++ and helped me build a full working game, the total code generation time for chatgpt “thinking” was on the order of one hour. I don’t know any senior engineer who could help me (a c++ newbie) write a whole working game engine with all the necessary systems in less than one hour of “thinking time”. I’m starting to think that all the “naysayers” of chatgpt simply used it once or twice, saw one error and dismissed the whole thing. Or they’re simply afraid for their own skills and avoid chatgpt as the alternative scares them too much.
- YeGoblynQueenne 3y ago
  Can you share a link to the game you made with ChatGPT?
- bcrosby95 3y ago
  Yes, we employ them regularly. And yes, I use both ChatGPT and copilot regularly. In my experience, if you compare the unedited output of ChatGPT to the unedited output of someone with a couple years of experience in the given domain, ChatGPT will have more subtle bugs than the human provided output. Hence my between junior & mid assessment. It doesn't mean that it's useless though. Having a tool at that level of experience for every domain in the world that can cook up code near instantaneously is damn useful. And I assume it'll only get better.

Yodel0914 3y ago

> I think GPT4 right now is about on-par with a mid level developer. At writing code, that might be a slight exaggeration. At explaining code it's at least at mid level, possibly better.

saberience 3y ago
How many mid level developers do you know that understand the syntax and apis of every single programming language in existence? I’ve used chatgpt recently to build an entire c++ game with sound, keyboard and mouse control, ai, etc, having never used c++ before. I’ve also never built a game before. I’ve worked with a lot mid level developers and I’ve never seen any that can write me a a c++ file giving me working code for a collision system in 30 seconds. Let me know if you find one though.