36 ms·

It's quickly apparent that the people impressed with ChatGPT's code are generally solving pretty toy problems and want an alternative to SO for "how to do X".

tombert 4y ago
And that’s fine for the average HN audience; what disturbs me is that hiring managers see articles like “I had ChatGPT rewrite my code and it went 100000% faster!!!”, and then decide that they don’t need more than one engineer. If we could automate away engineers that’s fine with me, but i just don’t think that ChatGPT is there yet. I actually think Wolfram is kind of onto something with having ChatGPT work with Alpha to be used for stuff that has “objective” answers.
- ryokeken 4y ago
  that's what unions are for if only an engineer could engineer a solution to what afflicts unions, dat be great
- alfalfasprout 4y ago
  The comment above captures it well though-- many of us aren't just writing stuff you can copy/paste from stackoverflow. When you're working on things where you're operating with internal business concepts and data with set meanings the difficulty is rarely writing the code but rather reasoning about how to cleanly approach the problem in the first place. The thing is, the hiring managers reacting like that to ChatGPT are also the same folks who would've considered outsourcing engineering anyways to one of those offshore "consultancies". Tools like ChatGPT will ultimately be a replacement for use cases where you've already fully scoped out what to build and how to build it and the implementation is straightforward.
jerf 4y ago
One of the problems with ChatGPT is that for the most part, the things that people are most likely to feed it are also the things it is most likely to have training data for. So, you say, "Write me a heap for Java", and it spits out amazing, wonderful code that is almost right. Well, yeah, that's cool and all, though I would point out that its training data probably included a heap that was entirely correct, but, still, it's a common question. But I'm not writing a heap for Java. I'm taking business object X and stuffing through system Y and feeding the result to system Z, all internal to the company I work for. Good luck with getting ChatGPT to do that. But it deceives people, because the very questions they are most likely to ask as a test are the exact questions it knows the answer to, for the very reason that they are also the most likely questions to be in the training data. (This article kind of doubles as great set of examples of just how quickly you get into confident nonsense if you stick a toe slightly out of line in your query. For instance, even if it knows "What is Y" and "What is Z", it can't necessarily combine them in any correct way. It'll spew out a lot of words that sound very confident, but it'll make mistakes without any indication they were mistakes.)
- alfalfasprout 4y ago
  > But I'm not writing a heap for Java. I'm taking business object X and stuffing through system Y and feeding the result to system Z, all internal to the company I work for. Good luck with getting ChatGPT to do that. Exactly. Nearly every example I've seen is something that I could easily find an answer for on SO for anyways. ChatGPT customizes it as needed which is nice (though it often makes glaring and not so glaring mistakes).
  - phatfish 4y ago
    It's interesting, StackOverflow will give you (after sufficient searching) an existing solution close to what you want. It's pretty certain to be accurate/working in the context of the question asked on SO. You as the "problem solver" just have to adapt the similar solution to your own problem. ChatGPT will try to respond with exactly what you want, but with (multiple) errors that must be fixed. I'll take the working example and adapt it myself. There was a blog post on SO about this recently, they are banning accounts posting ChatGPT answers because they appear reasonable at first glance, but try and execute the code and it has syntax errors or is just plain wrong.
- adamsmith143 4y ago
  "Guys the first version of a machine to generate code hasn't put me out of a job, it's obviously bollocks." !remindme 10 years
  - tombert 4y ago
    I cannot speak for anyone else, but the disappointment for me came from abundance of clickbaitey articles/videos saying stuff like “ChatGPT rewrote my website 10x better than me!!!” I don’t dispute that it’s super cool technology made by some equally cool people, and honestly I would be perfectly fine with it putting me out of the job if it gets to that point.
    - adamsmith143 4y ago
      Test it for yourself. The beta is free. Whats the point in reading about it when you can easily try it out
IshKebab 4y ago
Well yes because that's about the limit of what it is capable of at the moment, but that limit is very impressive! I don't know why so many people are unimpressed with it because it isn't some kind of all knowing hyperintelligent god. This guy is literally unimpressed with it because it can't solve Fermat's last theorem! Wtf?
- tombert 4y ago
  I said it was “neat”! It’s impressive tech, but giving a false-but-believable proof is substantially worse than saying “I don’t know how to do that”. If it had come back with “I don’t know how to give a proof of Fermat’s Last Theorem in Isabelle”, I wouldn’t have been unimpressed; there are plenty of very cool things it can do. I just think that it being confidently incorrect is a bad thing. EDIT: I should point out, it also was wrong with substantially simpler proofs, like the proof of infinite primes.

tombert 4y ago

And that’s fine for the average HN audience; what disturbs me is that hiring managers see articles like “I had ChatGPT rewrite my code and it went 100000% faster!!!”, and then decide that they don’t need more than one engineer. If we could automate away engineers that’s fine with me, but i just don’t think that ChatGPT is there yet. I actually think Wolfram is kind of onto something with having ChatGPT work with Alpha to be used for stuff that has “objective” answers.

ryokeken 4y ago
that's what unions are for if only an engineer could engineer a solution to what afflicts unions, dat be great
alfalfasprout 4y ago
The comment above captures it well though-- many of us aren't just writing stuff you can copy/paste from stackoverflow. When you're working on things where you're operating with internal business concepts and data with set meanings the difficulty is rarely writing the code but rather reasoning about how to cleanly approach the problem in the first place. The thing is, the hiring managers reacting like that to ChatGPT are also the same folks who would've considered outsourcing engineering anyways to one of those offshore "consultancies". Tools like ChatGPT will ultimately be a replacement for use cases where you've already fully scoped out what to build and how to build it and the implementation is straightforward.

jerf 4y ago

One of the problems with ChatGPT is that for the most part, the things that people are most likely to feed it are also the things it is most likely to have training data for. So, you say, "Write me a heap for Java", and it spits out amazing, wonderful code that is almost right. Well, yeah, that's cool and all, though I would point out that its training data probably included a heap that was entirely correct, but, still, it's a common question. But I'm not writing a heap for Java. I'm taking business object X and stuffing through system Y and feeding the result to system Z, all internal to the company I work for. Good luck with getting ChatGPT to do that. But it deceives people, because the very questions they are most likely to ask as a test are the exact questions it knows the answer to, for the very reason that they are also the most likely questions to be in the training data. (This article kind of doubles as great set of examples of just how quickly you get into confident nonsense if you stick a toe slightly out of line in your query. For instance, even if it knows "What is Y" and "What is Z", it can't necessarily combine them in any correct way. It'll spew out a lot of words that sound very confident, but it'll make mistakes without any indication they were mistakes.)

alfalfasprout 4y ago
> But I'm not writing a heap for Java. I'm taking business object X and stuffing through system Y and feeding the result to system Z, all internal to the company I work for. Good luck with getting ChatGPT to do that. Exactly. Nearly every example I've seen is something that I could easily find an answer for on SO for anyways. ChatGPT customizes it as needed which is nice (though it often makes glaring and not so glaring mistakes).
- phatfish 4y ago
  It's interesting, StackOverflow will give you (after sufficient searching) an existing solution close to what you want. It's pretty certain to be accurate/working in the context of the question asked on SO. You as the "problem solver" just have to adapt the similar solution to your own problem. ChatGPT will try to respond with exactly what you want, but with (multiple) errors that must be fixed. I'll take the working example and adapt it myself. There was a blog post on SO about this recently, they are banning accounts posting ChatGPT answers because they appear reasonable at first glance, but try and execute the code and it has syntax errors or is just plain wrong.
adamsmith143 4y ago
"Guys the first version of a machine to generate code hasn't put me out of a job, it's obviously bollocks." !remindme 10 years
- tombert 4y ago
  I cannot speak for anyone else, but the disappointment for me came from abundance of clickbaitey articles/videos saying stuff like “ChatGPT rewrote my website 10x better than me!!!” I don’t dispute that it’s super cool technology made by some equally cool people, and honestly I would be perfectly fine with it putting me out of the job if it gets to that point.
  - adamsmith143 4y ago
    Test it for yourself. The beta is free. Whats the point in reading about it when you can easily try it out

IshKebab 4y ago

Well yes because that's about the limit of what it is capable of at the moment, but that limit is very impressive! I don't know why so many people are unimpressed with it because it isn't some kind of all knowing hyperintelligent god. This guy is literally unimpressed with it because it can't solve Fermat's last theorem! Wtf?

tombert 4y ago
I said it was “neat”! It’s impressive tech, but giving a false-but-believable proof is substantially worse than saying “I don’t know how to do that”. If it had come back with “I don’t know how to give a proof of Fermat’s Last Theorem in Isabelle”, I wouldn’t have been unimpressed; there are plenty of very cool things it can do. I just think that it being confidently incorrect is a bad thing. EDIT: I should point out, it also was wrong with substantially simpler proofs, like the proof of infinite primes.