4 ms·

In my experience chatGPT is more useful than StackOverflow in 9 out of 10 cases because it can generate custom tailor-made code for your exact use case. Sure, i

by umvi 4y ago

In my experience chatGPT is more useful than StackOverflow in 9 out of 10 cases because it can generate custom tailor-made code for your exact use case. Sure, it might not be "correct" in the ultra-pedantic StackOverflow definition of "100% generalizably correct", but usually it's 90+% correct for your use-case and only needs slight alteration to be 100% correct for your use-case. Such generated code is not a "threat" or "misinformation" or whatever author's point is. This is going to be a productivity multiplier for programmers so that programmers can do more faster than ever before!

Retric 4y ago
It works really well if slapping together something vaguely close to correct is good enough. This covers a significant fraction of all programming work.
- simonw 4y ago
  It works great for slapping together something vaguely close to correct... and then iterating on it until it's exactly right. This makes is a fantastic productivity boost if you have the skill to identity and fix any problems with the code it has generated. If you've ever participated in code review as a reviewer you probably have those skills already.
  - Retric 4y ago
    Starting from subtly incorrect code makes it much harder to come to the correct solution. To use a classic example, the A = (B + C) /2 line that fails due to bounds checking issues is quite literally worse than useless because you basically need to understand why it’s wrong before you notice it’s wrong. The eyes just slide over the common idiom thinking in terms of average not what the code actually does. Code reviews most easily reveal a very different kind of error. ‘Why does this code look more complicated than it should be?’ is a great hint that something is fishy. However, chatGPT is basically engineered to pass the smell test not work.
    - rakejake 4y ago
      Yeah I am surprised how many people are of the opinion that code reviews are easy. Errors like the one you mention are especially hard to catch unless you know what you are looking for.
josefrichter 4y ago
The interesting thing is that on SO, the good stuff gets upvoted, while the bad/incorrect stuff is being pushed down or out completely. And it’s not rare that you see poor/wrong code snippets. Thanks to this mechanism, it actually shouldn’t be harmful to have auto-generated responses there, because they get reviewed and curated by humans. Isn’t the combination of AI legwork with Human review the best thing we have so far?
- area51org 4y ago
  Adding a bunch of noise to that situation is less than helpful.
- awkward 4y ago
  StackOverflow and Wikipedia both depend on a particular dynamic. It's costly to come up with nonsense that looks right enough to pass, and it's easy to hit revert or downvote that content if it's fishy. AI generation of nonsense flips that - it's easy and limitlessly scalable to come up with "rightish" stuff, and cleanup still takes human effort that scales at the number of engaged users.
  - Eleison23 4y ago
  - josefrichter 4y ago
    That’s actually a remarkably insightful comment. Thank you for that.
aftbit 4y ago
It's also more willing to confidently invent wrong answers than StackOverflow. Try asking it to generate a Rust function that uses the `ripgrep` crate to search text with regular expressions. The ripgrep create doesn't expose an interface to do that (as far as I can tell) but ChatGPT is happy to generate some plausible-sounding but totally incorrect code.
- antonvs 4y ago
  Sure, but for an expert user, it doesn’t take long to figure out when you have a wrong answer. And in a case like this, the wrong answer is because it was set up with a question that doesn’t make sense - ripgrep is explicitly described as not being a library. This kind of objection seems to me to be in the same general category as saying that you can use a programming language to write incorrect code, which is not a meaningful objection to programming languages.
  - deleted 4y ago
    [deleted]
  - versteegen 4y ago
    Right. People need to realise that ChatGPT shouldn't be used in the same way as Google or asking a question on SO. It doesn't have the same capabilities and drawbacks. In particular because of it not understanding the big picture it won't point out that you're asking the wrong question like an expert human would. (Does it ever?)
    - antonvs 4y ago
      > it won't point out that you're asking the wrong question like an expert human would. (Does it ever?) I don't think it will - its design doesn't really allow for that. It's generating responses to the prompts it's given based on its trained knowledge, but it doesn't have (enough of?) the kind of meta-reasoning required to step back, gain an understanding of the context of the question, and propose a different solution.
  - chrisco255 4y ago

by umvi 4y ago

Retric 4y ago

It works really well if slapping together something vaguely close to correct is good enough. This covers a significant fraction of all programming work.

simonw 4y ago
It works great for slapping together something vaguely close to correct... and then iterating on it until it's exactly right. This makes is a fantastic productivity boost if you have the skill to identity and fix any problems with the code it has generated. If you've ever participated in code review as a reviewer you probably have those skills already.
- Retric 4y ago
  Starting from subtly incorrect code makes it much harder to come to the correct solution. To use a classic example, the A = (B + C) /2 line that fails due to bounds checking issues is quite literally worse than useless because you basically need to understand why it’s wrong before you notice it’s wrong. The eyes just slide over the common idiom thinking in terms of average not what the code actually does. Code reviews most easily reveal a very different kind of error. ‘Why does this code look more complicated than it should be?’ is a great hint that something is fishy. However, chatGPT is basically engineered to pass the smell test not work.
  - rakejake 4y ago
    Yeah I am surprised how many people are of the opinion that code reviews are easy. Errors like the one you mention are especially hard to catch unless you know what you are looking for.

josefrichter 4y ago

The interesting thing is that on SO, the good stuff gets upvoted, while the bad/incorrect stuff is being pushed down or out completely. And it’s not rare that you see poor/wrong code snippets. Thanks to this mechanism, it actually shouldn’t be harmful to have auto-generated responses there, because they get reviewed and curated by humans. Isn’t the combination of AI legwork with Human review the best thing we have so far?

area51org 4y ago
Adding a bunch of noise to that situation is less than helpful.
awkward 4y ago
StackOverflow and Wikipedia both depend on a particular dynamic. It's costly to come up with nonsense that looks right enough to pass, and it's easy to hit revert or downvote that content if it's fishy. AI generation of nonsense flips that - it's easy and limitlessly scalable to come up with "rightish" stuff, and cleanup still takes human effort that scales at the number of engaged users.
- Eleison23 4y ago
- josefrichter 4y ago
  That’s actually a remarkably insightful comment. Thank you for that.

aftbit 4y ago

It's also more willing to confidently invent wrong answers than StackOverflow. Try asking it to generate a Rust function that uses the `ripgrep` crate to search text with regular expressions. The ripgrep create doesn't expose an interface to do that (as far as I can tell) but ChatGPT is happy to generate some plausible-sounding but totally incorrect code.

antonvs 4y ago
Sure, but for an expert user, it doesn’t take long to figure out when you have a wrong answer. And in a case like this, the wrong answer is because it was set up with a question that doesn’t make sense - ripgrep is explicitly described as not being a library. This kind of objection seems to me to be in the same general category as saying that you can use a programming language to write incorrect code, which is not a meaningful objection to programming languages.
- deleted 4y ago
  [deleted]
- versteegen 4y ago
  Right. People need to realise that ChatGPT shouldn't be used in the same way as Google or asking a question on SO. It doesn't have the same capabilities and drawbacks. In particular because of it not understanding the big picture it won't point out that you're asking the wrong question like an expert human would. (Does it ever?)
  - antonvs 4y ago
    > it won't point out that you're asking the wrong question like an expert human would. (Does it ever?) I don't think it will - its design doesn't really allow for that. It's generating responses to the prompts it's given based on its trained knowledge, but it doesn't have (enough of?) the kind of meta-reasoning required to step back, gain an understanding of the context of the question, and propose a different solution.
- chrisco255 4y ago