4 ms·

I'm a long time StackOverflow user. A few days ago I had some confusion about how to use a React Native library and docs were also not super-clear for me. Ask

I'm a long time StackOverflow user. A few days ago I had some confusion about how to use a React Native library and docs were also not super-clear for me. Asked StackOverflow. Then I said "let's see what happens if I copy paste the exact same question into GPT-4", and did it. Had my answer in 30 seconds, which explained the issue very well and worked. A few hours later I got an answer on SO too telling almost exactly the same, which was right. Anectodal for sure, but ChatGPT and similar tools will keep evolving faster and will become only better. Not surprising to see decline in SO. But I believe there will be an equilibrium as LLMs will need that training data too.

new2this 3y ago
I've been learning to code for the past year. StackOverflow was hugely important in the beginning. I haven't used it in probably 2 months because ChatGPT does an excellent job of not only answering my questions, but also allowing me to ask follow up questions like "what does that specific line of code do?"
david422 3y ago
Ok, but where is ChatGPT getting that answer?
- CameronNemo 3y ago
  I think this is a major problem with LLMs. You often want to know where the answer came from not just so that you can verify the correctness, but also so you can do your own research, learn adjacent topics, and solidify your knowledge.
ChatGTP 3y ago
Had my answer in 30 seconds, which explained the issue very well and worked. A few hours later I got an answer on SO too telling almost exactly the same, which was right. Wow, how strange...
- can16358p 3y ago
  If you imply the person copy pasted it from ChatGPT, it's probably not as the answer was from a very reputable user especially on that framework, and even if the essence was the same as the GPT answer, it wasn't in the same style.
myth_drannon 3y ago
Without the answers from SO, ChatGPT quality will decline as well. Wait 2-4 years for new tech come up, ChatGPT won't have the training data and noone is answering on SO anymore. You would just get wrong answers.
- kolinko 3y ago
  LLMs can navigate sources and generate both documentation and potential answers directly from sources.
  - codethief 3y ago
    Many questions on SO are exactly about things that are not covered by the docs of the tool/language/library in question.
- deleted 3y ago
  [deleted]
- __MatrixMan__ 3y ago
  The ChatGPT of 4 years from now will get its answers from the copilot-enabled editors of people who know how to do the thing. If we're lucky it'll also be paying those people royalties, but I'm not holding my breath.
  - mtVessel 3y ago
    How does it distinguish between people who know how to do the thing and people who use copilot, but are terrible at their jobs?
    - inconceivable 3y ago
      vscode is owned by microsoft and could send runtime results back. so is github...
    - __MatrixMan__ 3y ago
      It gives bad advice to users, from which it learns that that advice didn't work, and only the good stuff gets used going forward (may take some time to retrain and incorporate user feedback). I expect a proliferation of advice that works well enough to convince you that your problem is solved, but which might be subtly broken in ways that you have to come back to. Tracing the fix-it sessions backwards in time to the broke-it sessions (such that the AI can learn from its mistakes) might be tricky unless we somehow manage to connect copiloting sessions through git history. Imagine a big database of changes where the AI is waiting to see whether the code persists, or if somebody comes along and fixes it.
nikanj 3y ago
I’m most surprised with you getting an answer from SO, instead of getting your question shot down by overzealous mods
gadders 3y ago
This works for now, but I wonder what will happen for brand new Framework X in 3 years time if people have stopped contributing to sites like SO. Where will ChatGPT get the knowledge to process and summarise and feedback to people? Blog pages? As I understand it, someone needs to be creating the information that goes into the top of the funnel. I don't think ChatGPT can create an answer from scratch.
- can16358p 3y ago
  That's exactly why I wrote the last part.
lizard 3y ago
It seems like there's an opportunity here for SO to use an LLM to provide an immediate response to questions, then allow the community to discuss, clarify, expand, and even correct the response. This could even be done collaboratively with the LLM to improve its training with real expertise instead of feeding back into itself. This would be a bit of a product shift for SO though, and some people would undoubtedly be opposed to it and leave the platform. But if LLMs pose a threat to the platform it may be better than to alternative.
- CameronNemo 3y ago
  Considering SO's corpus is liberally licensed, couldn't anyone take up this project?

new2this 3y ago

I've been learning to code for the past year. StackOverflow was hugely important in the beginning. I haven't used it in probably 2 months because ChatGPT does an excellent job of not only answering my questions, but also allowing me to ask follow up questions like "what does that specific line of code do?"

david422 3y ago

Ok, but where is ChatGPT getting that answer?

CameronNemo 3y ago
I think this is a major problem with LLMs. You often want to know where the answer came from not just so that you can verify the correctness, but also so you can do your own research, learn adjacent topics, and solidify your knowledge.

ChatGTP 3y ago

Had my answer in 30 seconds, which explained the issue very well and worked. A few hours later I got an answer on SO too telling almost exactly the same, which was right. Wow, how strange...

can16358p 3y ago
If you imply the person copy pasted it from ChatGPT, it's probably not as the answer was from a very reputable user especially on that framework, and even if the essence was the same as the GPT answer, it wasn't in the same style.

myth_drannon 3y ago

Without the answers from SO, ChatGPT quality will decline as well. Wait 2-4 years for new tech come up, ChatGPT won't have the training data and noone is answering on SO anymore. You would just get wrong answers.

kolinko 3y ago
LLMs can navigate sources and generate both documentation and potential answers directly from sources.
- codethief 3y ago
  Many questions on SO are exactly about things that are not covered by the docs of the tool/language/library in question.
deleted 3y ago
[deleted]
__MatrixMan__ 3y ago
The ChatGPT of 4 years from now will get its answers from the copilot-enabled editors of people who know how to do the thing. If we're lucky it'll also be paying those people royalties, but I'm not holding my breath.
- mtVessel 3y ago
  How does it distinguish between people who know how to do the thing and people who use copilot, but are terrible at their jobs?
  - inconceivable 3y ago
    vscode is owned by microsoft and could send runtime results back. so is github...
  - __MatrixMan__ 3y ago
    It gives bad advice to users, from which it learns that that advice didn't work, and only the good stuff gets used going forward (may take some time to retrain and incorporate user feedback). I expect a proliferation of advice that works well enough to convince you that your problem is solved, but which might be subtly broken in ways that you have to come back to. Tracing the fix-it sessions backwards in time to the broke-it sessions (such that the AI can learn from its mistakes) might be tricky unless we somehow manage to connect copiloting sessions through git history. Imagine a big database of changes where the AI is waiting to see whether the code persists, or if somebody comes along and fixes it.

nikanj 3y ago

I’m most surprised with you getting an answer from SO, instead of getting your question shot down by overzealous mods

gadders 3y ago

This works for now, but I wonder what will happen for brand new Framework X in 3 years time if people have stopped contributing to sites like SO. Where will ChatGPT get the knowledge to process and summarise and feedback to people? Blog pages? As I understand it, someone needs to be creating the information that goes into the top of the funnel. I don't think ChatGPT can create an answer from scratch.

can16358p 3y ago
That's exactly why I wrote the last part.

lizard 3y ago

It seems like there's an opportunity here for SO to use an LLM to provide an immediate response to questions, then allow the community to discuss, clarify, expand, and even correct the response. This could even be done collaboratively with the LLM to improve its training with real expertise instead of feeding back into itself. This would be a bit of a product shift for SO though, and some people would undoubtedly be opposed to it and leave the platform. But if LLMs pose a threat to the platform it may be better than to alternative.

CameronNemo 3y ago
Considering SO's corpus is liberally licensed, couldn't anyone take up this project?