4 ms·

What are the circumstances where StackOverflow is still competitive against ChatGPT? As far as I can tell SO.com and google are both obsolete for getting answer

by roenxi 3y ago

What are the circumstances where StackOverflow is still competitive against ChatGPT? As far as I can tell SO.com and google are both obsolete for getting answers to programming questions. I haven't used either in a while and as locally hosted LLMs get more pervasive I don't see the situation tilting back towards those services.

dwheeler 3y ago
I think it's very likely that StackExchange is the key data source for ChatGPT answers of this kind. If you want LLMs to give good answers, it's important to have their sources correct.
viraptor 3y ago
Time passing is the circumstance. ChatGPT knows the answers from SO, but for new languages the quality will degrade over time. ChatGPT already isn't great at answering questions about new version of Swift for example. It won't be able to use new languages / APIs either. Given how rarely the model is updated (officially anyway), it will always lag behind the new tech. It may be actually in the interest of OpenAI to start paying SO for the data and make sure it flows to the community in some way. Otherwise they will be starved.
layer8 3y ago
This is just one aspect, but some people strongly prefer typing only a few search terms and then exploring links over holding a linear synchronous conversation with a chatbot. It’s a significantly different usage mode. It’s a bit like browsing through a book vs. interviewing a random person who read the book.
treis 3y ago
A SO post usually has an answer that works and is a reasonably good idea. ChatGPT lacks the ability to see if its responses pass even a basic sanity test. Sometimes it's answers are good and sometimes they are totally wrong. Hard for a person asking the question to determine which answer they got.
- rolisz 3y ago
  With code interpreter, ChatGPT can check that it's code at least runs without errors
  - dharmab 3y ago
    rm -rf $INSTALL_DIR/* this line of code has at least two gotchas. if you run it to test it out, there is a good chance you will destroy your data. It is otherwise a valid line of code. This is not a theoretical problem: https://github.com/valvesoftware/steam-for-linux/issues/3671 https://github.com/valvesoftware/steam-for-linux/issues/3671
    - rolisz 3y ago
      ChatGPT warns about gotchas with that line.
      - danShumway 3y ago
        The point being that: > With code interpreter, ChatGPT can check that it's code at least runs without errors is not a solution to gotchas/caveats that build up over time that ChatGPT won't know about. ChatGPT can warn about caveats from that line of code because it's old code. It does not have a magical defense that will allow it to warn about caveats, errors, security risks, or deprecations that are introduced after its knowledge cutoff. The best solution that we've been able to find so far is for ChatGPT to do a search and feed the results into its context. But that has its own problems, and importantly it only works if the underlying search engines are still returning up-to-date relevant results and documentation from sites like StackOverflow.
  - treis 3y ago
    Yes, but ultimately it's still a "statistically likely" answer and not the "right" one. Often the two overlap and maybe one day they'll make it an overlapping venn diagram. But that day is not today.
wodenokoto 3y ago
If SO can’t compete on hard problems then we should all change job titles to prompt engineers.
- twelve40 3y ago
  That's a strange claim. SO's short q-and-a format doesn't even begin to tackle hard problems.
remram 3y ago
We can't keep training ChatGPT on SO and then pretend that SO is made obsolete by ChatGPT. That makes no sense.
dpkirchner 3y ago
IME ChatGPT provides more useful results than SO, especially because ChatGPT doesn't try to guess what I'm asking and takes my request at face value (by its very nature). If ChatGPT struggles with a particular topic, companies like OpenAI could hire some subject matter experts (SMEs) to write material that could be used for future training. I don't see how SO can survive much longer. Two things SO could do to beat ChatGPT as it exists now: 1) Require all questions and answers to include the names of tools and their version numbers. Hide Qs and As until those are added. This is so critical, I have no idea why it wasn't required since day 1. 2) Discourage answers that don't answer the question, through moderation. If you don't know the answer, don't post, or at least leave your notes as a comment. Bonus: Stop removing chats. The number of times we see someone link to a chat that 404s is too damned high. Cf: complaints from folks about the unsearchability of Discord chats wrt software tools.
- em-bee 3y ago
  i never understood why new users are allowed to post answers but not comments. it doesn't make sense. i mean it makes sense to restrict what new users can do, but the current mode results in new users posting comments as answers, and it irritates me every time i see one. on the other hand maybe it avoids answers as comments. but in the end maybe it should just be easier to convert between the two. i like your idea of including versions.

by roenxi 3y ago

dwheeler 3y ago

I think it's very likely that StackExchange is the key data source for ChatGPT answers of this kind. If you want LLMs to give good answers, it's important to have their sources correct.

viraptor 3y ago

Time passing is the circumstance. ChatGPT knows the answers from SO, but for new languages the quality will degrade over time. ChatGPT already isn't great at answering questions about new version of Swift for example. It won't be able to use new languages / APIs either. Given how rarely the model is updated (officially anyway), it will always lag behind the new tech. It may be actually in the interest of OpenAI to start paying SO for the data and make sure it flows to the community in some way. Otherwise they will be starved.

layer8 3y ago

This is just one aspect, but some people strongly prefer typing only a few search terms and then exploring links over holding a linear synchronous conversation with a chatbot. It’s a significantly different usage mode. It’s a bit like browsing through a book vs. interviewing a random person who read the book.

treis 3y ago

A SO post usually has an answer that works and is a reasonably good idea. ChatGPT lacks the ability to see if its responses pass even a basic sanity test. Sometimes it's answers are good and sometimes they are totally wrong. Hard for a person asking the question to determine which answer they got.

rolisz 3y ago
With code interpreter, ChatGPT can check that it's code at least runs without errors
- dharmab 3y ago
  rm -rf $INSTALL_DIR/* this line of code has at least two gotchas. if you run it to test it out, there is a good chance you will destroy your data. It is otherwise a valid line of code. This is not a theoretical problem: https://github.com/valvesoftware/steam-for-linux/issues/3671 https://github.com/valvesoftware/steam-for-linux/issues/3671
  - rolisz 3y ago
    ChatGPT warns about gotchas with that line.
    - danShumway 3y ago
      The point being that: > With code interpreter, ChatGPT can check that it's code at least runs without errors is not a solution to gotchas/caveats that build up over time that ChatGPT won't know about. ChatGPT can warn about caveats from that line of code because it's old code. It does not have a magical defense that will allow it to warn about caveats, errors, security risks, or deprecations that are introduced after its knowledge cutoff. The best solution that we've been able to find so far is for ChatGPT to do a search and feed the results into its context. But that has its own problems, and importantly it only works if the underlying search engines are still returning up-to-date relevant results and documentation from sites like StackOverflow.
- treis 3y ago
  Yes, but ultimately it's still a "statistically likely" answer and not the "right" one. Often the two overlap and maybe one day they'll make it an overlapping venn diagram. But that day is not today.

wodenokoto 3y ago

If SO can’t compete on hard problems then we should all change job titles to prompt engineers.

twelve40 3y ago
That's a strange claim. SO's short q-and-a format doesn't even begin to tackle hard problems.

remram 3y ago

We can't keep training ChatGPT on SO and then pretend that SO is made obsolete by ChatGPT. That makes no sense.

dpkirchner 3y ago

IME ChatGPT provides more useful results than SO, especially because ChatGPT doesn't try to guess what I'm asking and takes my request at face value (by its very nature). If ChatGPT struggles with a particular topic, companies like OpenAI could hire some subject matter experts (SMEs) to write material that could be used for future training. I don't see how SO can survive much longer. Two things SO could do to beat ChatGPT as it exists now: 1) Require all questions and answers to include the names of tools and their version numbers. Hide Qs and As until those are added. This is so critical, I have no idea why it wasn't required since day 1. 2) Discourage answers that don't answer the question, through moderation. If you don't know the answer, don't post, or at least leave your notes as a comment. Bonus: Stop removing chats. The number of times we see someone link to a chat that 404s is too damned high. Cf: complaints from folks about the unsearchability of Discord chats wrt software tools.

em-bee 3y ago
i never understood why new users are allowed to post answers but not comments. it doesn't make sense. i mean it makes sense to restrict what new users can do, but the current mode results in new users posting comments as answers, and it irritates me every time i see one. on the other hand maybe it avoids answers as comments. but in the end maybe it should just be easier to convert between the two. i like your idea of including versions.