3 ms·

> We observe significant declines in both website visits and question volumes at Stack Overflow, particularly around topics where ChatGPT excels. By contrast, a

> We observe significant declines in both website visits and question volumes at Stack Overflow, particularly around topics where ChatGPT excels. By contrast, activity in Reddit communities shows no evidence of decline, suggesting the importance of social fabric as a buffer against the community-degrading effects of LLMs. This equates declining activity with degraded community, which I don’t think is a valid assumption to make. Taking Stack Overflow as an example, LLMs should be decimating question volumes there. That’s the whole point of LLMs! To answer questions! But which questions are being eliminated? The boring, rote ones that can be answered by a glance at the documentation? Or the interesting ones that need expert knowledge and insight? My guess is that LLMs displace the majority of the former but less of the latter. So what’s left once you take the lazy spam questions away from Stack Overflow? I would say that if LLMs can act as first-line support for those questions instead of expecting humans to answer them manually, the result will be a healthier community that more experts are willing to participate in. And sure enough, the paper reports: > upon ChatGPT’s release, a systematic rise began to take place, such that users were increasingly likely to be more established, older accounts. The implication of this result is that newer user accounts became systematically less likely to participate in the Stack Overflow community after ChatGPT became available. Figure 6 depicts the effects, indicating that questions exhibited a systematic rise in complexity following the release of ChatGPT. That looks like an improvement to me, not degradation.

doe_eyes 2y ago
I think the mistake you might be making here is that some level of routine engagement and mentorship is needed to sustain the community. If I'm an expert, but I can only say something useful once every six months when some super-niche question drops, I'm just gonna stop visiting the site because of the negative feedback loop of going there and not finding anything to do 99% of the time. The other extreme - drowning in "noob" questions - is also harmful, which is why you end up with FAQs and rules. But there are human dynamics that underpin all these communities, and LLMs are undoubtedly destroying the incentives to participate.
- svara 2y ago
  But that's really not how it is - LLMs have their uses, obviously, but they really fall apart as soon as you're trying to do something even just slightly off the beaten path. As such, they're certainly still useful for answering questions for which StackOverflow has a dozen duplicates, and that may very well be a net positive for StackOverflow. Here's an example of something that came up in my work the other day - fiddling with prompts and checking the responses is ultimately more work than just writing the code from scratch. GPT4 and Sonnet 3.5 never came close to a correct response, even though the code always looks sort of "directionally correct". https://chatgpt.com/share/ba85ced9-63d6-4fb3-8dc1-b8efbaa58fa5 https://chatgpt.com/share/ba85ced9-63d6-4fb3-8dc1-b8efbaa58f... And that's despite this almost certainly being somewhere on Github in some form.
- JimDabell 2y ago
  > If I'm an expert, but I can only say something useful once every six months when some super-niche question drops > The other extreme - drowning in "noob" questions - is also harmful, which is why you end up with FAQs and rules. This isn’t reflective of what actually happened with Stack Overflow though. These are not equal dangers. There were plenty of reasonable questions, then it became overwhelmed by people who didn’t want to read tutorials or documentation but instead wanted other people to do their work for them or to have the documentation looked up and recited to them by a person not a search engine. Stack Overflow didn’t much care because it’s increased engagement, but the expert users absolutely did care and started to leave. “There aren’t enough interesting questions” is not a danger. “Experts don’t want to be a flesh interface to a search engine” is not only a danger but has already been harming the community for a long time. LLMs have the chance to reverse this damage, and they can do this by selectively reducing the volume of the harmful engagement. So I am rejecting the notion that declining activity necessarily means degraded community.

doe_eyes 2y ago

I think the mistake you might be making here is that some level of routine engagement and mentorship is needed to sustain the community. If I'm an expert, but I can only say something useful once every six months when some super-niche question drops, I'm just gonna stop visiting the site because of the negative feedback loop of going there and not finding anything to do 99% of the time. The other extreme - drowning in "noob" questions - is also harmful, which is why you end up with FAQs and rules. But there are human dynamics that underpin all these communities, and LLMs are undoubtedly destroying the incentives to participate.

svara 2y ago
But that's really not how it is - LLMs have their uses, obviously, but they really fall apart as soon as you're trying to do something even just slightly off the beaten path. As such, they're certainly still useful for answering questions for which StackOverflow has a dozen duplicates, and that may very well be a net positive for StackOverflow. Here's an example of something that came up in my work the other day - fiddling with prompts and checking the responses is ultimately more work than just writing the code from scratch. GPT4 and Sonnet 3.5 never came close to a correct response, even though the code always looks sort of "directionally correct". https://chatgpt.com/share/ba85ced9-63d6-4fb3-8dc1-b8efbaa58fa5 https://chatgpt.com/share/ba85ced9-63d6-4fb3-8dc1-b8efbaa58f... And that's despite this almost certainly being somewhere on Github in some form.
JimDabell 2y ago
> If I'm an expert, but I can only say something useful once every six months when some super-niche question drops > The other extreme - drowning in "noob" questions - is also harmful, which is why you end up with FAQs and rules. This isn’t reflective of what actually happened with Stack Overflow though. These are not equal dangers. There were plenty of reasonable questions, then it became overwhelmed by people who didn’t want to read tutorials or documentation but instead wanted other people to do their work for them or to have the documentation looked up and recited to them by a person not a search engine. Stack Overflow didn’t much care because it’s increased engagement, but the expert users absolutely did care and started to leave. “There aren’t enough interesting questions” is not a danger. “Experts don’t want to be a flesh interface to a search engine” is not only a danger but has already been harming the community for a long time. LLMs have the chance to reverse this damage, and they can do this by selectively reducing the volume of the harmful engagement. So I am rejecting the notion that declining activity necessarily means degraded community.