4 ms·

Looking at that graph though, what it tells me is that stackoverflow was entirely unaffected by ChatGPT, and the giant spike is just humanity collectively jumpi

Looking at that graph though, what it tells me is that stackoverflow was entirely unaffected by ChatGPT, and the giant spike is just humanity collectively jumping on ChatGPT in addition to folks using Stackoverflow in exactly the same way as before? Same for that "answers over time" graph: that's literally what you'd expect for a knowledge base. There has to be an initial trend upwards as the site starts to accumulate information (i.e. questions with answers, not just questions) and initially gains popularity as a place to get help by asking questions. But, as the number of questions and answers increases, the utility changes from a place to "get help by asking questions" to "a place to get help by searching for answers". At that point, the curve levels off, and as the number of typical beginner and intermediate questions die off because they've already been asked and answered many times, the number of new questions has to trend downward. It'll be slow, because we're constantly inventing new programming languages, libraries, frameworks, etc, but the majority of questions for those are simply duplicates of things asked about previous languages, libraries, and frameworks, with their answers being perfectly usable by folks looking for answers. And this is exactly the graph we see: rather than a sign of decline, it's a graph that shows the change from a predominantly Q&A site to predominantly being a knowledge repository. ChatGPT will, of course, have some impact on the graph, but the number of posts on Stackoverflow today that are "How do I X? I asked ChatGPT but that didn't help me" is staggering. ChatGPT may have initially driven down searches across the rest of the web, from Google to Stackoverflow to your mom's recipe website, but the novelty's already worn off, and overall it's done precious little to cut into the utility of Stackoverflow, as far as this data suggests.

SV_BubbleTime 3y ago
This is a good point. With the infinite number of uses outside of programming, you wouldn’t expect to see SO interest correlate with GPT. And if it’s not negative that’s a win. SO has accessibility problems for beginners and a meta game for more advanced users. But there is no evidence of a death spiral here. If anything… GPT could help SO by relieving the massive number of basic questions that GPT actually can answer.
- TheRealPomax 3y ago
  The simple questions are certainly the ones that ChatGPT is least likely to reconstruct wrong, so especially for super popular languages with huge bodies of work on the internet, you'd expect ChatGPT to act as a decent pre-filter, finally ensuring folks don't end up asking "how do I sort this list based on properties inside my objects" for the millionth time =)
indeed30 3y ago
> At that point, the curve levels off, and as the number of typical questions die off because they've already been asked and answered many times, the number of new questions has to trend downward Yes, absolutely, and the article addresses precisely that point. But the questions graph shows a drop of about 40% at exactly the point in time when ChatGPT launches, which is much too extreme to explain away like that.
- TheRealPomax 3y ago
  Except the graph cuts off before 2023, so it tells us nothing about the actual effect it had, it cherry picked the initial drop off (or worse, they just used https://insights.stackoverflow.com/trends https://insights.stackoverflow.com/trends, didn't even look at the time axis, and went "this proves our point"). It's Q3 of 2023, if you're going to argue a decline, why are you not looking at the most important three quarters of data?
  - indeed30 3y ago
    No it doesn't - note the axis marks are every two years.
    - TheRealPomax 3y ago
      Ah, true. That said, some of these are going back up. Java and C#, for instance, are clearly trending up again, as is Swift. Python's down, but by 10%ish, not 40%. And JS is down a lot but that's the one language you'd expect ChatGPT to actually be the best at "helping" with, given that it's the web's own programming language, so it being trained disproportionately more on JS than anything else would be hardly surprising. And then there's languages for which ChatGPT might as well not exist: Lisp, Go, Rust, etc. Heck, if we were to look at Rust, for instance, ChatGPT did literally nothing. Its rise has been, and continues to be, meteoric. And if we look at arduino questions, you might even conclude that ChatGPT was a factor in driving people to Stackoverflow, not away from it. Data science is hard, and certainly much harder than most folks (especially those with a programming background) think it is.
bluetomcat 3y ago
It's a negative dynamic that starts from the popular, "low-hanging fruit" kinds of questions like "how do you subtract 2 dates in framework X". Such a question would gain many points both to the OP and the answerers. The best answer to a question like that is short, unambiguous and concise. Consequently, you have high-reputation users skilled in answering popular questions. Then the popular questions are mostly over and people start asking questions about a particular problem hidden in their piece of code, and high-rep users sneer at it because they see it as "not researched".
- anigbrowl 3y ago
  high-rep users sneer at it That sneering and the general idea that some questions are worthy and others not is one of the most offputting things about SE, followed by duplicative and unhelpful me-too answers that are very obvious reputation farming. When I took up programming again after a long hiatus I found SE very helpful at first but got sick of it within a matter of months because the meta game/* is horrible. * the social dynamics in a community driven website that are wholly orthogonal to and often end up subverting the site's stated purpose by leveraging the stated ethos and decision infrastructure in pursuit of selfish ends. Other examples include Wikipedia edit wars or abusive forms of legalism and political brinksmanship.
  - sam0x17 3y ago
    Yeah this always turned me off of contributing in earnest to SE. The purpose is supposed to be a general purpose site with an answer to _all_ questions, but in practice it's just a site with answers to questions an expert might have. Almost everything below that level of technicality gets culled.
  - thaeli 3y ago
    The biggest problem with SE for me, and this is related to the culture issues you're talking about, is that the site has no good way of deprecating "formerly correct" answers. Even if a better, more correct answer is posted later, the reputation system has a huge incumbency bias in favor of older answers that have accumulated upvotes by being the best available answer at the time. Their knowledge repository is slowly rotting under the weight of having to ask every time "okay, is this correct-sounding, highly upvoted answer actually (still) correct, or is it 10 years out of date?"
mlinhares 3y ago
As a longtime contributor and moderator on SO this is, unfortunately, very true. The community has always been toxic, being a newbie on SO requires a really thick skin or being willing to withstand a lot of abuse to use the site (if you're asking new questions). The fact that there's very no off topic also makes it nearly impossible to "build" a community out of the site as well, it's all too focused on just questions and answers and I feel no need to go back to talk to anyone because I've never met anyone at SO while I made many longtime friends in other forums. It also saddens me that SO has killed all these smaller and more or less insular communities, i see very little movement on the old mailing lists and forums that are still up, all questions moved to SO in a way or another (even for non-English speaking communities) so I'm not sure if the site has been positive for the tech community in general. I had so much coaching and help from people in the local and regional communities I was a part of when I was in college and later starting my career and now there's nothing like it around anymore because SO sucked all the oxygen around them. Not sure where i'd be if I was starting right now.

SV_BubbleTime 3y ago

This is a good point. With the infinite number of uses outside of programming, you wouldn’t expect to see SO interest correlate with GPT. And if it’s not negative that’s a win. SO has accessibility problems for beginners and a meta game for more advanced users. But there is no evidence of a death spiral here. If anything… GPT could help SO by relieving the massive number of basic questions that GPT actually can answer.

TheRealPomax 3y ago
The simple questions are certainly the ones that ChatGPT is least likely to reconstruct wrong, so especially for super popular languages with huge bodies of work on the internet, you'd expect ChatGPT to act as a decent pre-filter, finally ensuring folks don't end up asking "how do I sort this list based on properties inside my objects" for the millionth time =)

indeed30 3y ago

> At that point, the curve levels off, and as the number of typical questions die off because they've already been asked and answered many times, the number of new questions has to trend downward Yes, absolutely, and the article addresses precisely that point. But the questions graph shows a drop of about 40% at exactly the point in time when ChatGPT launches, which is much too extreme to explain away like that.

TheRealPomax 3y ago
Except the graph cuts off before 2023, so it tells us nothing about the actual effect it had, it cherry picked the initial drop off (or worse, they just used https://insights.stackoverflow.com/trends https://insights.stackoverflow.com/trends, didn't even look at the time axis, and went "this proves our point"). It's Q3 of 2023, if you're going to argue a decline, why are you not looking at the most important three quarters of data?
- indeed30 3y ago
  No it doesn't - note the axis marks are every two years.
  - TheRealPomax 3y ago
    Ah, true. That said, some of these are going back up. Java and C#, for instance, are clearly trending up again, as is Swift. Python's down, but by 10%ish, not 40%. And JS is down a lot but that's the one language you'd expect ChatGPT to actually be the best at "helping" with, given that it's the web's own programming language, so it being trained disproportionately more on JS than anything else would be hardly surprising. And then there's languages for which ChatGPT might as well not exist: Lisp, Go, Rust, etc. Heck, if we were to look at Rust, for instance, ChatGPT did literally nothing. Its rise has been, and continues to be, meteoric. And if we look at arduino questions, you might even conclude that ChatGPT was a factor in driving people to Stackoverflow, not away from it. Data science is hard, and certainly much harder than most folks (especially those with a programming background) think it is.

bluetomcat 3y ago

It's a negative dynamic that starts from the popular, "low-hanging fruit" kinds of questions like "how do you subtract 2 dates in framework X". Such a question would gain many points both to the OP and the answerers. The best answer to a question like that is short, unambiguous and concise. Consequently, you have high-reputation users skilled in answering popular questions. Then the popular questions are mostly over and people start asking questions about a particular problem hidden in their piece of code, and high-rep users sneer at it because they see it as "not researched".

anigbrowl 3y ago
high-rep users sneer at it That sneering and the general idea that some questions are worthy and others not is one of the most offputting things about SE, followed by duplicative and unhelpful me-too answers that are very obvious reputation farming. When I took up programming again after a long hiatus I found SE very helpful at first but got sick of it within a matter of months because the meta game/* is horrible. * the social dynamics in a community driven website that are wholly orthogonal to and often end up subverting the site's stated purpose by leveraging the stated ethos and decision infrastructure in pursuit of selfish ends. Other examples include Wikipedia edit wars or abusive forms of legalism and political brinksmanship.
- sam0x17 3y ago
  Yeah this always turned me off of contributing in earnest to SE. The purpose is supposed to be a general purpose site with an answer to _all_ questions, but in practice it's just a site with answers to questions an expert might have. Almost everything below that level of technicality gets culled.
- thaeli 3y ago
  The biggest problem with SE for me, and this is related to the culture issues you're talking about, is that the site has no good way of deprecating "formerly correct" answers. Even if a better, more correct answer is posted later, the reputation system has a huge incumbency bias in favor of older answers that have accumulated upvotes by being the best available answer at the time. Their knowledge repository is slowly rotting under the weight of having to ask every time "okay, is this correct-sounding, highly upvoted answer actually (still) correct, or is it 10 years out of date?"

mlinhares 3y ago

As a longtime contributor and moderator on SO this is, unfortunately, very true. The community has always been toxic, being a newbie on SO requires a really thick skin or being willing to withstand a lot of abuse to use the site (if you're asking new questions). The fact that there's very no off topic also makes it nearly impossible to "build" a community out of the site as well, it's all too focused on just questions and answers and I feel no need to go back to talk to anyone because I've never met anyone at SO while I made many longtime friends in other forums. It also saddens me that SO has killed all these smaller and more or less insular communities, i see very little movement on the old mailing lists and forums that are still up, all questions moved to SO in a way or another (even for non-English speaking communities) so I'm not sure if the site has been positive for the tech community in general. I had so much coaching and help from people in the local and regional communities I was a part of when I was in college and later starting my career and now there's nothing like it around anymore because SO sucked all the oxygen around them. Not sure where i'd be if I was starting right now.