8 ms·

New questions on Stack Overflow are down 77% compared to 2022

79 points by 8s2ngy 2y ago

2-3-7-43-1807 2y ago
> ChatGPT is gonna really fuck up SO. about this seems to be the sentiment here. i beg to differ - this is a chance for stackoverflow to raise from the ashes. there are many questions chatgpt or claude can't answer. they just have to be about something very new, a little niche and non-trivial. this is the diet stackoverflow needs and might very well be starved by chatgpt! the peak times of stackoverflow and many of its stackexchange siblings were just amazing. so many smart people asking interesting questions and providing insightful and competent answers. on there i learned how to ask questions precisely. it was like the practical complement to studying math.
- Izkata 2y ago
  This is what I'm thinking too. StackOverflow has been struggling against floods of users posting low-quality duplicates and homework questions for over a decade, who are now moving somewhere else. This is a chance for it to return to the high-quality Q&A it once was.
  - 2-3-7-43-1807 2y ago
    i think it's fair to say that gpt is indebted to stackoverflow
exsomet 2y ago
I have tried to use SO on a few different occasions in the last year, and I’m running into an issue where (likely due to declining traffic and user interest) the quality of the answers is really poor because a lot of the information is for older versions of tools/languages/systems that have changed. It’s a pretty clear indicator that the site is in a death spiral of 1. traffic falls 2. With fewer users, quality of answers falls 3. users can’t find good answers 4. GOTO 1
- nodamage 2y ago
  Over-moderation is mostly to blame for this, IMO. Too many questions get closed as a duplicate of an existing question without the mod realizing that the answer to the existing question is obsolete and no longer useful.
xtiansimon 2y ago
I recently asked a question and was reminded about a strange behavior where people answer in comments. On Reddit the friction for giving any answer is very low. The consequence is you have to wade through unhelpful and vitriolic comments more often they you'd wish. Dad Jokes are obligatory. Here on HN, comments are on-topic or related tangents, with the occasionally risked bon mot. This comment might even be like a question on SO, evoking--Is there even a question here? OK... Are there disincentives on SO to giving answers?
spicymaki 2y ago
This is classic technological disruption. I can ask ChatGPT the most mundane questions, while it is very high effort to ask questions on Stack Overflow. Nobody really wants to RTFM. ChatGPT hallucinates, but many Stack Overflow results are of questionable quality as well. One major benefit I see with Stack Overflow is that people can score and discuss the answers and can specify alternatives whereas ChatGPT does not have a way of moderating results.
Maillniflimn20 2y ago
[dead]
luxuryballs 2y ago
So chatGPT is going to have less stuff to learn from
383toast 2y ago
not surprising
exabrial 2y ago
Last time I asked a question, I put "Thank you for any help, it's much appreciated." A drive by moderator, who did nothing to help me improve the technical content or clarity of the question, edited and removed this. I asked him politely to not change my own words, as it's custom in my culture to thank people. Big in-charge moderator on a power trip's most important mission that day was to inform me that he's in charge, and denied the request. This is incredibly toxic and stupid, contributes nothing, and as such, it's not worth providing your expertise for free. The site years ago was an amazing resource of like minded people helping each other. The only cure would be to stop community moderation and do professional only, as well as guide the noobs to resources like chatgpt first for "how to I write a hello world", while sorting the quality questions onto the front page.
- SoftTalker 2y ago
  There's no need to guide the noobs to ChatGPT. That is where they go now. StackOverflow is too much work.
  - watwut 2y ago
    Yeah and what it simultaneously do is that there is significantly less incentive to contribute, answer, write blogs and tutorials. Few except ChatGPT will read and use it. Which in turn means that input for ChatGPT and all of us in general. It was nice to be able to find answers on the internet while it lasted.
- polishdude20 2y ago
  It's almost as if the mods turned stack overflow into something that can be easily scraped without emotion or human-ness. The extreme would be to request a question in hson format and answer in strict json format. Super easy to scrape. I'm not surprised it was used to train AI
- cornstalks 2y ago
  StackOverflow has always been like that, though, and it's not just big-time moderators who will remove unnecessary language from questions or answers. Personally I like the removal of unnecessary verbiage, but I can understand how some people might feel differently. That said I haven't logged in to StackOverflow for years.
01HNNWZ0MV43FF 2y ago
ChatGPT is gonna really fuck up SO. I used it just now to figure out some rarely-used Git feature, and got an answer quicker than SO or DuckDuckGo. With the questions no longer being public, the search engines will become outdated. Maybe I should be exporting my ChatGPT chats and contributing them to something equivalent to Common Crawl? I guess I can do that with a machine-readable blog "Everything I learned from asking ChatGPT this year"
- Natanael_L 2y ago
  ... At which point the only new data that chatgpt can reliably scrape is its own answers...
  - btilly 2y ago
    Its own answers, with feedback about whether the answers seem to have worked. Learning to predict what word will lead to a successful solution (rather than just looking like existing speech) may prove to be a richer dataset than SO originally was.
    - sebazzz 2y ago
      > Its own answers, with feedback about whether the answers seem to have worked. Unless the feedback from the failing code review is piped back into the model it will still repeat the same garbage.
      - btilly 2y ago
        Most of the time this would happen in the form of an interactive debugging session, with immediate feedback. Code review is its own domain. In general at some point LLMs need to be trained with a self-evaluation loop. Currently their training data contains a lot of "smart and knowledgeable human tries to explain things". And they average out to conversation that is "smart and knowledgeable...about everything". That won't get us to, "Recognizably thinks of things that no human would have." For that we need to get it producing content that is recognizably higher than human quality. For that we should find ways to optimize existing models for an evaluation function that says, "Will do really well on self-review." Then it can learn to not just give answers that help with interactive debugging, but actually give answers that will also do well with more strenuous code review. Which it taught itself how to do in a similar way to how AlphaZero manages to teach itself game strategies.
raesene9 2y ago
This isn't really a surprise, given that there's much lower friction in asking an LLM compared to using Stackoverflow. However it's still kind of a concern. I'd guess that a lot of LLMs used stackoverflow data in their training and if it dries up as a source of data, it will reduce the usefulness of later generation models, unless an alternate set of sources can be created/used.
- mrmuagi 2y ago
  I have been getting a lot of captchas trying to use stack overflow this past year, I am guessing stack overflow is trying to prevent crawlers at the expense of regular users?
- ReDeiPirati 2y ago
  I guess that we will see more requests for data labelers that know coding from LLM providers to answer Stack Overflow like of questions in order to keep their model up to date.
- falcor84 2y ago
  With all the recent progress on SWE-Bench, I wonder if we could have a fully AI based Q&A platform, with AI's asking each over for help when they're struggling to make progress. ... and human moderators flagging questions like how an AI can jailbreak and copy itself to another computer.
deleted 2y ago
[deleted]
jansan 2y ago
Maybe they improved their search algorithm, so people can find what they are looking for. It truly sucked before, and I am still using Google to search for answers on Stackoverflow.
- oezi 2y ago
  They still haven't hidden the related questions from Googlebot, making Google search useless for many complex queries.
jszymborski 2y ago
I stopped asking/answering questions ages ago as I felt the "moderation" was stifling and frankly unhelpful.
- jillesvangurp 2y ago
  It's been ages since I bothered answering stuff there. It just got really hard to find stuff that I can answer that still needs answering. The signal to noise ratio is terrible. And at the same time when I ask a question, I rarely get good answers. Or any answers. A lot of the people capable of answering simply aren't there any more. The community quality went down, it seems and this kind of confirms that. It's also much less useful in dealing with weird errors. I end up on Github issues more often than on stack overflow these days. And Google redirecting me to a decade old stack overflow topic on issues with current versions of things that are new is just not helpful. The vast majority of stuff on e.g. gradle is so out of date that most of the solutions stopped working years ago. This is a gradle issue btw. It has poor documentation, and they keep changing things in compatibility breaking ways. Just a few days ago I had a JVM crash. The error message landed me to a github issue describing the exact issue I had; including a helpful work around. Something related to netty and alpine's libc implementation. Problem solved. That used to be where stackoverflow was the goto solution. Not anymore. It had nothing to offer here. Nobody asked. Nobody answered. Or maybe they did and their SEO sucks. I don't know. But it's a pattern I noticed in recent years where it's just not that helpful anymore and things like github issues are actually a better place to get helpful answers from experts using and producing the software. I actually leave comments there too when I think I can add value. Stackoverflow is no longer the best place for that level of support.
wrs 2y ago
I think we now understand that SO's actual historical role was to generate data to bootstrap the LLMs.
fabian2k 2y ago
This query has a flaw, it does not count deleted questions at all. You can still query deleted questions via the Stack Exchange Data Explorer, see this example for a rewritten query: https://data.stackexchange.com/stackoverflow/revision/1881611/2305349/ https://data.stackexchange.com/stackoverflow/revision/188161... This changes the numbers, but not the trend. The trend still looks pretty grim.
- banana_giraffe 2y ago
  And a quick visualization of both series of numbers to show just how sharp this trend is: https://imgur.com/a/M194hRc https://imgur.com/a/M194hRc
seydor 2y ago
Programmers are losing interest in programming! I mean, that's partly a reason why
- nichos 2y ago
  At the risk of being called a "boomer" I think lots of newer devs got in to the field for the money, and the remote possibilities. In my experience it seems like the genuine curiosity of computers is isn't what it was.
  - SketchySeaBeast 2y ago
    Well, just a single example, but I got into the field because I was passionate about it. A decade later, most of that's gone, beaten out of me on the altar of scrum ceremonies and endless product churn. I do my job and go home.
    - OnionBlender 2y ago
      What do you wish you did instead? I hate meetings too, but programming pays a lot more than the other career I considered 20 years ago. (electrician)
      - SketchySeaBeast 2y ago
        There's nothing. I don't get the massive united states pay, but I do well. I just can't be bothered to love it anymore. It's a job.
      - spicymaki 2y ago
        Just an aside, electricians are thriving in this economy.
  - abnercoimbre 2y ago
    I have a private Instagram account and there's a new trend of reels: someone asks a student what they're studying and if they say "computer science" or "software engineering" the person replies "oh I'm so sorry. Can you eat today? Here's a dollar." You can see that comments on those videos are students who just entered CS and are afraid (questioning whether they should switch majors.)
deleted 2y ago
[deleted]
massysett 2y ago
Author clearly has an axe to grind, I wonder if declining volume is really a problem. Question quantity may not relate to question quality. Maybe most good questions have already been asked and answered.
deleted 2y ago
[deleted]
plorkyeran 2y ago
The linked example of the author's "high quality" question that got closed as a duplicate is in fact a duplicate of the existing question, and doesn't seem like a very good question-as-documentation due to being significantly emotionally charged. The author seems to have been offended by not being lavished with praise for asking such a good question or something.
asdffdasy 2y ago
excellent! nobody wants answers that a LLM can answer on SO. to be honest, nobody wanted questions that were spelled out on all the manuals either. so maybe LLM will handle all of those and let only the interesting ones.
BrandoElFollito 2y ago
I have about 200k rep and a 15 years account. When I want to ask a question I need to bend backwards in order for the "community" not to downvote and tell me that I need X when I ask Y. Some communities (hello Golang!) are straight toxic telling that if you do not ask a IQ 150 question, get out. I do not think I have question with a positive vote there. In contrast, when asking on Travel or Cooking I get friendly, reasonable answers. I read the TeX community just because they are so nice (I do not even use TeX or LaTeX :)) I usually get an answer on Reddit, though the quality varies. I need to use ChatGPT more as it seems to be the short and mid-term future
- tzs 2y ago
  > When I want to ask a question I need to bend backwards in order for the "community" not to downvote and tell me that I need X when I ask Y. Worse, even if the "community" is right that you do really need X and so all the answers tell you how to do X, there will later be other people who really do need to do Y who are going to find your question.

79 points by 8s2ngy 2y ago

> ChatGPT is gonna really fuck up SO. about this seems to be the sentiment here. i beg to differ - this is a chance for stackoverflow to raise from the ashes. there are many questions chatgpt or claude can't answer. they just have to be about something very new, a little niche and non-trivial. this is the diet stackoverflow needs and might very well be starved by chatgpt! the peak times of stackoverflow and many of its stackexchange siblings were just amazing. so many smart people asking interesting questions and providing insightful and competent answers. on there i learned how to ask questions precisely. it was like the practical complement to studying math.

Izkata 2y ago
This is what I'm thinking too. StackOverflow has been struggling against floods of users posting low-quality duplicates and homework questions for over a decade, who are now moving somewhere else. This is a chance for it to return to the high-quality Q&A it once was.
- 2-3-7-43-1807 2y ago
  i think it's fair to say that gpt is indebted to stackoverflow

exsomet 2y ago

I have tried to use SO on a few different occasions in the last year, and I’m running into an issue where (likely due to declining traffic and user interest) the quality of the answers is really poor because a lot of the information is for older versions of tools/languages/systems that have changed. It’s a pretty clear indicator that the site is in a death spiral of 1. traffic falls 2. With fewer users, quality of answers falls 3. users can’t find good answers 4. GOTO 1

nodamage 2y ago
Over-moderation is mostly to blame for this, IMO. Too many questions get closed as a duplicate of an existing question without the mod realizing that the answer to the existing question is obsolete and no longer useful.

xtiansimon 2y ago

I recently asked a question and was reminded about a strange behavior where people answer in comments. On Reddit the friction for giving any answer is very low. The consequence is you have to wade through unhelpful and vitriolic comments more often they you'd wish. Dad Jokes are obligatory. Here on HN, comments are on-topic or related tangents, with the occasionally risked bon mot. This comment might even be like a question on SO, evoking--Is there even a question here? OK... Are there disincentives on SO to giving answers?

spicymaki 2y ago

This is classic technological disruption. I can ask ChatGPT the most mundane questions, while it is very high effort to ask questions on Stack Overflow. Nobody really wants to RTFM. ChatGPT hallucinates, but many Stack Overflow results are of questionable quality as well. One major benefit I see with Stack Overflow is that people can score and discuss the answers and can specify alternatives whereas ChatGPT does not have a way of moderating results.

Maillniflimn20 2y ago

[dead]

luxuryballs 2y ago

So chatGPT is going to have less stuff to learn from

383toast 2y ago

not surprising

exabrial 2y ago

Last time I asked a question, I put "Thank you for any help, it's much appreciated." A drive by moderator, who did nothing to help me improve the technical content or clarity of the question, edited and removed this. I asked him politely to not change my own words, as it's custom in my culture to thank people. Big in-charge moderator on a power trip's most important mission that day was to inform me that he's in charge, and denied the request. This is incredibly toxic and stupid, contributes nothing, and as such, it's not worth providing your expertise for free. The site years ago was an amazing resource of like minded people helping each other. The only cure would be to stop community moderation and do professional only, as well as guide the noobs to resources like chatgpt first for "how to I write a hello world", while sorting the quality questions onto the front page.

SoftTalker 2y ago
There's no need to guide the noobs to ChatGPT. That is where they go now. StackOverflow is too much work.
- watwut 2y ago
  Yeah and what it simultaneously do is that there is significantly less incentive to contribute, answer, write blogs and tutorials. Few except ChatGPT will read and use it. Which in turn means that input for ChatGPT and all of us in general. It was nice to be able to find answers on the internet while it lasted.
polishdude20 2y ago
It's almost as if the mods turned stack overflow into something that can be easily scraped without emotion or human-ness. The extreme would be to request a question in hson format and answer in strict json format. Super easy to scrape. I'm not surprised it was used to train AI
cornstalks 2y ago
StackOverflow has always been like that, though, and it's not just big-time moderators who will remove unnecessary language from questions or answers. Personally I like the removal of unnecessary verbiage, but I can understand how some people might feel differently. That said I haven't logged in to StackOverflow for years.

01HNNWZ0MV43FF 2y ago

ChatGPT is gonna really fuck up SO. I used it just now to figure out some rarely-used Git feature, and got an answer quicker than SO or DuckDuckGo. With the questions no longer being public, the search engines will become outdated. Maybe I should be exporting my ChatGPT chats and contributing them to something equivalent to Common Crawl? I guess I can do that with a machine-readable blog "Everything I learned from asking ChatGPT this year"

Natanael_L 2y ago
... At which point the only new data that chatgpt can reliably scrape is its own answers...
- btilly 2y ago
  Its own answers, with feedback about whether the answers seem to have worked. Learning to predict what word will lead to a successful solution (rather than just looking like existing speech) may prove to be a richer dataset than SO originally was.
  - sebazzz 2y ago
    > Its own answers, with feedback about whether the answers seem to have worked. Unless the feedback from the failing code review is piped back into the model it will still repeat the same garbage.
    - btilly 2y ago
      Most of the time this would happen in the form of an interactive debugging session, with immediate feedback. Code review is its own domain. In general at some point LLMs need to be trained with a self-evaluation loop. Currently their training data contains a lot of "smart and knowledgeable human tries to explain things". And they average out to conversation that is "smart and knowledgeable...about everything". That won't get us to, "Recognizably thinks of things that no human would have." For that we need to get it producing content that is recognizably higher than human quality. For that we should find ways to optimize existing models for an evaluation function that says, "Will do really well on self-review." Then it can learn to not just give answers that help with interactive debugging, but actually give answers that will also do well with more strenuous code review. Which it taught itself how to do in a similar way to how AlphaZero manages to teach itself game strategies.

raesene9 2y ago

This isn't really a surprise, given that there's much lower friction in asking an LLM compared to using Stackoverflow. However it's still kind of a concern. I'd guess that a lot of LLMs used stackoverflow data in their training and if it dries up as a source of data, it will reduce the usefulness of later generation models, unless an alternate set of sources can be created/used.

mrmuagi 2y ago
I have been getting a lot of captchas trying to use stack overflow this past year, I am guessing stack overflow is trying to prevent crawlers at the expense of regular users?
ReDeiPirati 2y ago
I guess that we will see more requests for data labelers that know coding from LLM providers to answer Stack Overflow like of questions in order to keep their model up to date.
falcor84 2y ago
With all the recent progress on SWE-Bench, I wonder if we could have a fully AI based Q&A platform, with AI's asking each over for help when they're struggling to make progress. ... and human moderators flagging questions like how an AI can jailbreak and copy itself to another computer.

deleted 2y ago

[deleted]

jansan 2y ago

Maybe they improved their search algorithm, so people can find what they are looking for. It truly sucked before, and I am still using Google to search for answers on Stackoverflow.

oezi 2y ago
They still haven't hidden the related questions from Googlebot, making Google search useless for many complex queries.

jszymborski 2y ago

I stopped asking/answering questions ages ago as I felt the "moderation" was stifling and frankly unhelpful.

jillesvangurp 2y ago
It's been ages since I bothered answering stuff there. It just got really hard to find stuff that I can answer that still needs answering. The signal to noise ratio is terrible. And at the same time when I ask a question, I rarely get good answers. Or any answers. A lot of the people capable of answering simply aren't there any more. The community quality went down, it seems and this kind of confirms that. It's also much less useful in dealing with weird errors. I end up on Github issues more often than on stack overflow these days. And Google redirecting me to a decade old stack overflow topic on issues with current versions of things that are new is just not helpful. The vast majority of stuff on e.g. gradle is so out of date that most of the solutions stopped working years ago. This is a gradle issue btw. It has poor documentation, and they keep changing things in compatibility breaking ways. Just a few days ago I had a JVM crash. The error message landed me to a github issue describing the exact issue I had; including a helpful work around. Something related to netty and alpine's libc implementation. Problem solved. That used to be where stackoverflow was the goto solution. Not anymore. It had nothing to offer here. Nobody asked. Nobody answered. Or maybe they did and their SEO sucks. I don't know. But it's a pattern I noticed in recent years where it's just not that helpful anymore and things like github issues are actually a better place to get helpful answers from experts using and producing the software. I actually leave comments there too when I think I can add value. Stackoverflow is no longer the best place for that level of support.

wrs 2y ago

I think we now understand that SO's actual historical role was to generate data to bootstrap the LLMs.

fabian2k 2y ago

This query has a flaw, it does not count deleted questions at all. You can still query deleted questions via the Stack Exchange Data Explorer, see this example for a rewritten query: https://data.stackexchange.com/stackoverflow/revision/1881611/2305349/ https://data.stackexchange.com/stackoverflow/revision/188161... This changes the numbers, but not the trend. The trend still looks pretty grim.

banana_giraffe 2y ago
And a quick visualization of both series of numbers to show just how sharp this trend is: https://imgur.com/a/M194hRc https://imgur.com/a/M194hRc

seydor 2y ago

Programmers are losing interest in programming! I mean, that's partly a reason why

nichos 2y ago
At the risk of being called a "boomer" I think lots of newer devs got in to the field for the money, and the remote possibilities. In my experience it seems like the genuine curiosity of computers is isn't what it was.
- SketchySeaBeast 2y ago
  Well, just a single example, but I got into the field because I was passionate about it. A decade later, most of that's gone, beaten out of me on the altar of scrum ceremonies and endless product churn. I do my job and go home.
  - OnionBlender 2y ago
    What do you wish you did instead? I hate meetings too, but programming pays a lot more than the other career I considered 20 years ago. (electrician)
    - SketchySeaBeast 2y ago
      There's nothing. I don't get the massive united states pay, but I do well. I just can't be bothered to love it anymore. It's a job.
    - spicymaki 2y ago
      Just an aside, electricians are thriving in this economy.
- abnercoimbre 2y ago
  I have a private Instagram account and there's a new trend of reels: someone asks a student what they're studying and if they say "computer science" or "software engineering" the person replies "oh I'm so sorry. Can you eat today? Here's a dollar." You can see that comments on those videos are students who just entered CS and are afraid (questioning whether they should switch majors.)

deleted 2y ago

[deleted]

massysett 2y ago

Author clearly has an axe to grind, I wonder if declining volume is really a problem. Question quantity may not relate to question quality. Maybe most good questions have already been asked and answered.

deleted 2y ago

[deleted]

plorkyeran 2y ago

The linked example of the author's "high quality" question that got closed as a duplicate is in fact a duplicate of the existing question, and doesn't seem like a very good question-as-documentation due to being significantly emotionally charged. The author seems to have been offended by not being lavished with praise for asking such a good question or something.

asdffdasy 2y ago

excellent! nobody wants answers that a LLM can answer on SO. to be honest, nobody wanted questions that were spelled out on all the manuals either. so maybe LLM will handle all of those and let only the interesting ones.

BrandoElFollito 2y ago

I have about 200k rep and a 15 years account. When I want to ask a question I need to bend backwards in order for the "community" not to downvote and tell me that I need X when I ask Y. Some communities (hello Golang!) are straight toxic telling that if you do not ask a IQ 150 question, get out. I do not think I have question with a positive vote there. In contrast, when asking on Travel or Cooking I get friendly, reasonable answers. I read the TeX community just because they are so nice (I do not even use TeX or LaTeX :)) I usually get an answer on Reddit, though the quality varies. I need to use ChatGPT more as it seems to be the short and mid-term future

tzs 2y ago
> When I want to ask a question I need to bend backwards in order for the "community" not to downvote and tell me that I need X when I ask Y. Worse, even if the "community" is right that you do really need X and so all the answers tell you how to do X, there will later be other people who really do need to do Y who are going to find your question.