5 ms·

They used stack overflow to make their case, and report that user engagement has gone down after the release of ChatGPT. Could it not be the case that SO is les

• TeMPOraL 3y ago
  > We observe no change in the votes posts receive on Stack Overflow since the release of ChatGPT. This finding suggests that ChatGPT is displacing a wide variety of Stack Overflow posts, including high-quality content. My alternative hypothesis: ChatGPT is eliminating all the long-tail, garbage questions, so the vote patterns don't change because the missing questions weren't getting much of them anyway.
• smeagull 3y ago
  I'd expect user engagement to be going down in the same trend that has been occurring over the past five years as Stackoverflow is less and less useful.
  • ArthurAardvark 3y ago
    As someone who is getting their start with coding, where/what forums exist that have a high quality/helpful community? My biggest struggle has been with relatively simple questions – with a broad stroked theme/issue to 'em for the most part. AKA having a mentor or just a group of coders who are willing to help out if you are willing to be an active member (but I'm relatively useless, aka active maybe in an off-topic lounge part of it). I'd appreciate it if ya got the sauce, a PM if you'd like to keep it on the DL perhaps? Thanks!
    • extasia 3y ago
      Programming language discords might be good. Code review stack exchange has been pretty helpful, though maybe wait until you have a grasp on the basics before posting. If you struggle with broad issues a HN post is probably a good place to start, assuming you can express your question well. I imagine you'll get the highest signal to noise ratio here. Edit: btw nobody can pm you as you have no contact info in your bio afaict.
  • ryan_j_naughton 3y ago
    They used a difference-in-differences[1] to control for such longitudinal/temporal effects. Since ChatGPT isn't readily available in China or Russia, but StackOverflow is, they can compare how SO changed pre and post ChatGPT in countries where it is widely available and compare that to the pre and post in countries where it isn't available (basically as the control). [1] https://en.wikipedia.org/wiki/Difference_in_differences https://en.wikipedia.org/wiki/Difference_in_differences
• yieldcrv 3y ago
  the hypothesis is wrong to begin with they should ask people that ask questions, not people passively looking for answers the experience asking a question on stackexchange sites is horrible! each tag is its own community with edicts and customs you have no idea about which derail your path to actually getting an answer, the auto-moderation system is completely broken because it thinks your question should be replaced with one from 2013, or the human moderators unilaterally fix your question which then causes the auto-moderator to think its similar to the one from 2013 and unceremoniously replaces it With chatgpt you dont need to prove anything about what you tried, you dont need a reputation score to do anything, you dont need to be a steward of every upvote for all eternity, you dont need to engage in meta discussion to get your post out of deletion, and it answers faster. You don't need to be told to ask a separate question if you have any followups, with all of the same gamble of problems, while chatgpt already assumes what you will have a followup question about and just tells you all the gotchas. Because its read all the forum posts and has seen what the recurring issues are. its better, faster, and cheaper time wise its not a direct comparison, chatgpt is more like a pair programmer. If stackoverflow had an adhoc pair programming live session you could hop into it would be a better comparison.
  • version_five 3y ago
    This is a great point. I think the research is "biased" to begin with, in that it feels like you could easily p-hack your way into a "chat gpt has degraded X" kind of paper if that's what you want to show. But with SO in particular, you're right that there are actually clear improvements that it makes as a competitor. I can't think of a way, buy it would be interesting to see if there is a similar effect on Wikipedia contributions.
• zer8k 3y ago
  > They used stack overflow to make their case, and report that user engagement has gone down after the release of ChatGPT. Could it not be the case that SO is less adept at finding related/duplicate questions than ChatGPT? Given the later's facility with the language, I would expect it to be. So I look at the paper to see if they accounted for that, and find this. The moderation team and community in general on Stack Overflow is so toxic I'm not even sure you could control for that effect well enough to arrive at this conclusion. I would argue people are leaving because it's easier to ask ChatGPT your question than be flamed and banned for asking how to do something. A half right answer from ChatGPT is better than getting marked duplicate and closed because the moron moderation team can't detect nuance.
  • dotancohen 3y ago
    > the moron moderation team can't detect nuance. It is unwise to leave the detection of nuance to volunteer moderators. Rather, highlight the nuance in your question (or answer). That strategy had always landed me high value answers to my questions.
  • zogrodea 3y ago
    I have anecdotal stories about aa friend corroborating this. He had been rebuked and experienced an unwelcoming response on Stack Overflow in response to his questions while trying to learn web development, and found ChatGPT a great teacher in comparison.
    • zer8k 3y ago
      I'm a very experienced developer. Sometimes I have questions that are easier to ask a person than go dig through miles of documentation. I could easily frame a question correctly to get a response but even I feel extremely unwelcome there. I couldn't imagine being junior.
  • JuanPosadas 3y ago
    I've arrived from google on "closed" questions I needed answers for, XY questions where I had question X but not secret question Y, people trying to XY answer extremely simple and straightforward X-and-only-X questions, questions unanswered for years with several upvotes and me-toos. But shallow questions where the official docs are too raw or are missing a few specifics? Stack Overflow was good for that before chatgpt.
• bradgessler 3y ago
  What happens after LLMs kill off SO and then seek more updated training data? It seems like these “deaths” are either temporary or something else will pop-up that continuously improves and trains an LLMs with proprietary data. There’s def a shift in the social contract of the internet. We’re shifting from “publish a few things you know a lot bit in exchange to read stuff other smart people have shared” to “solve a problem with an LLM and train it in a manner that can be helpful to others” What’s unclear is how the middle of that will be monetized. Search engines figured out how to do it for the “publish and share” paradigm and made a boatload of money off of it after paying for the massive infrastructure required to index everything. How will LLMs do it without killing the data that trains it?
  • FeepingCreature 3y ago
    Would be cool to have projects write documentation in a form that's annotated for LLMs, ie. premarked topic breaks.
    • justinclift 3y ago
      > have projects write documentation ... I kind of expect those projects would want ChatGPT to write the documentation. ;)
    • spacebanana7 3y ago
      I think open source might benefit a lot from being so easy for LLMs to provide answers for. Hopefully they’ll be able to reason straight from the source code.
      • FeepingCreature 3y ago
        I've had GPT3 give me the wrong answers to questions. Then I google the topic, find the documentation and go "hm, yeah, I can see how it would think that." For instance, ask it what `set +o` does in Bash, then check the online docs.
  • cwkoss 3y ago
• friendzis 3y ago
  To expect change in vote counts and/or ratios, you have to assume that users have sort of "vote budget" that they are going to spend one way or another. If voting is mostly independent variable and mostly depends on particular user and particular post then you should not expect voting patterns/counts to change meaningfully. > Could it not be the case that SO is less adept at finding related/duplicate questions than ChatGPT? Given the later's facility with the language, I would expect it to be. Given the later's facility with the language, I would expect it to be a better search engine. I would expect that ChatGPT is replacing Google as sort of tokenizer. I.e. search pipeline changes from "form natural question -> input to google -> go to SO -> if first few links do not yield answer post new question" to "form natural question -> input to chatgpt -> extract keyword tokens -> input to google -> go to SO". There is an important bit in the article: > > Using data on programming language popularity on GitHub, we find that the most widely used languages tend to have larger relative declines in posting activity. Here we can form a hypothesis that reduction in post frequency comes from entry level posts with posters not knowing what to search for. Under this hypothesis ChatGPT has strongest effect on users using SO as knowledge base rather than Q&A forum. This user type distinction would affect posting frequency much more heavily than voting patterns.
• jerjerjer 3y ago
  Some of my colleagues are using ChatGPT and blindly follow its advice. ChatGPT's answers often look more persuasive than human answers on SO. Still, I find ChatGPT answers much harder to validate. In programming Q&A, the answer is usually a series of calls. When you try to apply the human solution (assuming it's at least somewhat correct), the error often lies in input data mismatch or some minor changes in requirements when the answer is not perfectly aligned. Rarely the calls themselves are wrong - they might be deprecated if it's an old answer, but generally, they do at least exist. With ChatGPT, you never know which particular part of the answer has been hallucinated. Also, as SO has more stringing question requirements, you are forced to construct a minimal case to reproduce the error. Often, composing an SO question got me straight to the cause of the error.
• devjab 3y ago
  In 2013 or maybe it was 2014 I was tasked with building a couple of things for our Sharepoint 2010 that ran our intranet. I’m a programmer but it was pretty far out of my area of expertise and it was hard to find anything documentation related, probably because Sharepoint 2013 had been released. Anyway, I made things work by doing a lot of google programming, of which a lot led me to SO. Most of it was build with JavaScript but I still needed to figure out how to work with Sharepoint APIs and Lists. Fast forward to 2023 and I’m tasked with working with our current Sharepoint. It’s still far out of my comfort zone, but the Microsoft documentation is on point and it turns out to be a fairly trivial task to engage with the APIs. This story isn’t really unique to Sharepoint, and while obviously very anecdotal, these days it’s very rare that your average blog spam or SO answers are more helpful to me than official documentation, and often the official GitHub code examples. Now I’m more experienced today than I was in 2013 so that helps as well, but for the most part, I think the key reduction for my personal google programming has been how much better we as software developers have become at documenting our tools. So in 2023 I almost never visit SO, not because of GPT but because the people who write frameworks also supply you with documentation and implementation examples. Again anecdotal, but I do use GPT quite a bit. Though rarely to help me figure something out. It writes my documentation, unless the code is very secret. It sometimes writes some basic code like auto-generating a docker file or basic API classes. Things that aren’t in competition with SO. I’m sure it’ll take over the role of SO for some parts, but to me personally, it seems far more likely that SO’s decline is linked to a range of other challenges as well.