5 ms·

> For instance, researchers found a 16% drop in activity on the coding website StackOverflow one year after the release of ChatGPT. This suggests AI assistance

> For instance, researchers found a 16% drop in activity on the coding website StackOverflow one year after the release of ChatGPT. This suggests AI assistance may already be reducing person-to-person interactions in some online communities. I don't get why this is a bad news. It meant that most likely these questions would have been duplicates and/or very easy, as ChatGPT could fix them. My issue with SO is on the opposite side of the spectrum. By the time I am ready to post a question after having done my homeworks, I know it is either an unfixable bug or out of reach for most if not all users, and the probability of getting an answer is very low. Most of the time I then answer my own questions a few months later.

clausecker 2y ago
> Most of the time I then answer my own questions a few months later. Thank you for doing that. This is very valuable.
lifeisstillgood 2y ago
Agreed - those who get the most out of SO seem to use it as a personal blogging tool - answering questions as a trigger for writing out what they know. I have always assumed that a good marketing ruse for FOSS would be to pay an intern to basically Q&A your entire docs into SO - if you do it right it would even build a better roadmap for your project.
creesch 2y ago
The article actually does explain this in the first few paragraphs. It is an issue because communities like StackOverflow are actually one of the sources of training material. LLM's are not capable of innovation, they can only do variations based on what is in their training material. People asking questions to other people about things does allow for actual new and fresh answer, where people might come up with solutions other people have not yet. And yes, you are right, some of these answers would be duplicates and easy to answer. But only to a certain point, there are a lot of answers on StackOverflow that are technically correct but severely outdated by this point. They have been superseeded by more modern ways of doing things or the APIs to do so have changed in newer versions of frameworks. With a human community you will have people leaving comments on older answers pointing this out and answering with the current answers on new questions. LLM's, again, are not capable of picking up this new information on their own. You can actually often see this in the answers you get from them, they are often ever so slightly dated already. When the communities that feed them die they will have less learning material so at the very least they will stagnate.
- ben_w 2y ago
  > LLM's are not capable of innovation, they can only do variations based on what is in their training material I've seen this claim a few times; what I have yet to see is a concrete definition of what constitutes "innovation" — the vagueness of what it means having been a problem I've seen even back when Dilbert occupied the cultural niche now filled by xkcd.
  - creesch 2y ago
    Innovation in the most.simplest way, let me give you a simple example. Give an LLM the documentation of a library/framework it has no training data on and ask it to implement a specific thing based on that documentation. It will struggle to come up with code that makes sense. Even if it does have training data on the language itself, similar frameworks, etc. Ask the same of a human who does have the same "experience" and they will be able to come up with something reasonable.
    - ben_w 2y ago
      That example doesn't match my experience of either LLMs (I've given them my own private projects and libraries and the best LLMs are fine) or human developers (who, myself included, say or think "WTF" a lot on new codebases), and also isn't a definition.
      - creesch 2y ago
        > (I've given them my own private projects and libraries and the best LLMs are fine) It all does depend on how much you have given them and how unique your libraries are. If you have given them a relatively simple library with a code base that uses all the API calls in that library in multiple ways, then yeah, an LLM will be able to work on that as you have given it plenty of examples to base answers on. Give them just jsdoc/javadoc documentation, and they will give you very little in the way of useful implementations. Where a human in your own words might "WTF" a bunch but will be able to produce something. In fact, the ability to say WTF might even be what is important here. As an LLM will not even stop to consider if things make sense. > and also isn't a definition. Sure it is, more specifically, it is part of a definition. You want it to be explicitly this or that, but things aren't that simple. Sure, we can put a dictionary definition on it and call it a day, but that isn't very helpful. When I talk about innovation in this context, I roughly mean the ability to think of solutions where no previous examples are directly available. Extending the definition of innovation, I am highly doubtful the current generation of LLMs can implement frameworks/libraries based on new principles unless very specifically prompted. In which case you have a human architect behind it all, and I still believe you will end up with code that follows paradigms and structures that are at most "current".
ajsnigrutin 2y ago
> . By the time I am ready to post a question after having done my homeworks, I know it is either an unfixable bug or out of reach for most if not all users, and the probability of getting an answer is very low. And even then it gets marked as duplicate of something unrelated.

clausecker 2y ago

> Most of the time I then answer my own questions a few months later. Thank you for doing that. This is very valuable.

lifeisstillgood 2y ago

Agreed - those who get the most out of SO seem to use it as a personal blogging tool - answering questions as a trigger for writing out what they know. I have always assumed that a good marketing ruse for FOSS would be to pay an intern to basically Q&A your entire docs into SO - if you do it right it would even build a better roadmap for your project.

creesch 2y ago

The article actually does explain this in the first few paragraphs. It is an issue because communities like StackOverflow are actually one of the sources of training material. LLM's are not capable of innovation, they can only do variations based on what is in their training material. People asking questions to other people about things does allow for actual new and fresh answer, where people might come up with solutions other people have not yet. And yes, you are right, some of these answers would be duplicates and easy to answer. But only to a certain point, there are a lot of answers on StackOverflow that are technically correct but severely outdated by this point. They have been superseeded by more modern ways of doing things or the APIs to do so have changed in newer versions of frameworks. With a human community you will have people leaving comments on older answers pointing this out and answering with the current answers on new questions. LLM's, again, are not capable of picking up this new information on their own. You can actually often see this in the answers you get from them, they are often ever so slightly dated already. When the communities that feed them die they will have less learning material so at the very least they will stagnate.

ben_w 2y ago
> LLM's are not capable of innovation, they can only do variations based on what is in their training material I've seen this claim a few times; what I have yet to see is a concrete definition of what constitutes "innovation" — the vagueness of what it means having been a problem I've seen even back when Dilbert occupied the cultural niche now filled by xkcd.
- creesch 2y ago
  Innovation in the most.simplest way, let me give you a simple example. Give an LLM the documentation of a library/framework it has no training data on and ask it to implement a specific thing based on that documentation. It will struggle to come up with code that makes sense. Even if it does have training data on the language itself, similar frameworks, etc. Ask the same of a human who does have the same "experience" and they will be able to come up with something reasonable.
  - ben_w 2y ago
    That example doesn't match my experience of either LLMs (I've given them my own private projects and libraries and the best LLMs are fine) or human developers (who, myself included, say or think "WTF" a lot on new codebases), and also isn't a definition.
    - creesch 2y ago
      > (I've given them my own private projects and libraries and the best LLMs are fine) It all does depend on how much you have given them and how unique your libraries are. If you have given them a relatively simple library with a code base that uses all the API calls in that library in multiple ways, then yeah, an LLM will be able to work on that as you have given it plenty of examples to base answers on. Give them just jsdoc/javadoc documentation, and they will give you very little in the way of useful implementations. Where a human in your own words might "WTF" a bunch but will be able to produce something. In fact, the ability to say WTF might even be what is important here. As an LLM will not even stop to consider if things make sense. > and also isn't a definition. Sure it is, more specifically, it is part of a definition. You want it to be explicitly this or that, but things aren't that simple. Sure, we can put a dictionary definition on it and call it a day, but that isn't very helpful. When I talk about innovation in this context, I roughly mean the ability to think of solutions where no previous examples are directly available. Extending the definition of innovation, I am highly doubtful the current generation of LLMs can implement frameworks/libraries based on new principles unless very specifically prompted. In which case you have a human architect behind it all, and I still believe you will end up with code that follows paradigms and structures that are at most "current".

ajsnigrutin 2y ago

> . By the time I am ready to post a question after having done my homeworks, I know it is either an unfixable bug or out of reach for most if not all users, and the probability of getting an answer is very low. And even then it gets marked as duplicate of something unrelated.