4 ms·

For now, yes, because the question you asked was likely answered by someone on Stack Overflow (or Reddit, or Github, or wherever else) already and then made its

by paxys 2y ago

For now, yes, because the question you asked was likely answered by someone on Stack Overflow (or Reddit, or Github, or wherever else) already and then made its way to the LLM's training data set. What happens when a brand new language or library or tool is released though and you run into a unique problem for the first time? When all human forums have been shut down, and AI still isn't intelligent enough to figure out the answer on its own?

rapatel0 2y ago
Counterarguement: This will be solved mostly by documentation. Historically, most of my SO usage boils down to: 1) finding how to implement something esoteric that results in finding a clever solution or a under described feature flag in a function/tool 2) finding a workaround bugfix for a broken feature in some software (>70% of the time finding link to a github issue in the description If we consider that LLMs are functionally an information retrieval function containing natural language program subroutines. In this context, a web-browser enabled LLM should be able to determine go to source and return a functional answer on a model that is not pretrained on the source. So as long as there is good documentation on a particular piece of software, we should theoretically be able to generalize to non-existing tools. At least long enough for there to be a newly-created training dataset from people hitting the problem for the first time. Side note: In some sense, the foundation model labs are aggregating the Question-Answer pairs (typically from stackoverflow) from their user data. I wouldn't be surprised if they created a stackoverflow clone at some point to opensource the dataset creation and labeling efforts. This is basically what community notes is for X and now Facebook
- johnkizer 2y ago
  Counter-counterargument: "So as long as there is good documentation" feels a bit like relying for success on the least important deliverable to people funding a project, and least interesting process step to people building it, going really well.
sbuttgereit 2y ago
I think there's a difference between Stack Overflow and/or Reddit vs. specific community led forums, or even GitHub, where questions also get answered. Just considering Stack Overflow for a moment, they exist to profit from their product of consolidating questions/answers. When the LLM can answer most questions more efficiently, they've lost much of their value proposition in terms of product... and perhaps their business along with it. Many of the community forums, however, tend to not be businesses per se. Sure they'll see less traffic, but that might not matter to them. In fact, it might even be better to an extent because they often aren't monetizing their services and so LLMs carrying some weight can help reduce costs. Under those circumstances, LLMs may not be nearly so bad and they, themselves, will still have sources of training data. For example, I read the Elixir Forums for language announcements, feature discussions, occasionally to ask questions that I can't resolve with research, and even to answer some questions. I've also got LLMs fairly well integrated into my workflow. I don't see that pattern changing: neither less Elixir Forum or less reliance on the LLM. What has changed is I don't use search as much as I use to, nor do I use Stack Overflow as much. So I do expect the big aggregators to go away, those not tied to monetizing their knowledge transfer I expect to see less overall traffic, but not less meaningful and substantive interactions.

by paxys 2y ago

rapatel0 2y ago

Counterarguement: This will be solved mostly by documentation. Historically, most of my SO usage boils down to: 1) finding how to implement something esoteric that results in finding a clever solution or a under described feature flag in a function/tool 2) finding a workaround bugfix for a broken feature in some software (>70% of the time finding link to a github issue in the description If we consider that LLMs are functionally an information retrieval function containing natural language program subroutines. In this context, a web-browser enabled LLM should be able to determine go to source and return a functional answer on a model that is not pretrained on the source. So as long as there is good documentation on a particular piece of software, we should theoretically be able to generalize to non-existing tools. At least long enough for there to be a newly-created training dataset from people hitting the problem for the first time. Side note: In some sense, the foundation model labs are aggregating the Question-Answer pairs (typically from stackoverflow) from their user data. I wouldn't be surprised if they created a stackoverflow clone at some point to opensource the dataset creation and labeling efforts. This is basically what community notes is for X and now Facebook

johnkizer 2y ago
Counter-counterargument: "So as long as there is good documentation" feels a bit like relying for success on the least important deliverable to people funding a project, and least interesting process step to people building it, going really well.

sbuttgereit 2y ago

I think there's a difference between Stack Overflow and/or Reddit vs. specific community led forums, or even GitHub, where questions also get answered. Just considering Stack Overflow for a moment, they exist to profit from their product of consolidating questions/answers. When the LLM can answer most questions more efficiently, they've lost much of their value proposition in terms of product... and perhaps their business along with it. Many of the community forums, however, tend to not be businesses per se. Sure they'll see less traffic, but that might not matter to them. In fact, it might even be better to an extent because they often aren't monetizing their services and so LLMs carrying some weight can help reduce costs. Under those circumstances, LLMs may not be nearly so bad and they, themselves, will still have sources of training data. For example, I read the Elixir Forums for language announcements, feature discussions, occasionally to ask questions that I can't resolve with research, and even to answer some questions. I've also got LLMs fairly well integrated into my workflow. I don't see that pattern changing: neither less Elixir Forum or less reliance on the LLM. What has changed is I don't use search as much as I use to, nor do I use Stack Overflow as much. So I do expect the big aggregators to go away, those not tied to monetizing their knowledge transfer I expect to see less overall traffic, but not less meaningful and substantive interactions.