3 ms·

The article actually does explain this in the first few paragraphs. It is an issue because communities like StackOverflow are actually one of the sources of tr

The article actually does explain this in the first few paragraphs. It is an issue because communities like StackOverflow are actually one of the sources of training material. LLM's are not capable of innovation, they can only do variations based on what is in their training material. People asking questions to other people about things does allow for actual new and fresh answer, where people might come up with solutions other people have not yet. And yes, you are right, some of these answers would be duplicates and easy to answer. But only to a certain point, there are a lot of answers on StackOverflow that are technically correct but severely outdated by this point. They have been superseeded by more modern ways of doing things or the APIs to do so have changed in newer versions of frameworks. With a human community you will have people leaving comments on older answers pointing this out and answering with the current answers on new questions. LLM's, again, are not capable of picking up this new information on their own. You can actually often see this in the answers you get from them, they are often ever so slightly dated already. When the communities that feed them die they will have less learning material so at the very least they will stagnate.

ben_w 2y ago
> LLM's are not capable of innovation, they can only do variations based on what is in their training material I've seen this claim a few times; what I have yet to see is a concrete definition of what constitutes "innovation" — the vagueness of what it means having been a problem I've seen even back when Dilbert occupied the cultural niche now filled by xkcd.
- creesch 2y ago
  Innovation in the most.simplest way, let me give you a simple example. Give an LLM the documentation of a library/framework it has no training data on and ask it to implement a specific thing based on that documentation. It will struggle to come up with code that makes sense. Even if it does have training data on the language itself, similar frameworks, etc. Ask the same of a human who does have the same "experience" and they will be able to come up with something reasonable.
  - ben_w 2y ago
    That example doesn't match my experience of either LLMs (I've given them my own private projects and libraries and the best LLMs are fine) or human developers (who, myself included, say or think "WTF" a lot on new codebases), and also isn't a definition.
    - creesch 2y ago
      > (I've given them my own private projects and libraries and the best LLMs are fine) It all does depend on how much you have given them and how unique your libraries are. If you have given them a relatively simple library with a code base that uses all the API calls in that library in multiple ways, then yeah, an LLM will be able to work on that as you have given it plenty of examples to base answers on. Give them just jsdoc/javadoc documentation, and they will give you very little in the way of useful implementations. Where a human in your own words might "WTF" a bunch but will be able to produce something. In fact, the ability to say WTF might even be what is important here. As an LLM will not even stop to consider if things make sense. > and also isn't a definition. Sure it is, more specifically, it is part of a definition. You want it to be explicitly this or that, but things aren't that simple. Sure, we can put a dictionary definition on it and call it a day, but that isn't very helpful. When I talk about innovation in this context, I roughly mean the ability to think of solutions where no previous examples are directly available. Extending the definition of innovation, I am highly doubtful the current generation of LLMs can implement frameworks/libraries based on new principles unless very specifically prompted. In which case you have a human architect behind it all, and I still believe you will end up with code that follows paradigms and structures that are at most "current".
      - ben_w 2y ago
        > Sure it is, more specifically, it is part of a definition. You want it to be explicitly this or that, but things aren't that simple. Sure, we can put a dictionary definition on it and call it a day, but that isn't very helpful. When I talk about innovation in this context, I roughly mean the ability to think of solutions where no previous examples are directly available. A letter can part of some alphabet, but it is not itself an alphabet. A word can be part of some dictionary, but it is not itself a dictionary. An anecdote can be part of a dataset, but it is not itself a dataset. I think the definition you now give seems to be reasonable, but I'd like to compare that with the example to illustrate the problem of vagueness here: > Give an LLM the documentation of a library/framework it has no training data on and ask it to implement a specific thing based on that documentation. vs. > When I talk about innovation in this context, I roughly mean the ability to think of solutions where no previous examples are directly available. The example you give is the kind of thing that LLMs are really good at: translating one thing into another thing. The original use case of the transformer model was natural language translation, and they can do that well even when there's no explicit map from the input language to the output language, as they learn an intermediate representation for all. The same applies when the languages are jargon and synthetic: "business plan", "code" and "documentation". Is that innovation? It meets the example you gave, but does it meet the definition you gave? It depends exactly what you mean by "no *previous* examples": myself, I would say that "documentation" counts as "a previous example". But the more I think about it, the less that actually feels like innovation. It's in-context learning, which is neat, but it doesn't seem to meet what I'd expect from real innovation — at most it is the corporate (and government) self-congratulation that gets called by the same word. > Extending the definition of innovation, I am highly doubtful the current generation of LLMs can implement frameworks/libraries based on new principles unless very specifically prompted. In which case you have a human architect behind it all, and I still believe you will end up with code that follows paradigms and structures that are at most "current". This is a confusing point: that sounds like it's about following a new principle, whereas I would expect "innovation" to involve designing and specifying a new principle, determining its weaknesses through experiment. I believe LLMs are used as part of larger systems to do this, though I have no frame of reference for the quality, and I would accept that "LLM as component of bigger system" is different from "LLM" — a letter can part of some alphabet, but it is not itself an alphabet, etc.
isaacfung 2y ago
When people say "LLMs are not capable of innovation", what exactly do they consider as innovation? If LLMs are not capable of innovation on their own, what if we augment them with means to interact with the environment so they can obtain new training data? e.g. The minecraft bot Voyager can explore the game environment and extend its skill library(stored as a vector database), is that considered as innovation? There are also systems like leandojo/alphaproof that discover new proofs and use LLM in non-trivial ways(not just naively predict the next token in one shot). Reinforcement learning algorithms like AlphaGo/AlphaZero use self play and use monte carlo tree search to learn to outperform humans. You can similarly use LLMs to generate actions and estimate state values(check the language agent tree search paper). Most people use LLMs by prompting them with some additional context(chat history and data retrived from database) but there is nothing that stops us from continuously improving a LLM(either by modifying its weight or augmenting it with external database) by asking it to evaluate the task outcome/error message and feeding it back to the LLM. We can also ask it to just keep on generating new tasks to experiment with the environment/internet to get new knowledge.
- creesch 2y ago
  > what if we augment them with means to interact with the environment so they can obtain new training data? So far, the current generation of LLMs that are in widespread use do not have that ability as far as I am aware. To actually do it to a degree that would rival human learning, they would need access to a lot more environment than you might be thinking of. Sure, for programming, the most basic environment would be the platform to run the code on and the output of the code. But choices in programming are made based on more things like performance, load impact, behavior in production environments, interaction with other applications, platform logging, adjacent application logging. Or even before that, using previous experience to judge specifications for an application which takes things in account like the expected user base, costs, etc. The real world is a lot more complex than a minecraft world or a game of Go. Which is to say, I am not saying that it is impossible. I am sure research is ongoing to do exactly that. But the LLMs that are currently already disrupting communities like StackOverflow are not doing any of that. Given how complex the task is to plug in all relevant stimuli and the fact that for now they can get by without doing all of this I think things are more likely to get worse before they potentially will get better.

ben_w 2y ago

> LLM's are not capable of innovation, they can only do variations based on what is in their training material I've seen this claim a few times; what I have yet to see is a concrete definition of what constitutes "innovation" — the vagueness of what it means having been a problem I've seen even back when Dilbert occupied the cultural niche now filled by xkcd.

creesch 2y ago
Innovation in the most.simplest way, let me give you a simple example. Give an LLM the documentation of a library/framework it has no training data on and ask it to implement a specific thing based on that documentation. It will struggle to come up with code that makes sense. Even if it does have training data on the language itself, similar frameworks, etc. Ask the same of a human who does have the same "experience" and they will be able to come up with something reasonable.
- ben_w 2y ago
  That example doesn't match my experience of either LLMs (I've given them my own private projects and libraries and the best LLMs are fine) or human developers (who, myself included, say or think "WTF" a lot on new codebases), and also isn't a definition.
  - creesch 2y ago
    > (I've given them my own private projects and libraries and the best LLMs are fine) It all does depend on how much you have given them and how unique your libraries are. If you have given them a relatively simple library with a code base that uses all the API calls in that library in multiple ways, then yeah, an LLM will be able to work on that as you have given it plenty of examples to base answers on. Give them just jsdoc/javadoc documentation, and they will give you very little in the way of useful implementations. Where a human in your own words might "WTF" a bunch but will be able to produce something. In fact, the ability to say WTF might even be what is important here. As an LLM will not even stop to consider if things make sense. > and also isn't a definition. Sure it is, more specifically, it is part of a definition. You want it to be explicitly this or that, but things aren't that simple. Sure, we can put a dictionary definition on it and call it a day, but that isn't very helpful. When I talk about innovation in this context, I roughly mean the ability to think of solutions where no previous examples are directly available. Extending the definition of innovation, I am highly doubtful the current generation of LLMs can implement frameworks/libraries based on new principles unless very specifically prompted. In which case you have a human architect behind it all, and I still believe you will end up with code that follows paradigms and structures that are at most "current".
    - ben_w 2y ago
      > Sure it is, more specifically, it is part of a definition. You want it to be explicitly this or that, but things aren't that simple. Sure, we can put a dictionary definition on it and call it a day, but that isn't very helpful. When I talk about innovation in this context, I roughly mean the ability to think of solutions where no previous examples are directly available. A letter can part of some alphabet, but it is not itself an alphabet. A word can be part of some dictionary, but it is not itself a dictionary. An anecdote can be part of a dataset, but it is not itself a dataset. I think the definition you now give seems to be reasonable, but I'd like to compare that with the example to illustrate the problem of vagueness here: > Give an LLM the documentation of a library/framework it has no training data on and ask it to implement a specific thing based on that documentation. vs. > When I talk about innovation in this context, I roughly mean the ability to think of solutions where no previous examples are directly available. The example you give is the kind of thing that LLMs are really good at: translating one thing into another thing. The original use case of the transformer model was natural language translation, and they can do that well even when there's no explicit map from the input language to the output language, as they learn an intermediate representation for all. The same applies when the languages are jargon and synthetic: "business plan", "code" and "documentation". Is that innovation? It meets the example you gave, but does it meet the definition you gave? It depends exactly what you mean by "no *previous* examples": myself, I would say that "documentation" counts as "a previous example". But the more I think about it, the less that actually feels like innovation. It's in-context learning, which is neat, but it doesn't seem to meet what I'd expect from real innovation — at most it is the corporate (and government) self-congratulation that gets called by the same word. > Extending the definition of innovation, I am highly doubtful the current generation of LLMs can implement frameworks/libraries based on new principles unless very specifically prompted. In which case you have a human architect behind it all, and I still believe you will end up with code that follows paradigms and structures that are at most "current". This is a confusing point: that sounds like it's about following a new principle, whereas I would expect "innovation" to involve designing and specifying a new principle, determining its weaknesses through experiment. I believe LLMs are used as part of larger systems to do this, though I have no frame of reference for the quality, and I would accept that "LLM as component of bigger system" is different from "LLM" — a letter can part of some alphabet, but it is not itself an alphabet, etc.

isaacfung 2y ago

When people say "LLMs are not capable of innovation", what exactly do they consider as innovation? If LLMs are not capable of innovation on their own, what if we augment them with means to interact with the environment so they can obtain new training data? e.g. The minecraft bot Voyager can explore the game environment and extend its skill library(stored as a vector database), is that considered as innovation? There are also systems like leandojo/alphaproof that discover new proofs and use LLM in non-trivial ways(not just naively predict the next token in one shot). Reinforcement learning algorithms like AlphaGo/AlphaZero use self play and use monte carlo tree search to learn to outperform humans. You can similarly use LLMs to generate actions and estimate state values(check the language agent tree search paper). Most people use LLMs by prompting them with some additional context(chat history and data retrived from database) but there is nothing that stops us from continuously improving a LLM(either by modifying its weight or augmenting it with external database) by asking it to evaluate the task outcome/error message and feeding it back to the LLM. We can also ask it to just keep on generating new tasks to experiment with the environment/internet to get new knowledge.

creesch 2y ago
> what if we augment them with means to interact with the environment so they can obtain new training data? So far, the current generation of LLMs that are in widespread use do not have that ability as far as I am aware. To actually do it to a degree that would rival human learning, they would need access to a lot more environment than you might be thinking of. Sure, for programming, the most basic environment would be the platform to run the code on and the output of the code. But choices in programming are made based on more things like performance, load impact, behavior in production environments, interaction with other applications, platform logging, adjacent application logging. Or even before that, using previous experience to judge specifications for an application which takes things in account like the expected user base, costs, etc. The real world is a lot more complex than a minecraft world or a game of Go. Which is to say, I am not saying that it is impossible. I am sure research is ongoing to do exactly that. But the LLMs that are currently already disrupting communities like StackOverflow are not doing any of that. Given how complex the task is to plug in all relevant stimuli and the fact that for now they can get by without doing all of this I think things are more likely to get worse before they potentially will get better.