4 ms·

All innovation is composition plus random generation. Which LLMs already do. LLMs have a rudimentary form of innovation. It's not quite as good as humans but i

All innovation is composition plus random generation. Which LLMs already do. LLMs have a rudimentary form of innovation. It's not quite as good as humans but it's getting there.

sidlls 3y ago
All innovation is composition plus random generation. That’s a fairly bold claim. What supporting data do you have to justify it?
- corethree 3y ago
  There are two types of statements that can be made in this world. One is data driven based on evidence. The other is logic driven based on axioms and the logical implications of said axioms. My statement is derived from the later. Therefore evidence is unnecessary. It is niave to blindly faith in all truth in the hands of data without understanding nuances between the relationship of data and logic. If you have a pure function that takes the input, the output of that function has 3 possible outcomes. 1. The output of that function is some transformation of the input. 2. The output of the function has nothing to do with the input and is thus generated randomly generated. 3. The output of the function is a combination of both random generation and input transformation. In this case your brain is the function. Input parameters are 100 percent of your existing knowledge. That includes genetic knowledge such as instinctual/rational processing behaviors evolved into the structure of your brain through evolution and learned knowledge such as what you gain when you read a book. The result of the "innovation" operation performed by the brain includes novel output by definition. Thus by logic it must be must include components that are from a certain perspective randomly generated. I guess at first glance it doesn't appear randomly generated because we vet the output and verify it and iterate over several pieces of randomly generated information. Additionally it's not completely random as we only try and test ideas within the realm of possibility. For example I'm not going to account for the possibility that my car will transform into a rock tomorrow that's just too random. But make no mistake. Innovation must be partly randomly generated. Even the idea itself of composing two existing components of knowledge together is itself randomly generated. That being said if we want to be pedantic, there's no real known way to randomly generate stuff via an algorithm. So I use the term "randomly generate" very loosely. Think of it in a similar way to random number generation on your computer: Random from a practical perspective but not technically random.
  - gamacodre 3y ago
    Couple of thoughts: You are assuming that a pure function has access to some source of randomness. I think that's a bit less than pure unless a source of randomness is one of the inputs. Doesn't change your point at all, but it distracted me from what you were saying. Speaking as someone who has occasionally invented new things, I think you underestimate the variety of outputs that can arise from a lifetime of accumulated experiential cruft and a situational fitting function (e.g. "we need a working teleporter to stay competitive"), without bringing in randomness at all.
    - corethree 3y ago
      Right and in this case you describe you'd be composing ideas to form new ideas right? And you'd do it within the parameters of the situation. So you would only select ideas that are in the bounds of "teleportation". But then out of that set of ideas how do you choose which ideas to compose to formulate the new idea? So you have idea A and idea B. And you randomly formulate a new compositional rule as an idea: A + B. Because A + B didn't exist as an idea before, it was randomly generated by you. >I think you underestimate the variety of outputs that can arise from a lifetime of accumulated experiential cruft and a situational fitting function Well it's actually possible to mathematically calculate the total amount of compositions If I know the total amount of experimental cruft you have accumulated in your lifetime. It's a combinatorics problem and the output of that is, you're right, extremely huge. We have fitting functions that reduce it, of course, but within this fitting function we're just iterating through all the remaining possibilities or aka "randomly generating" the new idea.
  - hackerlight 3y ago
    If we want to understand chemistry we should study chemistry at the level of chemistry, not study lower-level physics simply because chemistry is in theory explained by physics. We need to pick the right level of abstraction. Creativity will be explainable in terms of composition and randomness in the same way that chemistry is explainable in terms of physics. Yes, these are the building blocks, but the building blocks by itself don't help you much because the complicated phenomena still hasn't been explained. All of the magic is in how the building blocks come together to produce the output.
AlexandrB 3y ago
Even if one accepts your premise, the key question is: composition of what? I think people underestimate the volume and variety of "training data" that humans are exposed to over a lifetime. It's not just text and images - it's feelings (pain, cold, heat), emotions, sounds, smells, and other experiences that originate within the human body as well. Human innovation can arise by using these experiences as source data for composition of text or images. LLMs, by contrast, are limited to training on text and images/video exclusively.
- corethree 3y ago
  >Human innovation can arise by using these experiences as source data for composition of text or images. LLMs, by contrast, are limited to training on text and images/video exclusively. Right but my concept still applies. What's unique about the LLM is that it is fed a massive amount of textual data in such a way that it can essentially output text AS if it were a human that does experience those emotions. From the perspective of you and me, this is no different then what we experience in real life. How can you be sure the people around you feel emotions just as you do? Do you just assume it? Why shouldn't you assume it for an LLM? The human like the LLM uses English to describe and communicate to you. This in terms of raw logical evidence we can't confirm if LLMs feel emotions any more than our ability to confirm whether other humans can feel emotions. The technical evidence for both is relatively identical.
Jensson 3y ago
Innovation isn't just testing something new, it is a new thing that improves something or is valuable in some way. So doing random combinations of things isn't innovation, it is just noise.
- corethree 3y ago
  Then where did the new idea come from? What algorithm allows for the generation of completely novel things without knowledge of the output encoded in the algorithm itself? It must be "randomly" generated by logic. Again I like to emphasize that it's not "truly random" we can't even define randomness formally with a function so this doesn't exist. It's more of a combinatorics algorithm where we iterate through every possible and likely permutation.

All innovation is composition plus random generation. Which LLMs already do. LLMs have a rudimentary form of innovation. It's not quite as good as humans but it's getting there.

sidlls 3y ago

All innovation is composition plus random generation. That’s a fairly bold claim. What supporting data do you have to justify it?

corethree 3y ago
There are two types of statements that can be made in this world. One is data driven based on evidence. The other is logic driven based on axioms and the logical implications of said axioms. My statement is derived from the later. Therefore evidence is unnecessary. It is niave to blindly faith in all truth in the hands of data without understanding nuances between the relationship of data and logic. If you have a pure function that takes the input, the output of that function has 3 possible outcomes. 1. The output of that function is some transformation of the input. 2. The output of the function has nothing to do with the input and is thus generated randomly generated. 3. The output of the function is a combination of both random generation and input transformation. In this case your brain is the function. Input parameters are 100 percent of your existing knowledge. That includes genetic knowledge such as instinctual/rational processing behaviors evolved into the structure of your brain through evolution and learned knowledge such as what you gain when you read a book. The result of the "innovation" operation performed by the brain includes novel output by definition. Thus by logic it must be must include components that are from a certain perspective randomly generated. I guess at first glance it doesn't appear randomly generated because we vet the output and verify it and iterate over several pieces of randomly generated information. Additionally it's not completely random as we only try and test ideas within the realm of possibility. For example I'm not going to account for the possibility that my car will transform into a rock tomorrow that's just too random. But make no mistake. Innovation must be partly randomly generated. Even the idea itself of composing two existing components of knowledge together is itself randomly generated. That being said if we want to be pedantic, there's no real known way to randomly generate stuff via an algorithm. So I use the term "randomly generate" very loosely. Think of it in a similar way to random number generation on your computer: Random from a practical perspective but not technically random.
- gamacodre 3y ago
  Couple of thoughts: You are assuming that a pure function has access to some source of randomness. I think that's a bit less than pure unless a source of randomness is one of the inputs. Doesn't change your point at all, but it distracted me from what you were saying. Speaking as someone who has occasionally invented new things, I think you underestimate the variety of outputs that can arise from a lifetime of accumulated experiential cruft and a situational fitting function (e.g. "we need a working teleporter to stay competitive"), without bringing in randomness at all.
  - corethree 3y ago
    Right and in this case you describe you'd be composing ideas to form new ideas right? And you'd do it within the parameters of the situation. So you would only select ideas that are in the bounds of "teleportation". But then out of that set of ideas how do you choose which ideas to compose to formulate the new idea? So you have idea A and idea B. And you randomly formulate a new compositional rule as an idea: A + B. Because A + B didn't exist as an idea before, it was randomly generated by you. >I think you underestimate the variety of outputs that can arise from a lifetime of accumulated experiential cruft and a situational fitting function Well it's actually possible to mathematically calculate the total amount of compositions If I know the total amount of experimental cruft you have accumulated in your lifetime. It's a combinatorics problem and the output of that is, you're right, extremely huge. We have fitting functions that reduce it, of course, but within this fitting function we're just iterating through all the remaining possibilities or aka "randomly generating" the new idea.
- hackerlight 3y ago
  If we want to understand chemistry we should study chemistry at the level of chemistry, not study lower-level physics simply because chemistry is in theory explained by physics. We need to pick the right level of abstraction. Creativity will be explainable in terms of composition and randomness in the same way that chemistry is explainable in terms of physics. Yes, these are the building blocks, but the building blocks by itself don't help you much because the complicated phenomena still hasn't been explained. All of the magic is in how the building blocks come together to produce the output.

AlexandrB 3y ago

Even if one accepts your premise, the key question is: composition of what? I think people underestimate the volume and variety of "training data" that humans are exposed to over a lifetime. It's not just text and images - it's feelings (pain, cold, heat), emotions, sounds, smells, and other experiences that originate within the human body as well. Human innovation can arise by using these experiences as source data for composition of text or images. LLMs, by contrast, are limited to training on text and images/video exclusively.

corethree 3y ago
>Human innovation can arise by using these experiences as source data for composition of text or images. LLMs, by contrast, are limited to training on text and images/video exclusively. Right but my concept still applies. What's unique about the LLM is that it is fed a massive amount of textual data in such a way that it can essentially output text AS if it were a human that does experience those emotions. From the perspective of you and me, this is no different then what we experience in real life. How can you be sure the people around you feel emotions just as you do? Do you just assume it? Why shouldn't you assume it for an LLM? The human like the LLM uses English to describe and communicate to you. This in terms of raw logical evidence we can't confirm if LLMs feel emotions any more than our ability to confirm whether other humans can feel emotions. The technical evidence for both is relatively identical.

Jensson 3y ago

Innovation isn't just testing something new, it is a new thing that improves something or is valuable in some way. So doing random combinations of things isn't innovation, it is just noise.

corethree 3y ago
Then where did the new idea come from? What algorithm allows for the generation of completely novel things without knowledge of the output encoded in the algorithm itself? It must be "randomly" generated by logic. Again I like to emphasize that it's not "truly random" we can't even define randomness formally with a function so this doesn't exist. It's more of a combinatorics algorithm where we iterate through every possible and likely permutation.