5 ms·

One can casually observe that LLMs quite excel at composition though: gluing together pieces of knowledge in ways no one did before (examples: a program that do

by vemv 3y ago

One can casually observe that LLMs quite excel at composition though: gluing together pieces of knowledge in ways no one did before (examples: a program that does X using language Y, a painting that mashes up two themes). Most knowledge workers' activities aren't innovative or imitational - similarly, we compose stuff, so LLMs are a fair competitor.

troelsSteegin 3y ago
... and our judgments of composition favor recall over precision. Which is to say that as human judges we're good at finding sense in something generated if it's plausible. We'll take a proposition (eg in a composition) and construct a justification for it. "I can see how that makes sense". I am conflating plausible combination with high recall. Precision, in this sense, is that no proposition would be acceptable without evidence or precedent. With all precision, no new idea is ok. So a "knob" for recall vs precision on a generative system would nice. Is "temperature" that knob?
- deleted 3y ago
  [deleted]
- b33j0r 3y ago
  The very first python app I wrote was a statistical bard generator. I had python calculate word frequencies from a Shakespeare corpus and spit them back out with a probability distribution. Honestly it read just as well ;) We’re simpler than we imagine.
  - Qem 3y ago
    > We’re simpler than we imagine. Indeed, creativity (not just randomness) is a quite high bar, that I think even most humans don't pass. Most of us don't create a lot of new things and ideas in our daily lives. We mostly follow the steps of those that came before them, with small adjustments here and there.
    - b33j0r 3y ago
      Recombination makes dna and rna every day. Our limitation in seeing this is our short and fleeting lives. There is a pub:grill in Tempe called monkeypants. Is that a new idea or just clever?
      - dontupvoteme 3y ago
        It's random-ish and "novel", but that could very easily be a reddit username from years ago.
        b33j0r 3y ago
        Haha this thread has gotten too deep, but I appreciated your accurate addition with a lol!
  - shrimpx 3y ago
    Btw Andrej Karpathy builds a Shakespeare LLM here: https://www.youtube.com/watch?v=kCc8FmEb1nY https://www.youtube.com/watch?v=kCc8FmEb1nY
corethree 3y ago
All innovation is composition plus random generation. Which LLMs already do. LLMs have a rudimentary form of innovation. It's not quite as good as humans but it's getting there.
- sidlls 3y ago
  All innovation is composition plus random generation. That’s a fairly bold claim. What supporting data do you have to justify it?
  - corethree 3y ago
    There are two types of statements that can be made in this world. One is data driven based on evidence. The other is logic driven based on axioms and the logical implications of said axioms. My statement is derived from the later. Therefore evidence is unnecessary. It is niave to blindly faith in all truth in the hands of data without understanding nuances between the relationship of data and logic. If you have a pure function that takes the input, the output of that function has 3 possible outcomes. 1. The output of that function is some transformation of the input. 2. The output of the function has nothing to do with the input and is thus generated randomly generated. 3. The output of the function is a combination of both random generation and input transformation. In this case your brain is the function. Input parameters are 100 percent of your existing knowledge. That includes genetic knowledge such as instinctual/rational processing behaviors evolved into the structure of your brain through evolution and learned knowledge such as what you gain when you read a book. The result of the "innovation" operation performed by the brain includes novel output by definition. Thus by logic it must be must include components that are from a certain perspective randomly generated. I guess at first glance it doesn't appear randomly generated because we vet the output and verify it and iterate over several pieces of randomly generated information. Additionally it's not completely random as we only try and test ideas within the realm of possibility. For example I'm not going to account for the possibility that my car will transform into a rock tomorrow that's just too random. But make no mistake. Innovation must be partly randomly generated. Even the idea itself of composing two existing components of knowledge together is itself randomly generated. That being said if we want to be pedantic, there's no real known way to randomly generate stuff via an algorithm. So I use the term "randomly generate" very loosely. Think of it in a similar way to random number generation on your computer: Random from a practical perspective but not technically random.
grogenaut 3y ago
Is the LLM doing the innovation or is the human doing the innoviation via the prompt?
Racing0461 3y ago
Favor composition over innovation.

by vemv 3y ago

troelsSteegin 3y ago

... and our judgments of composition favor recall over precision. Which is to say that as human judges we're good at finding sense in something generated if it's plausible. We'll take a proposition (eg in a composition) and construct a justification for it. "I can see how that makes sense". I am conflating plausible combination with high recall. Precision, in this sense, is that no proposition would be acceptable without evidence or precedent. With all precision, no new idea is ok. So a "knob" for recall vs precision on a generative system would nice. Is "temperature" that knob?

deleted 3y ago
[deleted]
b33j0r 3y ago
The very first python app I wrote was a statistical bard generator. I had python calculate word frequencies from a Shakespeare corpus and spit them back out with a probability distribution. Honestly it read just as well ;) We’re simpler than we imagine.
- Qem 3y ago
  > We’re simpler than we imagine. Indeed, creativity (not just randomness) is a quite high bar, that I think even most humans don't pass. Most of us don't create a lot of new things and ideas in our daily lives. We mostly follow the steps of those that came before them, with small adjustments here and there.
  - b33j0r 3y ago
    Recombination makes dna and rna every day. Our limitation in seeing this is our short and fleeting lives. There is a pub:grill in Tempe called monkeypants. Is that a new idea or just clever?
    - dontupvoteme 3y ago
      It's random-ish and "novel", but that could very easily be a reddit username from years ago.
      - b33j0r 3y ago
        Haha this thread has gotten too deep, but I appreciated your accurate addition with a lol!
- shrimpx 3y ago
  Btw Andrej Karpathy builds a Shakespeare LLM here: https://www.youtube.com/watch?v=kCc8FmEb1nY https://www.youtube.com/watch?v=kCc8FmEb1nY

corethree 3y ago

All innovation is composition plus random generation. Which LLMs already do. LLMs have a rudimentary form of innovation. It's not quite as good as humans but it's getting there.

sidlls 3y ago
All innovation is composition plus random generation. That’s a fairly bold claim. What supporting data do you have to justify it?
- corethree 3y ago
  There are two types of statements that can be made in this world. One is data driven based on evidence. The other is logic driven based on axioms and the logical implications of said axioms. My statement is derived from the later. Therefore evidence is unnecessary. It is niave to blindly faith in all truth in the hands of data without understanding nuances between the relationship of data and logic. If you have a pure function that takes the input, the output of that function has 3 possible outcomes. 1. The output of that function is some transformation of the input. 2. The output of the function has nothing to do with the input and is thus generated randomly generated. 3. The output of the function is a combination of both random generation and input transformation. In this case your brain is the function. Input parameters are 100 percent of your existing knowledge. That includes genetic knowledge such as instinctual/rational processing behaviors evolved into the structure of your brain through evolution and learned knowledge such as what you gain when you read a book. The result of the "innovation" operation performed by the brain includes novel output by definition. Thus by logic it must be must include components that are from a certain perspective randomly generated. I guess at first glance it doesn't appear randomly generated because we vet the output and verify it and iterate over several pieces of randomly generated information. Additionally it's not completely random as we only try and test ideas within the realm of possibility. For example I'm not going to account for the possibility that my car will transform into a rock tomorrow that's just too random. But make no mistake. Innovation must be partly randomly generated. Even the idea itself of composing two existing components of knowledge together is itself randomly generated. That being said if we want to be pedantic, there's no real known way to randomly generate stuff via an algorithm. So I use the term "randomly generate" very loosely. Think of it in a similar way to random number generation on your computer: Random from a practical perspective but not technically random.

grogenaut 3y ago

Is the LLM doing the innovation or is the human doing the innoviation via the prompt?

Racing0461 3y ago

Favor composition over innovation.