6 ms·

I'm not convinced. The entire premise here is selection bias at its worst. Here are the steps to create an LLM: 1. Tokenize the training corpus 2. Train the

I'm not convinced. The entire premise here is selection bias at its worst. Here are the steps to create an LLM: 1. Tokenize the training corpus 2. Train the model using the tokens 3. Critique the result 4. Repeat steps 2 & 3 until satisfied We skipped the most important step! 0. Write and curate the training corpus There is an intense focus on the training steps, but that is entirely ignorant of the elephant in the room: the original training data itself. Where did it come from? What does it mean? What is it for? What does it know? We don't train models on noise. We train them on coherent data. The very act of writing is where intelligence lies. A model can only present that intelligence. It cannot create it, and it is not aware of it. There is no objective-thought function. If there was, then the LLM could simply tell us about it. A model is like a coloring book: it doesn't matter how incoherently you stumble around, as long as you stay between the lines. The picture was not invented by coloring it: it was invented by drawing the lines.

doctoboggan 3y ago
> A model can only present that intelligence. It cannot create it, and it is not aware of it. There is no objective-thought function. That's just like, your opinion man. I see both sides of this argument make statements like this that are clearly opinion and not backed up by facts. The truth is for now, we can't say yay or nay on whether the model has reasoning abilities, creative abilities, or a coherent internal world view. It's still a debate which is why there are academics like those in the OP trying to come up with theories that make predictions and then running experiments. It's still early days. If you were to ask my opinion, I'd say these current models are probably closer to the stochastic parrot than an AGI, but there is enough there to start wondering, especially if we have years of improvements coming down the pipeline. These aren't easy questions, and anyone espousing certainty now is doing themselves and others a disservice.
- 2024throwaway 3y ago
  Do you understand how these models work under the hood? It's literally just picking the next most likely word, based on some fancy math. There is nothing that can create new insights or anything approaching intelligence.
  - sowbug 3y ago
    "The statement that P=NP is [...]" (true or false) The article uses the concept of irony and the word "because" as examples of fairly sophisticated next-word predictors. You can imagine others that require even more understanding.
  - doctoboggan 3y ago
    Yes, I understand how these models work under the hood as well as any lay person can. I worked through Andrej's video series and built my own toy model. Your argument is missing the whole concept of emergence, which is what is at question here. How does emergence come about, under what circumstances, and what is possible in those circumstance? It's an open question. Following your logic, one could argue that physics is basic and follows well understood rules. We can do some fancy math to predict what will happen in a given particle interaction. How would these basic particles ever give rise to life and minds with reasoning ability? Well you and I are an existence proof that basic rules when followed on massive systems can give rise to emergent behavior.
    - 2024throwaway 3y ago
      Following your logic, the million monkeys hammering randomly on typewriters that eventually produce the full works of Shakespeare, did so not through pure randomness, but by actually understanding the plight of Romeo and Juliet and the motivation behind Hamlet. Edit since I can't reply: "I don't like your answer, therefore you aren't engaging in good faith." lol
      - doctoboggan 3y ago
        I am aware of the thought experiment you are referring to, but if I witnessed a group of monkeys write the full works of Shakespeare in my lifetime then yes, I would actually consider it something worthy of investigating. However I can see you are not engaging in this conversation in good faith so I don't think there is anything else to discus with you.
        3y ago
- chlorion 3y ago
  Not all of it is an opinion. The LLM does not have the ability to tell you what it knows and what it doesn't know about a subject. In order to get information out of it, you need to present it with a pattern that matches with something it was trained on. An example, is that an LLM can tell you how many programming languages it knows, how much it knows about them (which one it understands best), and how many it doesn't know. If you ask it a question like "how to do X in python", it will have been trained on something that roughly matches that and give a response, but if you ask "what things do you know how to do in python", and it simply is not capable of giving a response. I'm not sure that this means it doesn't have some form of understanding of things, and I do believe that AGI is possible to create in general with sufficient technology, and I do realize that our brains may work in a similar way, but I think LLMs are missing something when it comes to understanding and creating that we have not yet figured out.
  - thomastjeffery 3y ago
    I think that something - or at least one of them - might be objectivity. An LLM has all of the what and none of the why.
  - altruios 3y ago
    well... what we have an an LLM doesn't is state and feedback and restraint, a sense of time. We have an internal state of some kind, an LLM cheats having a state by having every input be the entire conversation + one new input. making the context grow overtime making that state unsustainable. The other thing an LLM is missing is proper feedback, it receives text chunk prompts and output text chunks. since it's output will never 'change the llm's internal state', it can not have a feedback loop. It can mimic one with the trick of refeeding messages as context, but that's untenable after a certain size input. Another is restraint, we human's don't have to respond to everything. we can choose to ignore input entirely, leave people on read, etc... LLM's respond to each input regardless of it is appropriate, and they only respond to stimuli (maybe we do as well, just subtle real world effects). last is a sense of time, which I don't see how the LLM can manage at all. It has no clock. Give an LLM a rich state (okay, maybe rename it too), have it be 'always on', give it a camera and microphone feed for inputs, give it a robot arm to control (pass the output of that back into the model for feedback), give it a second brain to decided to talk or not, and give it a clock too... and what comes out would look leagues better than current AI.
- thomastjeffery 3y ago
  > That's just like, your opinion man. Yeah, that's why I literally said so. You are welcome to disagree with me. My opinion isn't the whole point I intended to make here. I think there is too much focus on "wow look at the emergent behavior of this LLM!", and not enough focus on "where could this specific behavior come from, and why?" The second question may hold the key to both understanding what we have, and making something better.
Lonestar1440 3y ago
Right, it seems incorrect to claim that "The model can generate text that it couldn’t possibly have seen in the training data" when it's been trained on all manner of humans writing about all manner of things. It seems likely that the training data included text involving " self-serving bias, metaphor, statistical syllogism and common-knowledge physics". So, when given a prompt asking for these things; the model is able to 'Stochastic Parrot' an appropriate response. The graph theory work is clever, but the assumptions about the Models, and about Language, seem incorrect.
jordanpg 3y ago
> We don't train models on noise. We train them on coherent data. The very act of writing is where intelligence lies. A model can only present that intelligence. It cannot create it, and it is not aware of it. There is no objective-thought function. If there was, then the LLM could simply tell us about it. We don't train humans on noise. We train them on intangible and tangible things in the world. The very act of choosing and ordering the things is where intelligence lies. A human can only present that intelligence. It cannot create it, and it is not aware of it. There is no objective-thought function. If there was, then the human could simply tell us about it. Turtles all the way down.
- thomastjeffery 3y ago
  Let me put it this way: the text in the training corpus may very well contain the objective thought function. But to the LLM, that is indistinguishable from anything else. It's just another set of patterns between tokens. From the LLM's perspective, there is only "what", and never "why". The "what" may even contain the "why"! Even so, it is always "what" to the LLM's eyes. There is an objective difference between ideas, expression, and logic in my mind. I can't tell you precisely what that difference is made of, but I'm quite certain that LLMs are missing it.
quonn 3y ago
That sounds about right. There is definitely nothing in the _design_ of the model or in the training procedure that is already and by itself intelligent, instead it is an intelligence extraction process. The intelligence is in the data and some of that apparently gets distilled into the model. So we should stop thinking that the intelligence is the (untrained) brain and realize that it has always been the text. I think it makes more sense to compare the current models to unconscious thought or intuition. So the result just kind of pops up as a result of a computation, just like unconscious thought pops up for humans as a result of a (hidden) computation. Humans do conscious reasoning by bringing their thoughts into memory and keeping them there, stretching the process out over time. In some sense LLMs being able to solve some things by being instructed to think step by steps reminds me of that. So perhaps adding some rumination will create a layer on top that appears closer to what we think of as thinking.
weregiraffe 3y ago
Marinate the nether rod Power drill the yippee bog Pressure wash the quiver bone Tokenize the training corpus
truculent 3y ago
Here are the steps to create a sentient primate: 1. Create some amino acids 2. Let the amino acids mix and recombine 3. Let the ones best at reproducing themselves outcompete the rest 4. Repeat steps 2 & 3 until satisfied Of course, this ignores the elephant in the room. These are just _proteins_. What does it mean? What is it for? What does it know? Clearly a mess of amino acids and derivatives can’t actually produce intelligence or understanding. At best, it can merely parrot the intelligence encoded in the base pairs.
- hnfong 3y ago
  Hence the conclusion must be: clearly we have intelligence because God created us. :)
- thomastjeffery 3y ago
  The question we are asking is, "Where does the complexity come from?" A complex result can come from a complex system, or a simple system with complex input (emergence). Human genetics is a complex system. My argument is that an LLM is a simple system. The complexity of its output comes from the complexity of its input, not the system itself. That complexity was created by human minds, not by AI. Without the human minds, there would be no complex LLM output.
  - truculent 3y ago
    Thanks - I think I understand you a bit better now. I suppose I was arguing that genetics is a simple system with emergent complexity (and that perhaps LLMs could be too), but I suspect that betrays my limited understanding of genetics! > We don't train models on noise. We train them on coherent data. The very act of writing is where intelligence lies. A model can only present that intelligence. It cannot create it, and it is not aware of it. I think my main sticking point is the last sentence, which feels like a jump to me. For all the wisdom in your comments on complexity and invention, how does that link to awareness or understanding. Why does understanding require “ownership” of the complexity, so to speak? To me, that’s not clear; could these be emergent properties of a system where the complexity is generated elsewhere, for instance?
lowbloodsugar 3y ago
And this is different from a baby how, other than that the baby takes several years to train? Do you train babies on noise? (There's a dystopian alternate-universe short story on training babies on noise to make them cattle).
- thomastjeffery 3y ago
  Where did the first language come from? Has anyone shown that you can leave a collection of LLMs in an empty environment, and watch them invent their own language or mathematics? I doubt that would work.
lebaux 3y ago
Hah, you would not believe it, but I am planning to use LLM to solve math problems by training it on the posts of my blog. Of course, my blog is silly nonsense, but it will be a fun exercise and address the elephant in the room as well. We will see what results we get for some fun prompts!

doctoboggan 3y ago

> A model can only present that intelligence. It cannot create it, and it is not aware of it. There is no objective-thought function. That's just like, your opinion man. I see both sides of this argument make statements like this that are clearly opinion and not backed up by facts. The truth is for now, we can't say yay or nay on whether the model has reasoning abilities, creative abilities, or a coherent internal world view. It's still a debate which is why there are academics like those in the OP trying to come up with theories that make predictions and then running experiments. It's still early days. If you were to ask my opinion, I'd say these current models are probably closer to the stochastic parrot than an AGI, but there is enough there to start wondering, especially if we have years of improvements coming down the pipeline. These aren't easy questions, and anyone espousing certainty now is doing themselves and others a disservice.

2024throwaway 3y ago
Do you understand how these models work under the hood? It's literally just picking the next most likely word, based on some fancy math. There is nothing that can create new insights or anything approaching intelligence.
- sowbug 3y ago
  "The statement that P=NP is [...]" (true or false) The article uses the concept of irony and the word "because" as examples of fairly sophisticated next-word predictors. You can imagine others that require even more understanding.
- doctoboggan 3y ago
  Yes, I understand how these models work under the hood as well as any lay person can. I worked through Andrej's video series and built my own toy model. Your argument is missing the whole concept of emergence, which is what is at question here. How does emergence come about, under what circumstances, and what is possible in those circumstance? It's an open question. Following your logic, one could argue that physics is basic and follows well understood rules. We can do some fancy math to predict what will happen in a given particle interaction. How would these basic particles ever give rise to life and minds with reasoning ability? Well you and I are an existence proof that basic rules when followed on massive systems can give rise to emergent behavior.
  - 2024throwaway 3y ago
    Following your logic, the million monkeys hammering randomly on typewriters that eventually produce the full works of Shakespeare, did so not through pure randomness, but by actually understanding the plight of Romeo and Juliet and the motivation behind Hamlet. Edit since I can't reply: "I don't like your answer, therefore you aren't engaging in good faith." lol
    - doctoboggan 3y ago
      I am aware of the thought experiment you are referring to, but if I witnessed a group of monkeys write the full works of Shakespeare in my lifetime then yes, I would actually consider it something worthy of investigating. However I can see you are not engaging in this conversation in good faith so I don't think there is anything else to discus with you.
      - 3y ago
chlorion 3y ago
Not all of it is an opinion. The LLM does not have the ability to tell you what it knows and what it doesn't know about a subject. In order to get information out of it, you need to present it with a pattern that matches with something it was trained on. An example, is that an LLM can tell you how many programming languages it knows, how much it knows about them (which one it understands best), and how many it doesn't know. If you ask it a question like "how to do X in python", it will have been trained on something that roughly matches that and give a response, but if you ask "what things do you know how to do in python", and it simply is not capable of giving a response. I'm not sure that this means it doesn't have some form of understanding of things, and I do believe that AGI is possible to create in general with sufficient technology, and I do realize that our brains may work in a similar way, but I think LLMs are missing something when it comes to understanding and creating that we have not yet figured out.
- thomastjeffery 3y ago
  I think that something - or at least one of them - might be objectivity. An LLM has all of the what and none of the why.
- altruios 3y ago
  well... what we have an an LLM doesn't is state and feedback and restraint, a sense of time. We have an internal state of some kind, an LLM cheats having a state by having every input be the entire conversation + one new input. making the context grow overtime making that state unsustainable. The other thing an LLM is missing is proper feedback, it receives text chunk prompts and output text chunks. since it's output will never 'change the llm's internal state', it can not have a feedback loop. It can mimic one with the trick of refeeding messages as context, but that's untenable after a certain size input. Another is restraint, we human's don't have to respond to everything. we can choose to ignore input entirely, leave people on read, etc... LLM's respond to each input regardless of it is appropriate, and they only respond to stimuli (maybe we do as well, just subtle real world effects). last is a sense of time, which I don't see how the LLM can manage at all. It has no clock. Give an LLM a rich state (okay, maybe rename it too), have it be 'always on', give it a camera and microphone feed for inputs, give it a robot arm to control (pass the output of that back into the model for feedback), give it a second brain to decided to talk or not, and give it a clock too... and what comes out would look leagues better than current AI.
thomastjeffery 3y ago
> That's just like, your opinion man. Yeah, that's why I literally said so. You are welcome to disagree with me. My opinion isn't the whole point I intended to make here. I think there is too much focus on "wow look at the emergent behavior of this LLM!", and not enough focus on "where could this specific behavior come from, and why?" The second question may hold the key to both understanding what we have, and making something better.

Lonestar1440 3y ago

Right, it seems incorrect to claim that "The model can generate text that it couldn’t possibly have seen in the training data" when it's been trained on all manner of humans writing about all manner of things. It seems likely that the training data included text involving " self-serving bias, metaphor, statistical syllogism and common-knowledge physics". So, when given a prompt asking for these things; the model is able to 'Stochastic Parrot' an appropriate response. The graph theory work is clever, but the assumptions about the Models, and about Language, seem incorrect.

jordanpg 3y ago

> We don't train models on noise. We train them on coherent data. The very act of writing is where intelligence lies. A model can only present that intelligence. It cannot create it, and it is not aware of it. There is no objective-thought function. If there was, then the LLM could simply tell us about it. We don't train humans on noise. We train them on intangible and tangible things in the world. The very act of choosing and ordering the things is where intelligence lies. A human can only present that intelligence. It cannot create it, and it is not aware of it. There is no objective-thought function. If there was, then the human could simply tell us about it. Turtles all the way down.

thomastjeffery 3y ago
Let me put it this way: the text in the training corpus may very well contain the objective thought function. But to the LLM, that is indistinguishable from anything else. It's just another set of patterns between tokens. From the LLM's perspective, there is only "what", and never "why". The "what" may even contain the "why"! Even so, it is always "what" to the LLM's eyes. There is an objective difference between ideas, expression, and logic in my mind. I can't tell you precisely what that difference is made of, but I'm quite certain that LLMs are missing it.

quonn 3y ago

That sounds about right. There is definitely nothing in the _design_ of the model or in the training procedure that is already and by itself intelligent, instead it is an intelligence extraction process. The intelligence is in the data and some of that apparently gets distilled into the model. So we should stop thinking that the intelligence is the (untrained) brain and realize that it has always been the text. I think it makes more sense to compare the current models to unconscious thought or intuition. So the result just kind of pops up as a result of a computation, just like unconscious thought pops up for humans as a result of a (hidden) computation. Humans do conscious reasoning by bringing their thoughts into memory and keeping them there, stretching the process out over time. In some sense LLMs being able to solve some things by being instructed to think step by steps reminds me of that. So perhaps adding some rumination will create a layer on top that appears closer to what we think of as thinking.

weregiraffe 3y ago

Marinate the nether rod Power drill the yippee bog Pressure wash the quiver bone Tokenize the training corpus

truculent 3y ago

Here are the steps to create a sentient primate: 1. Create some amino acids 2. Let the amino acids mix and recombine 3. Let the ones best at reproducing themselves outcompete the rest 4. Repeat steps 2 & 3 until satisfied Of course, this ignores the elephant in the room. These are just _proteins_. What does it mean? What is it for? What does it know? Clearly a mess of amino acids and derivatives can’t actually produce intelligence or understanding. At best, it can merely parrot the intelligence encoded in the base pairs.

hnfong 3y ago
Hence the conclusion must be: clearly we have intelligence because God created us. :)
thomastjeffery 3y ago
The question we are asking is, "Where does the complexity come from?" A complex result can come from a complex system, or a simple system with complex input (emergence). Human genetics is a complex system. My argument is that an LLM is a simple system. The complexity of its output comes from the complexity of its input, not the system itself. That complexity was created by human minds, not by AI. Without the human minds, there would be no complex LLM output.
- truculent 3y ago
  Thanks - I think I understand you a bit better now. I suppose I was arguing that genetics is a simple system with emergent complexity (and that perhaps LLMs could be too), but I suspect that betrays my limited understanding of genetics! > We don't train models on noise. We train them on coherent data. The very act of writing is where intelligence lies. A model can only present that intelligence. It cannot create it, and it is not aware of it. I think my main sticking point is the last sentence, which feels like a jump to me. For all the wisdom in your comments on complexity and invention, how does that link to awareness or understanding. Why does understanding require “ownership” of the complexity, so to speak? To me, that’s not clear; could these be emergent properties of a system where the complexity is generated elsewhere, for instance?

lowbloodsugar 3y ago

And this is different from a baby how, other than that the baby takes several years to train? Do you train babies on noise? (There's a dystopian alternate-universe short story on training babies on noise to make them cattle).

thomastjeffery 3y ago
Where did the first language come from? Has anyone shown that you can leave a collection of LLMs in an empty environment, and watch them invent their own language or mathematics? I doubt that would work.

lebaux 3y ago

Hah, you would not believe it, but I am planning to use LLM to solve math problems by training it on the posts of my blog. Of course, my blog is silly nonsense, but it will be a fun exercise and address the elephant in the room as well. We will see what results we get for some fun prompts!