4 ms·

>> We don't really know how they work inside. Yes, we do. We know their design principles, their operation principles, the designs of specific instances, and e

>> We don't really know how they work inside. Yes, we do. We know their design principles, their operation principles, the designs of specific instances, and everything else that has anything to do with "how they work inside". We know how they work inside. We know how they work inside and out. What we can't do is predict their output, given some input (i.e. a prompt). That's because Large Language Models are so, well, large, and complex, that it's impossible to reproduce their calculations without ... a Large Language Model. In the same way, we can't predict the output of a Random Number Generator. Very much like a large language model, we know how an RNG works, but if we could predict its behaviour, then it wouldn't be working correctly. But why is the inability to predict the behaviour of a complex system reason to imagine all sorts of mysterious and fantastical things happening inside the system, such as it "thinking", or "developing some kind of internal world model"? Those are not things that language models, small or large, are designed to do, and there is no obvious reason why they should be able to do them. There is no reason to assume they can do them either: the fact that they are models of language, trained to reproduce language, and nothing else, suffices to explain their observed behaviour. There are no mysteries of intelligence, or artificial intelligence, to uncover in the study of large language models' behaviour. I am perfectly well aware that there are many people who absolutely want there to be such mysteries, there, and that will overlook everything we know about how language models actually work, to be able to fantasise about those mysteries. But those are fantasies. They are superstitions. Superstitious beliefs about an artifact of technology, that we have created ourselves, and that some of us have now chosen to revere and wonder at as if it was an artifact of an alien civilisation fallen from the sky. Also, on a personal note: not you too, Animats... come on, you're a knowledgeable one. Don't just get swept up in the crowd's madness like that.

auggierose 4y ago
Just because we know the definition of something, doesn't mean we know all there is to know about it. Every mathematician knows that. Yes, we know the definition of GPT. But that it is so good at doing what it does, with such a freakingly simple method, one word after another, is emergent behaviour, and wasn't known to us before seeing it with our own eyes. We still don't know WHY it is so good at it. If we did, maybe we could construct one a hundred times smaller.
- juve1996 4y ago
  I don't think that matters as much as we think it does, though. There's a lot of internal things we don't know about how the brain works - but we can still describe the macro.
- YeGoblynQueenne 4y ago
  >> Just because we know the definition of something, doesn't mean we know all there is to know about it. Every mathematician knows that. Well, I don't know about mathematicians. I'm a computer scientist. I study computers. And I know that the behaviour of a computer is entirely determined by its design, and its inputs. If a computer is doing something that is not in its design to do, then that's because there is an error in the implementation of its design. And if it's difficult, or impossible, to anticipate the behaviour of a computer, that's because it's programmed to do something unpredictable- like in my RNG example above. Computers are machines. Programs are other machines, running on computers. There are no spirits in any level of a computer's functionality, no magick, and no metaphysical mysteries to solve. Maybe that's not how it works in other sciences, that study the natural world, where there are too many unknowns. But in computers there are no unknowns. We make them, and we know how they work. Regarding LLLMs, we know why they're so good at what they do, because we know what they do, because we are the ones who made them to do it, and we know that what they do gets better the larger the number of parameters and the larger the data they're trained on. We've known that for decades. And as to "emergent behaviour", that is just asking for a mystery to be there, when no mystery is anywhere near the whole thing. "Emergent behaviour" just means what I say above, that there's a complex system that displays behaviour that is difficult to predict, even if its design principles are very well known. There's a time to wonder, and wonder is the seed of science. But to wonder about a machine that is doing what it's designed to do, is not the wonder that leads to science. That kind of wonder, well, it reminds me of the old joke: They ask a mystic what is the greatest mystery in nature. "The thermos!" he says. "It keeps my tea hot in the winter, but cold in the summer. That little bottle! How does it know?"
  - auggierose 4y ago
    > Well, I don't know about mathematicians. I'm a computer scientist. Does an LLM gain "consciousness", like ours? I doubt it. There is a doubt here though, because while I know how an LLM works, I don't know how we work. Can an LLM, through training, gain an "understanding" that rivals our notion of consciousness and can in fact simulate it, purely through mathematical models it forms? Well, you are not a mathematician, so how would you know? You say we have known for decades that LLMs would work? Liar.
diziet 4y ago
Here is an example of a paper on inner alignment misalignment with a relatively simple level of complexity and even reward/punishment layers visualized. https://arxiv.org/pdf/2105.14111.pdf https://arxiv.org/pdf/2105.14111.pdf The authors show specific examples of goal misalignment. To say that we know how something works because we know the underlying principles is not correct. What is going on in a 150gb~ 65B parameter model (ie, llama 65B) of matrix multiplications is not just complexity, it is tough to say "we know how they work inside". We can explain backpropagation and transformers and RLHF principles. When you scale that to the size of GPT2, the model layers become practically inscrutable to our current understanding. You can have a conceptual understanding of chemistry and physics, but that knowledge is not enough to look at a human brain and say "We know how it works inside!".
- YeGoblynQueenne 4y ago
  Again, you're talking about unpredictability and the inability to precisely trace the execution of a complex procedure, not about inability to understand how something works, or its general capabilities. Like I say in my other comments above, that a system is unpredictable isn't reason to assume that it's doing something it wasn't designed to do. Not when the system in question is a computer system, or a program, and we know how it was made and why (and given that the unpredictable behaviour is not the result of bugs; I don't think anyone is saying that the "sparks of AGI" in LLMs are the result of bugs). >> To say that we know how something works because we know the underlying principles is not correct. Yes, it is, absolutely. That's the entire point of science, to understand the principles by which the world works so we can explain observations of natural phenomena. Whether this is easy or hard to do, is another matter, and, again, we're not talking about the natural world here but about computers, that we have made ourselves. Nobody expects an airplane to take off on itself and start chasing birds in the sky. What is the reason to assume that Large Language Models are doing something they weren't made to do? >> You can have a conceptual understanding of chemistry and physics, but that knowledge is not enough to look at a human brain and say "We know how it works inside!". Yes, but we don't make human brains. We make computers though and we know how they're made, and how they work. Thanks for the paper. I read the abstract, but it seems to me it's also talking about unpredictable behaviour, rather than some mysterious forces hiding in the depths of a language model's weights. This paper lists many more instances of unpredictable behaviour of machine learning systems: The Surprising Creativity of Digital Evolution: A Collection of Anecdotes from the Evolutionary Computation and Artificial Life Research Communities https://arxiv.org/abs/1803.03453 https://arxiv.org/abs/1803.03453 I recommend it, it's an entertaining read, if nothing else.

auggierose 4y ago

Just because we know the definition of something, doesn't mean we know all there is to know about it. Every mathematician knows that. Yes, we know the definition of GPT. But that it is so good at doing what it does, with such a freakingly simple method, one word after another, is emergent behaviour, and wasn't known to us before seeing it with our own eyes. We still don't know WHY it is so good at it. If we did, maybe we could construct one a hundred times smaller.

juve1996 4y ago
I don't think that matters as much as we think it does, though. There's a lot of internal things we don't know about how the brain works - but we can still describe the macro.
YeGoblynQueenne 4y ago
>> Just because we know the definition of something, doesn't mean we know all there is to know about it. Every mathematician knows that. Well, I don't know about mathematicians. I'm a computer scientist. I study computers. And I know that the behaviour of a computer is entirely determined by its design, and its inputs. If a computer is doing something that is not in its design to do, then that's because there is an error in the implementation of its design. And if it's difficult, or impossible, to anticipate the behaviour of a computer, that's because it's programmed to do something unpredictable- like in my RNG example above. Computers are machines. Programs are other machines, running on computers. There are no spirits in any level of a computer's functionality, no magick, and no metaphysical mysteries to solve. Maybe that's not how it works in other sciences, that study the natural world, where there are too many unknowns. But in computers there are no unknowns. We make them, and we know how they work. Regarding LLLMs, we know why they're so good at what they do, because we know what they do, because we are the ones who made them to do it, and we know that what they do gets better the larger the number of parameters and the larger the data they're trained on. We've known that for decades. And as to "emergent behaviour", that is just asking for a mystery to be there, when no mystery is anywhere near the whole thing. "Emergent behaviour" just means what I say above, that there's a complex system that displays behaviour that is difficult to predict, even if its design principles are very well known. There's a time to wonder, and wonder is the seed of science. But to wonder about a machine that is doing what it's designed to do, is not the wonder that leads to science. That kind of wonder, well, it reminds me of the old joke: They ask a mystic what is the greatest mystery in nature. "The thermos!" he says. "It keeps my tea hot in the winter, but cold in the summer. That little bottle! How does it know?"
- auggierose 4y ago
  > Well, I don't know about mathematicians. I'm a computer scientist. Does an LLM gain "consciousness", like ours? I doubt it. There is a doubt here though, because while I know how an LLM works, I don't know how we work. Can an LLM, through training, gain an "understanding" that rivals our notion of consciousness and can in fact simulate it, purely through mathematical models it forms? Well, you are not a mathematician, so how would you know? You say we have known for decades that LLMs would work? Liar.

diziet 4y ago

Here is an example of a paper on inner alignment misalignment with a relatively simple level of complexity and even reward/punishment layers visualized. https://arxiv.org/pdf/2105.14111.pdf https://arxiv.org/pdf/2105.14111.pdf The authors show specific examples of goal misalignment. To say that we know how something works because we know the underlying principles is not correct. What is going on in a 150gb~ 65B parameter model (ie, llama 65B) of matrix multiplications is not just complexity, it is tough to say "we know how they work inside". We can explain backpropagation and transformers and RLHF principles. When you scale that to the size of GPT2, the model layers become practically inscrutable to our current understanding. You can have a conceptual understanding of chemistry and physics, but that knowledge is not enough to look at a human brain and say "We know how it works inside!".

YeGoblynQueenne 4y ago
Again, you're talking about unpredictability and the inability to precisely trace the execution of a complex procedure, not about inability to understand how something works, or its general capabilities. Like I say in my other comments above, that a system is unpredictable isn't reason to assume that it's doing something it wasn't designed to do. Not when the system in question is a computer system, or a program, and we know how it was made and why (and given that the unpredictable behaviour is not the result of bugs; I don't think anyone is saying that the "sparks of AGI" in LLMs are the result of bugs). >> To say that we know how something works because we know the underlying principles is not correct. Yes, it is, absolutely. That's the entire point of science, to understand the principles by which the world works so we can explain observations of natural phenomena. Whether this is easy or hard to do, is another matter, and, again, we're not talking about the natural world here but about computers, that we have made ourselves. Nobody expects an airplane to take off on itself and start chasing birds in the sky. What is the reason to assume that Large Language Models are doing something they weren't made to do? >> You can have a conceptual understanding of chemistry and physics, but that knowledge is not enough to look at a human brain and say "We know how it works inside!". Yes, but we don't make human brains. We make computers though and we know how they're made, and how they work. Thanks for the paper. I read the abstract, but it seems to me it's also talking about unpredictable behaviour, rather than some mysterious forces hiding in the depths of a language model's weights. This paper lists many more instances of unpredictable behaviour of machine learning systems: The Surprising Creativity of Digital Evolution: A Collection of Anecdotes from the Evolutionary Computation and Artificial Life Research Communities https://arxiv.org/abs/1803.03453 https://arxiv.org/abs/1803.03453 I recommend it, it's an entertaining read, if nothing else.