3 ms·

Here is an example of a paper on inner alignment misalignment with a relatively simple level of complexity and even reward/punishment layers visualized. https:

by diziet 4y ago

Here is an example of a paper on inner alignment misalignment with a relatively simple level of complexity and even reward/punishment layers visualized. https://arxiv.org/pdf/2105.14111.pdf https://arxiv.org/pdf/2105.14111.pdf The authors show specific examples of goal misalignment. To say that we know how something works because we know the underlying principles is not correct. What is going on in a 150gb~ 65B parameter model (ie, llama 65B) of matrix multiplications is not just complexity, it is tough to say "we know how they work inside". We can explain backpropagation and transformers and RLHF principles. When you scale that to the size of GPT2, the model layers become practically inscrutable to our current understanding. You can have a conceptual understanding of chemistry and physics, but that knowledge is not enough to look at a human brain and say "We know how it works inside!".

YeGoblynQueenne 4y ago
Again, you're talking about unpredictability and the inability to precisely trace the execution of a complex procedure, not about inability to understand how something works, or its general capabilities. Like I say in my other comments above, that a system is unpredictable isn't reason to assume that it's doing something it wasn't designed to do. Not when the system in question is a computer system, or a program, and we know how it was made and why (and given that the unpredictable behaviour is not the result of bugs; I don't think anyone is saying that the "sparks of AGI" in LLMs are the result of bugs). >> To say that we know how something works because we know the underlying principles is not correct. Yes, it is, absolutely. That's the entire point of science, to understand the principles by which the world works so we can explain observations of natural phenomena. Whether this is easy or hard to do, is another matter, and, again, we're not talking about the natural world here but about computers, that we have made ourselves. Nobody expects an airplane to take off on itself and start chasing birds in the sky. What is the reason to assume that Large Language Models are doing something they weren't made to do? >> You can have a conceptual understanding of chemistry and physics, but that knowledge is not enough to look at a human brain and say "We know how it works inside!". Yes, but we don't make human brains. We make computers though and we know how they're made, and how they work. Thanks for the paper. I read the abstract, but it seems to me it's also talking about unpredictable behaviour, rather than some mysterious forces hiding in the depths of a language model's weights. This paper lists many more instances of unpredictable behaviour of machine learning systems: The Surprising Creativity of Digital Evolution: A Collection of Anecdotes from the Evolutionary Computation and Artificial Life Research Communities https://arxiv.org/abs/1803.03453 https://arxiv.org/abs/1803.03453 I recommend it, it's an entertaining read, if nothing else.
- diziet 4y ago
  I wanted to give another parallel - we understand what DNA does from a structural level of the building blocks up, but... We're not close to being able to bio-engineer a 20 point IQ improvement, or the ability to regrow a missing limb, or the ability to grow muscle on demand, or the ability to forgo lipogenesis beyond some amount or or or... To me the epistemological question on this is not just understanding the parts but being able to reason with some high degree of certainty about cause and effect and be able to simulate things, etc. I do not think it is totally correct to say that since we initiated the training runs on the LLMs and hence made them we understand them. There is a large difference between computers as physical engineered devices and X terabyte matrix weights that are a result of an emergent iterative process wherein we train it on a vast amount of data. The latest estimation for GPT3 training set is 500B tokens. Unpredictable behavior manifests itself in the size of models demonstrated in that paper, which are quite small. Even with interpretability tools we have a long way to understanding relatively smaller models, as shown in said paper. The state of the art for the size for current models is vastly larger, and the interpretability challenges are larger too. We're not talking about word2vec here but we're talking about neural nets. Can a large enough neural net with enough layers reproduce the effects of the human brain? Due to speed of neurons firing a lot of actions humans take are akin a few dozens to low hundred layers of a neural net. GPT3 has 96 layers. Yes, there are possibly temporal signals and more complexity in the way the human brain works, but we can solve that with scale. Imagine a model, running on a computer, that is enough to almost perfectly simulate the thinking process a human brain does. Do we know how it works? I don't think we can say we know how human brains work from an information processing capacity. I think we're defining the problem differently. Understanding the underlying hardware (though frankly, I for example don't know what is running on Intel's Management Engine) and principles of software still do not necessarily scale to understanding, especially when in a relatively short time we will have multi trillion parameter multi terabyte scale LLMs or other NN powered models. Now, let's scale it a few more orders of magnitude... Thank you for linking the paper on The Surprising Creativity of Digital Evolution.

by diziet 4y ago

YeGoblynQueenne 4y ago

Again, you're talking about unpredictability and the inability to precisely trace the execution of a complex procedure, not about inability to understand how something works, or its general capabilities. Like I say in my other comments above, that a system is unpredictable isn't reason to assume that it's doing something it wasn't designed to do. Not when the system in question is a computer system, or a program, and we know how it was made and why (and given that the unpredictable behaviour is not the result of bugs; I don't think anyone is saying that the "sparks of AGI" in LLMs are the result of bugs). >> To say that we know how something works because we know the underlying principles is not correct. Yes, it is, absolutely. That's the entire point of science, to understand the principles by which the world works so we can explain observations of natural phenomena. Whether this is easy or hard to do, is another matter, and, again, we're not talking about the natural world here but about computers, that we have made ourselves. Nobody expects an airplane to take off on itself and start chasing birds in the sky. What is the reason to assume that Large Language Models are doing something they weren't made to do? >> You can have a conceptual understanding of chemistry and physics, but that knowledge is not enough to look at a human brain and say "We know how it works inside!". Yes, but we don't make human brains. We make computers though and we know how they're made, and how they work. Thanks for the paper. I read the abstract, but it seems to me it's also talking about unpredictable behaviour, rather than some mysterious forces hiding in the depths of a language model's weights. This paper lists many more instances of unpredictable behaviour of machine learning systems: The Surprising Creativity of Digital Evolution: A Collection of Anecdotes from the Evolutionary Computation and Artificial Life Research Communities https://arxiv.org/abs/1803.03453 https://arxiv.org/abs/1803.03453 I recommend it, it's an entertaining read, if nothing else.

diziet 4y ago
I wanted to give another parallel - we understand what DNA does from a structural level of the building blocks up, but... We're not close to being able to bio-engineer a 20 point IQ improvement, or the ability to regrow a missing limb, or the ability to grow muscle on demand, or the ability to forgo lipogenesis beyond some amount or or or... To me the epistemological question on this is not just understanding the parts but being able to reason with some high degree of certainty about cause and effect and be able to simulate things, etc. I do not think it is totally correct to say that since we initiated the training runs on the LLMs and hence made them we understand them. There is a large difference between computers as physical engineered devices and X terabyte matrix weights that are a result of an emergent iterative process wherein we train it on a vast amount of data. The latest estimation for GPT3 training set is 500B tokens. Unpredictable behavior manifests itself in the size of models demonstrated in that paper, which are quite small. Even with interpretability tools we have a long way to understanding relatively smaller models, as shown in said paper. The state of the art for the size for current models is vastly larger, and the interpretability challenges are larger too. We're not talking about word2vec here but we're talking about neural nets. Can a large enough neural net with enough layers reproduce the effects of the human brain? Due to speed of neurons firing a lot of actions humans take are akin a few dozens to low hundred layers of a neural net. GPT3 has 96 layers. Yes, there are possibly temporal signals and more complexity in the way the human brain works, but we can solve that with scale. Imagine a model, running on a computer, that is enough to almost perfectly simulate the thinking process a human brain does. Do we know how it works? I don't think we can say we know how human brains work from an information processing capacity. I think we're defining the problem differently. Understanding the underlying hardware (though frankly, I for example don't know what is running on Intel's Management Engine) and principles of software still do not necessarily scale to understanding, especially when in a relatively short time we will have multi trillion parameter multi terabyte scale LLMs or other NN powered models. Now, let's scale it a few more orders of magnitude... Thank you for linking the paper on The Surprising Creativity of Digital Evolution.