3 ms·

>This is just rephrasing "it's not useful for AI." When you say "model intelligence", what you're referring to is AI. I think this is somewhat circular. You're

>This is just rephrasing "it's not useful for AI." When you say "model intelligence", what you're referring to is AI. I think this is somewhat circular. You're evaluating what techniques are useful for AI. Then you're looking at systems outside of AI and saying "those systems aren't relevant to human learning and can't be how humans learn, because it's not how we build AI." I'm not talking about AI. I'm taking about intelligence in general. Most Humans would be trashed on a formal logic test. Yes we can be logical no doubt but it's pretty clear we're not running off logic. If you have an idea of how something works and set off building one from that idea then failing to succeed calls into question the validity of that idea. This is science at its core. We couldn't really build artificial intelligence the way we thought intelligence worked. So does intelligence really work this way ? That is the crux of the issue. Intelligent creatures that run off logic basically only exist in fiction. Humans don't do it. Animals don't it. The machines we tried to build to do it this way mostly didn't work. The ones that did work are perfect and never err, more hints humans don't work this way. So we basically have no real indication that logic is at the heart of intelligence. >Does prediction require an accurate model of the world or not? Performant Prediction just requires a model, completely accurate or not. Newton's model of gravity is wrong but it's useful fiction so it even still gets taught in schools. Einstein's model is better than Newton's but is still wrong (does not reconcile with quantum mechanics). Doesn't stop it from making useful predictions. What I'm saying is: 1. Prediction requires a model. The model doesn't have to be "true" to be useful or performant. 2. Perfect Prediction requires a perfect model. What is a Language Model's goal ? What is the loss trending down to ? It's not to make useful predictions and stop. It is to make perfect predictions of the data they are presented (so of course perfect here doesn't mean a perfect view of the world but rather a perfect view of the world as humans see it) Everytime the machine uses its existing model to make an erring prediction, it's model is quite literally adjusted and changed to accommodate this error. But by bit this happens. As a result, what GPT-2 computes is wildly different from what GPT-3 computes and that is different from what GPT-4 computes. The goal is not to play chess. GPT is not going to stop on some arbitrary line of competency or error rate someone may draw up. The goal is to model the chess dataset it's been given. In Evolution, the pressures of two creatures can be just to fly. Plenty of wiggle room. In gradient descent prediction, the pressures are to fly as some particular creature flies. I'm not imagining 2 optimizers converging at random. What I'm saying is that the training paradigm is pushing things in that general direction. >It does bring us back to the original question: if GPT is modeling the world of chess in its answers, why does it make illegal chess moves at a significantly higher rate than humans? Why does it play chess differently from humans? Is it bad at chess, or... perhaps the way it models chess is fundamentally different from how humans model chess? The simple take is that if you have something hitting 1800 elo but it plays in a way that looks alien to how humans play, it's probably doing something different under the hood. It's definitely modelling Chess. Othello GPT will construct a board state of the pieces of the game before every prediction. https://thegradient.pub/othello/ https://thegradient.pub/othello/ But again, models don't have to perfect before they are performant. The road from A, the nothing predictor to B, the perfect predictor is not instantaneous. GPT is not done modelling Human Chess. >Those two claims don't jive with each other. If you're arguing that multiple approaches to prediction are possible and that they don't require knowledge or understanding of the underlying systems, then why do you think it's impossible for an AI to make predictions without accurately modeling the world? I believe I've already explained my point here. When I say multiple approaches, I'm not thinking model(s) vs no model. There will never be performant prediction that does not model its data in some way. But that model can be different. Newton and Einstein paint a very different picture of gravity. But just like Experimental Science trends towards more and more accurate models of the Universe with time, so does gradient descent prediction trend towards a more and more accurate model of its data. >If you're arguing that making more and more accurate predictions about the world fundamentally requires an AI to understand the underlying mechanisms of the world, then why are you so laissez-faire about efforts to understand how GPT works when making predictions about its capabilities? I simply said we can make performant predictions without understanding the internals. I'm being practical. That reality is far more likely than divining the meaning of billions of connections performing computations we had no hand in teaching. Also recall I said, "As a result, what GPT-2 computes is wildly different from what GPT-3 computes and that is different from what GPT-4 computes." This means knowing how 3 works internally doesn't mean you know how 4 works internally.

danShumway 3y ago
> Intelligent creatures that run off logic basically only exist in fiction. Of course humans don't run off of only logic, but the idea that humans don't use logical reasoning as part of our thinking patterns at all or as part of our learning processes is transparently false. We teach people logical systems, humans do use logic to learn, and we do very obviously use logical systems (as a subset of many systems) to think about problems. We don't only use logic, sometimes we think of ourselves as more logical than we actually are, but we do use logic as part of learning process or else the entire school system wouldn't work. Why are you so convinced that there is a singular learning process at the center of all human learning? Brains are complicated, there's no reason to believe that would be the case. Human brains develop skills using multiple strategies, logic being one of them. But regardless, we do know that whatever the learning process for humans is, we are not trained the same way that GPT is trained, and we know that we learn and exhibit skills in different orders than LLMs exhibit those same skills, and we know that our learning methods once we have matured regularly differ from GPT's training methods, and we know that efforts to imitate GPT's learning methods seem to produce worse results in multiple areas when used to teach human beings -- which is interesting to think about from your replication angle; why is that when we use GPT-style reinforcement training for humans the results are terrible? So I don't understand what's controversial about any of that or how anyone could argue against it, it is plainly observable from simply looking at the world that the way we teach kids and demonstrate learned skills does not perfectly map to GPT. ---- > If you have an idea of how something works and set off building one from that idea then failing to succeed calls into question the validity of that idea. This is science at its core. No, science is about testable predictions. Perfect replication within a lab is great, but it is not our baseline standard for whether or not something exists in reality. > Performant Prediction just requires a model, completely accurate or not. [...] Perfect Prediction requires a perfect model. No one at all at any point in this conversation has been saying that GPT doesn't have any model of chess, we're saying its model of chess seems to differ from the one that humans use. We are saying that it does not think about chess the same way that a human does. Honestly, this sounds like you're agreeing with me. Models that are not strictly mapped to reality can still be useful, that's why Newton's model of gravity is useful even though it's wrong. GPT can have a model of chess that is divorced from human understanding of chess, and that can still be a useful model, but it does not appear to be the same model that humans use. Notably, as you bring up, GPT is still learning and does not have a perfect prediction model, so we can throw that right out. We know that GPT does not have perfect internal model of chess because it's not producing perfect results. > Everytime the machine uses its existing model to make an erring prediction, it's model is quite literally adjusted and changed to accommodate this error. But by bit this happens. As a result, what GPT-2 computes is wildly different from what GPT-3 computes and that is different from what GPT-4 computes. So... again. They do have differing models then, even different versions of GPT have differing models from each other, and minor variations in training can produce very different internal models even if the underlying structures are the same. And this all really sounds like you're agreeing with me but are presenting it as a disagreement because... I don't know why, because the implication that GPT has a differing model to chess that is not based on reasoning about the ruleset in the same way as humans is somehow seen as a slight against the tool? Because you don't believe that humans ever learn by looking at rules and extrapolating from them, even though they very clearly and demonstrably do? Something else? Where is the beef here? I mean, you go on to say: > It's definitely modelling Chess. In response to a paragraph where I literally directly refer to GPT as having an internal model of chess. No one is saying that GPT doesn't use any kind of internal modeling when it approaches problems. But as you yourself point out "models don't have to be be perfect before they are performant." The fact that GPT can build a useful model for playing chess does not require it to have a perfect model of the rules of chess, and in fact a model of chess that was built on top of the rules of chess would produce different-looking results. And of course, modeling a representation of a board would not change that fact. ---- > I simply said we can make performant predictions without understanding the internals. Do you believe that it could be possible for GPT-4 to make performant predictions without understanding the internals of the systems it's making predictions about? Do you believe that GPT is trained to solve a practical problem, or that it's trained to perfectly model the world? -- because as you yourself say, those are different things and imperfect models can sometimes be more practical and efficient than a fully perfectly modeled system. How certain are you that a system like GPT-4 can't possibly make performant predictions about chess without accurately understanding the game's internals? I mean, you seem to understand that humans can make predictions that way, that we can utilize systems that we don't fully understand. Why couldn't GPT be doing that; do you think that GPT is not able to replicate human predictive capabilities?

danShumway 3y ago

> Intelligent creatures that run off logic basically only exist in fiction. Of course humans don't run off of only logic, but the idea that humans don't use logical reasoning as part of our thinking patterns at all or as part of our learning processes is transparently false. We teach people logical systems, humans do use logic to learn, and we do very obviously use logical systems (as a subset of many systems) to think about problems. We don't only use logic, sometimes we think of ourselves as more logical than we actually are, but we do use logic as part of learning process or else the entire school system wouldn't work. Why are you so convinced that there is a singular learning process at the center of all human learning? Brains are complicated, there's no reason to believe that would be the case. Human brains develop skills using multiple strategies, logic being one of them. But regardless, we do know that whatever the learning process for humans is, we are not trained the same way that GPT is trained, and we know that we learn and exhibit skills in different orders than LLMs exhibit those same skills, and we know that our learning methods once we have matured regularly differ from GPT's training methods, and we know that efforts to imitate GPT's learning methods seem to produce worse results in multiple areas when used to teach human beings -- which is interesting to think about from your replication angle; why is that when we use GPT-style reinforcement training for humans the results are terrible? So I don't understand what's controversial about any of that or how anyone could argue against it, it is plainly observable from simply looking at the world that the way we teach kids and demonstrate learned skills does not perfectly map to GPT. ---- > If you have an idea of how something works and set off building one from that idea then failing to succeed calls into question the validity of that idea. This is science at its core. No, science is about testable predictions. Perfect replication within a lab is great, but it is not our baseline standard for whether or not something exists in reality. > Performant Prediction just requires a model, completely accurate or not. [...] Perfect Prediction requires a perfect model. No one at all at any point in this conversation has been saying that GPT doesn't have any model of chess, we're saying its model of chess seems to differ from the one that humans use. We are saying that it does not think about chess the same way that a human does. Honestly, this sounds like you're agreeing with me. Models that are not strictly mapped to reality can still be useful, that's why Newton's model of gravity is useful even though it's wrong. GPT can have a model of chess that is divorced from human understanding of chess, and that can still be a useful model, but it does not appear to be the same model that humans use. Notably, as you bring up, GPT is still learning and does not have a perfect prediction model, so we can throw that right out. We know that GPT does not have perfect internal model of chess because it's not producing perfect results. > Everytime the machine uses its existing model to make an erring prediction, it's model is quite literally adjusted and changed to accommodate this error. But by bit this happens. As a result, what GPT-2 computes is wildly different from what GPT-3 computes and that is different from what GPT-4 computes. So... again. They do have differing models then, even different versions of GPT have differing models from each other, and minor variations in training can produce very different internal models even if the underlying structures are the same. And this all really sounds like you're agreeing with me but are presenting it as a disagreement because... I don't know why, because the implication that GPT has a differing model to chess that is not based on reasoning about the ruleset in the same way as humans is somehow seen as a slight against the tool? Because you don't believe that humans ever learn by looking at rules and extrapolating from them, even though they very clearly and demonstrably do? Something else? Where is the beef here? I mean, you go on to say: > It's definitely modelling Chess. In response to a paragraph where I literally directly refer to GPT as having an internal model of chess. No one is saying that GPT doesn't use any kind of internal modeling when it approaches problems. But as you yourself point out "models don't have to be be perfect before they are performant." The fact that GPT can build a useful model for playing chess does not require it to have a perfect model of the rules of chess, and in fact a model of chess that was built on top of the rules of chess would produce different-looking results. And of course, modeling a representation of a board would not change that fact. ---- > I simply said we can make performant predictions without understanding the internals. Do you believe that it could be possible for GPT-4 to make performant predictions without understanding the internals of the systems it's making predictions about? Do you believe that GPT is trained to solve a practical problem, or that it's trained to perfectly model the world? -- because as you yourself say, those are different things and imperfect models can sometimes be more practical and efficient than a fully perfectly modeled system. How certain are you that a system like GPT-4 can't possibly make performant predictions about chess without accurately understanding the game's internals? I mean, you seem to understand that humans can make predictions that way, that we can utilize systems that we don't fully understand. Why couldn't GPT be doing that; do you think that GPT is not able to replicate human predictive capabilities?