5 ms·

> What is the reason to believe that LLMs are an evolutionary step towards AGI at all? In my mind there is a rather large leap from estimating a conditional pro

> What is the reason to believe that LLMs are an evolutionary step towards AGI at all? In my mind there is a rather large leap from estimating a conditional probability of a next token over some space to a conscious entity with its own goals and purpose. In my highly-summarized opinion? When you have a challenging problem with tight constraints, like flight, independent solutions tend to converge toward the same analogous structures that effectively solve that problem, like wings (insects, bats, birds). LLMs are getting so good at mimicing human behavior that it's hard to believe their mathematical structure isn't a close analogue to similar structures in our own brain.* That clearly isn't all you need to make an AGI, but we know little enough about the human brain that I, at least, cannot be sure that there isn't one clever trick that advances an LLM into a general-reasoning agent with its own goals and purpose. I also wouldn't underestimate the power of token prediction. Predicting the future output of a black-box signal generator is a very general problem, whose most accurate solution is attained by running a copy of that black box internally. When that signal generator is human speech, there are some implications to that. (Although I certainly don't believe that LLMs emulate humans, it's now clear by experimental proof that our own thought process is much more compactly modellable than philosophers of previous decades believed). * That's a guess, and unrelated to the deliberately-designed analogy between neural nets and neurons. In LLMs we have built an airplane with wings whose physics we understand in detail; we also ourselves can fly somehow, but we cannot yet see any angel-wings on our back. The more similarities we observe in our flight characteristics, the more this signals that we might be flying the same way ourselves.

flir 3y ago
You presuppose that intelligence is like flight in the ways you've outlined (so solutions are going to converge). Frankly I don't know whether that's true or not, but I want to suggest that it's a bad bet: I would have sworn blind that consciousness is an essential component of intelligence, but the chatbots are starting to make that look like a poor assumption on my part. When we know so little about intelligence, can we really assume there's only one way to be intelligent? To extend your analogy, I think that the intelligence equivalents of helicopters and rockets are out there somewhere, waiting to be found. I think I'm with Dijkstra on this one: "The question of whether machines can think is about as relevant as the question of whether submarines can swim" I think we're going to end up with submarines (or helicopters), not dolphins (or birds). No animal has evolved wheels, but wheels are a pretty good solution to the problem of movement. Maybe it's truer to say there's only one way to evolve an intelligent mammal, because you have to work with what already exists in the mammalian body. But AI research isn't constrained in that way. (Not saying you're wrong, just arguing we don't know enough to know if you're right).
- mitthrowaway2 3y ago
  I agree we don't know enough to know if I'm right! I tried to use a lot of hedgy-words. But it's not a presupposition, merely a line of argument why it's not a complete absurdity to think LLMs might be a step towards AGI. I do think consciousness is beside the point, as we have no way to test whether LLMs are conscious, just like we can't test anything else. We don't know what consciousness is, nor what it isn't. I don't think Dijkstra's argument applies here. Whether submarines "swim" is a good point about our vague mental boundaries of the word "swim". But submarine propellers are absolutely a convergent structure for underwater propulsion: it's the same hydrodynamic-lift-generating motion of a fin, just continuous instead of reciprocating. That's very much more structurally similar than I expect LLMs are to any hardware we have in our heads. It's true that the solution space for AI is in some ways less constrained than for biological intelligence, but just like submarines and whales operate under the same Navier-Stokes equations, humans and AI must learn and reason under the same equations of probability. Working solutions will probably have some mathematical structure in common. I think more relevant is Von Neumann: "If you will tell me precisely what it is that a machine cannot do, then I can always make a machine which will do just that!" Whether a submarine swims is a matter of semantics, but if there's a manuever that a whale can execute that a submarine cannot, then at least we can all agree about the non-generality of its swimming. For AGI, I can't say whether it's conscious or really thinks, but for the sake of concrete argument, it's dangerous enough to be concerned if: - it can form and maintain an objective; it can identify plausible steps to achieve that objective; it can accurately predict human responses to its actions; it can decently model the environment, as we can; it can hide its objectives from interrogators, and convince them that its actions are in their interests; it can deliver enough value to be capable of earning money through its actions; it can propose ideas that can convince investors to part with $100 billion; it can design a chemical plant that appears at a cursory inspection to manufacture high-profit fluorochemicals, but which also actually manufactures and stores CFCs in sufficient quantity to threaten the viability of terrestrial agriculture.
  - kubanczyk 3y ago
    > it can form and maintain an objective; Yes and yes. > it can identify plausible steps to achieve that objective; It can predict. > it can accurately predict human responses to its actions; It can predict. > it can decently model the environment, as we can; It can predict. > it can hide its objectives from interrogators, and convince them that its actions are in their interests; It can predict. > it can deliver enough value to be capable of earning money through its actions; It can predict. > it can propose ideas that can convince investors to part with $100 billion; It can predict. > it can design a chemical plant... It can predict.
AlexandrB 3y ago
Flight is actually a perfect counterexample to x-risk nonsense. When flight was invented, people naturally assumed that it would continue advancing until we had flying cars and could get anywhere on the globe in a matter of minutes. Turns out there are both economic and practical limits to what is possible with flight and modern commercial airplanes don't look much different than those from 60 years ago. AGI/x-risk alarmists are looking at the Wright Brothers plane and trying to prevent/ban supersonic flying cars, even though it's not clear the technology will ever be capable of such a thing.
- mitthrowaway2 3y ago
  If we lived in a world were hypersonic birds flying anywhere on the globe in a matter of minutes, then I think it would be quite reasonable to anticipate airplanes catching up to them.

flir 3y ago

You presuppose that intelligence is like flight in the ways you've outlined (so solutions are going to converge). Frankly I don't know whether that's true or not, but I want to suggest that it's a bad bet: I would have sworn blind that consciousness is an essential component of intelligence, but the chatbots are starting to make that look like a poor assumption on my part. When we know so little about intelligence, can we really assume there's only one way to be intelligent? To extend your analogy, I think that the intelligence equivalents of helicopters and rockets are out there somewhere, waiting to be found. I think I'm with Dijkstra on this one: "The question of whether machines can think is about as relevant as the question of whether submarines can swim" I think we're going to end up with submarines (or helicopters), not dolphins (or birds). No animal has evolved wheels, but wheels are a pretty good solution to the problem of movement. Maybe it's truer to say there's only one way to evolve an intelligent mammal, because you have to work with what already exists in the mammalian body. But AI research isn't constrained in that way. (Not saying you're wrong, just arguing we don't know enough to know if you're right).

mitthrowaway2 3y ago
I agree we don't know enough to know if I'm right! I tried to use a lot of hedgy-words. But it's not a presupposition, merely a line of argument why it's not a complete absurdity to think LLMs might be a step towards AGI. I do think consciousness is beside the point, as we have no way to test whether LLMs are conscious, just like we can't test anything else. We don't know what consciousness is, nor what it isn't. I don't think Dijkstra's argument applies here. Whether submarines "swim" is a good point about our vague mental boundaries of the word "swim". But submarine propellers are absolutely a convergent structure for underwater propulsion: it's the same hydrodynamic-lift-generating motion of a fin, just continuous instead of reciprocating. That's very much more structurally similar than I expect LLMs are to any hardware we have in our heads. It's true that the solution space for AI is in some ways less constrained than for biological intelligence, but just like submarines and whales operate under the same Navier-Stokes equations, humans and AI must learn and reason under the same equations of probability. Working solutions will probably have some mathematical structure in common. I think more relevant is Von Neumann: "If you will tell me precisely what it is that a machine cannot do, then I can always make a machine which will do just that!" Whether a submarine swims is a matter of semantics, but if there's a manuever that a whale can execute that a submarine cannot, then at least we can all agree about the non-generality of its swimming. For AGI, I can't say whether it's conscious or really thinks, but for the sake of concrete argument, it's dangerous enough to be concerned if: - it can form and maintain an objective; it can identify plausible steps to achieve that objective; it can accurately predict human responses to its actions; it can decently model the environment, as we can; it can hide its objectives from interrogators, and convince them that its actions are in their interests; it can deliver enough value to be capable of earning money through its actions; it can propose ideas that can convince investors to part with $100 billion; it can design a chemical plant that appears at a cursory inspection to manufacture high-profit fluorochemicals, but which also actually manufactures and stores CFCs in sufficient quantity to threaten the viability of terrestrial agriculture.
- kubanczyk 3y ago
  > it can form and maintain an objective; Yes and yes. > it can identify plausible steps to achieve that objective; It can predict. > it can accurately predict human responses to its actions; It can predict. > it can decently model the environment, as we can; It can predict. > it can hide its objectives from interrogators, and convince them that its actions are in their interests; It can predict. > it can deliver enough value to be capable of earning money through its actions; It can predict. > it can propose ideas that can convince investors to part with $100 billion; It can predict. > it can design a chemical plant... It can predict.

AlexandrB 3y ago

Flight is actually a perfect counterexample to x-risk nonsense. When flight was invented, people naturally assumed that it would continue advancing until we had flying cars and could get anywhere on the globe in a matter of minutes. Turns out there are both economic and practical limits to what is possible with flight and modern commercial airplanes don't look much different than those from 60 years ago. AGI/x-risk alarmists are looking at the Wright Brothers plane and trying to prevent/ban supersonic flying cars, even though it's not clear the technology will ever be capable of such a thing.

mitthrowaway2 3y ago
If we lived in a world were hypersonic birds flying anywhere on the globe in a matter of minutes, then I think it would be quite reasonable to anticipate airplanes catching up to them.