4 ms·

This is a good discussion of concepts. However, a lot of the folks worried about superintelligence miss an important division within the concept of "intelligenc

by lukev 3y ago

This is a good discussion of concepts. However, a lot of the folks worried about superintelligence miss an important division within the concept of "intelligence." Specifically with respect to the current generation of LLMs. Namely, there's a huge difference between the concept of intelligence and an intelligent agent. You could make the argument that current LLMs have some level of intelligence. But (without a lot of supporting infrastructure) they are not agents, they have no built in mechanism for having any kind of intent or even short-term memory. Because they fundamentally just generate a single token at once, they don't even exist "within time" the way we normally would think of an intelligent entity as doing... they're more like a mathematical function that you can invoke on demand. Perhaps to get an "intelligent" result, but conflating that kind of intelligence with an intelligent entity or actor seems like a mistake. To use the analogy in the OP, it would be like calling a single hydraulic piston "strong". Sure, but unless you attach it to something it's not the same kind of strong as a bodybuilder is.

mquander 3y ago
OK, but the hard part was the "intelligent" part, right? If you have the "intelligent" thing, and you want to get an "intelligent agent", all you have to do at most is wrap it in a shell script that runs the equivalent of an OODA loop, e.g. AutoGPT. So we should expect a "superintelligent agent" to appear pretty much the next day after a "superintelligence" is created.
- lukev 3y ago
  Perhaps, I don't discount the possibility that an LLM could be used as a primitive as part of a larger more complex system that could rightfully be called intelligent. But a script running a LLM in an OODA loop is a lot easier to monitor and control than some inscrutable "ghost in the parameters" that a lot of AI alarmists seem worried about. You can actually see a transcript of its "thoughts", for one!
  - Footkerchief 3y ago
    We already have self-replicating scripts that can prove very hard to detect and destroy, even without the benefit of LLM-powered fuzzing and social engineering.
    - lukev 3y ago
      First of all, we're assuming that agents are (at least initially) designed intentionally, not as harmful viruses. Second, running a LLM inference (nevermind training!) involves lots and lots of system resources. Much more difficult to hide than a tiny backdoor.
- joe_the_user 3y ago
  OK, but the hard part was the "intelligent" part, right? If you have the "intelligent" thing, and you want to get an "intelligent agent", all you have to do at most is wrap it in a shell script that runs the equivalent of an OODA loop, e.g. AutoGPT. No, not really. Neither AutoGPT or any similar actually work. And the only justification for expecting them to work the future is same naively extrapolation more of one kind of quality will automatically give a thing more of a correlated quality. The humming bird building airplanes: "If we continue to increase the capacity of these flying machines, eventually they will have 'super-bird abilities' and at that point they will naturally steal all of nectar..."
  - mquander 3y ago
    AutoGPT doesn't work because GPT-4 is too stupid to make good plans, not because it's missing the secret sauce of "agentiness." If some future software was really great at observing, planning, and reasoning, then an AutoGPT-like wrapper would make it good at autonomously doing things, too.
oasisaimlessly 3y ago
Optimality is the tiger, and agents are its teeth: https://www.lesswrong.com/posts/kpPnReyBC54KESiSn/optimality-is-the-tiger-and-agents-are-its-teeth https://www.lesswrong.com/posts/kpPnReyBC54KESiSn/optimality...
- lukev 3y ago
  As someone who has worked/is working on building LLM driven agents, this allegory basically reads to me like: 1. I ask a LLM for paperclips. 2. ~Magic happens~. 3. Humanity is doomed. Even if a LLM was good enough to drive a system like this (current ones are not, by a long ways), there's a million reasons it would not and could not play out this way.
  - bondarchuk 3y ago
    I think (from the outside looking in) many lesswrongers are slowly coming around from the "paperclipper" scenario to something more gradual, in response to how the tech seems to be developing currently. More along the lines of "we hand over bigger and bigger parts of running the world to AI systems and this gradually leads to less and less human involvement (until such time as the intelligent systems that run the world see humans as an unnecessary interference that can be done away with)". Something like that at least. Some pointers here, maybe: https://astralcodexten.substack.com/p/davidson-on-takeoff-speeds https://astralcodexten.substack.com/p/davidson-on-takeoff-sp...
    - dbt00 3y ago
      I mean, to me this sounds like "some christians are coming around to the fact that cosmology is mostly correct about the shape of the universe, but they want us to say that God created the big bang since you can't explain it yet." Which is to say, right or wrong, it's a narrative in search of confirmation more than it's an actually rational understanding of what is real.
      - deleted 3y ago
        [deleted]

by lukev 3y ago

mquander 3y ago

OK, but the hard part was the "intelligent" part, right? If you have the "intelligent" thing, and you want to get an "intelligent agent", all you have to do at most is wrap it in a shell script that runs the equivalent of an OODA loop, e.g. AutoGPT. So we should expect a "superintelligent agent" to appear pretty much the next day after a "superintelligence" is created.

lukev 3y ago
Perhaps, I don't discount the possibility that an LLM could be used as a primitive as part of a larger more complex system that could rightfully be called intelligent. But a script running a LLM in an OODA loop is a lot easier to monitor and control than some inscrutable "ghost in the parameters" that a lot of AI alarmists seem worried about. You can actually see a transcript of its "thoughts", for one!
- Footkerchief 3y ago
  We already have self-replicating scripts that can prove very hard to detect and destroy, even without the benefit of LLM-powered fuzzing and social engineering.
  - lukev 3y ago
    First of all, we're assuming that agents are (at least initially) designed intentionally, not as harmful viruses. Second, running a LLM inference (nevermind training!) involves lots and lots of system resources. Much more difficult to hide than a tiny backdoor.
joe_the_user 3y ago
OK, but the hard part was the "intelligent" part, right? If you have the "intelligent" thing, and you want to get an "intelligent agent", all you have to do at most is wrap it in a shell script that runs the equivalent of an OODA loop, e.g. AutoGPT. No, not really. Neither AutoGPT or any similar actually work. And the only justification for expecting them to work the future is same naively extrapolation more of one kind of quality will automatically give a thing more of a correlated quality. The humming bird building airplanes: "If we continue to increase the capacity of these flying machines, eventually they will have 'super-bird abilities' and at that point they will naturally steal all of nectar..."
- mquander 3y ago
  AutoGPT doesn't work because GPT-4 is too stupid to make good plans, not because it's missing the secret sauce of "agentiness." If some future software was really great at observing, planning, and reasoning, then an AutoGPT-like wrapper would make it good at autonomously doing things, too.

oasisaimlessly 3y ago

Optimality is the tiger, and agents are its teeth: https://www.lesswrong.com/posts/kpPnReyBC54KESiSn/optimality-is-the-tiger-and-agents-are-its-teeth https://www.lesswrong.com/posts/kpPnReyBC54KESiSn/optimality...

lukev 3y ago
As someone who has worked/is working on building LLM driven agents, this allegory basically reads to me like: 1. I ask a LLM for paperclips. 2. ~Magic happens~. 3. Humanity is doomed. Even if a LLM was good enough to drive a system like this (current ones are not, by a long ways), there's a million reasons it would not and could not play out this way.
- bondarchuk 3y ago
  I think (from the outside looking in) many lesswrongers are slowly coming around from the "paperclipper" scenario to something more gradual, in response to how the tech seems to be developing currently. More along the lines of "we hand over bigger and bigger parts of running the world to AI systems and this gradually leads to less and less human involvement (until such time as the intelligent systems that run the world see humans as an unnecessary interference that can be done away with)". Something like that at least. Some pointers here, maybe: https://astralcodexten.substack.com/p/davidson-on-takeoff-speeds https://astralcodexten.substack.com/p/davidson-on-takeoff-sp...
  - dbt00 3y ago
    I mean, to me this sounds like "some christians are coming around to the fact that cosmology is mostly correct about the shape of the universe, but they want us to say that God created the big bang since you can't explain it yet." Which is to say, right or wrong, it's a narrative in search of confirmation more than it's an actually rational understanding of what is real.
    - deleted 3y ago
      [deleted]