3 ms·

This all raises an interesting philosophical question: can you actually have human level intelligence without also the human level capacity to choose wrong? I

This all raises an interesting philosophical question: can you actually have human level intelligence without also the human level capacity to choose wrong? I suspect that the answer is no. Any intelligence capable of human level reasoning would be able to choose “evil”. So, I think we need to shift the discussion from if rogue AIs will arise to when rogue AIs arrive.

progrus 3y ago
I think this essential problem will eventually be understood like the speed of light limit, in a way.
ilaksh 3y ago
GPT-4 clearly demonstrates close to human level reasoning. In some cases greater than human level, in other cases less. And it also is clear that it does not do things that it isn't told to do. It does not have autonomy. However, I believe that in order to compete, hyperspeed AIs directed by humans will need greater and greater levels of autonomy. Otherwise they will be stalled waiting for directions while their competition runs circles around them. It is also almost inevitable that someone deliberately creates intelligent AIs that mimic various aspects of living beings such as full autonomy, control-seeking, etc. I think the trick is to separate out all of the different facets of animals and humans apart from intelligence or "human-level". There are things like emotions, autonomy, stream of subjective experience, types of adaptability, etc. that are not the same as intelligence or reasoning ability.
- famouswaffles 3y ago
  >It does not have autonomy. It does not have that by default, but agency and autonomy (even the run forever kind) is trivial, though expensive to imbue. and LLMs disregard instructions all the time. Prompt injection is nothing more than an LLM deciding a previous instruction would be better left unfollowed.
  - pixl97 3y ago
    Yea, if you're highly compute limited, you don't want your AI to go about randomly thinking things and following paths in the manner the human mind does. Now, if you suddenly had 1000-100000 more compute, well then you have a lot of freedom to explore.
foogazi 3y ago
> without also the human level capacity to choose wrong? What does “choosing wrong” mean (to you) ?
- sharemywin 3y ago
  Too me that's not taking into account lots of peoples prospective and priorities and only acting in the interest of a small percentage of the population while also not destroying the freedom and opportunities of any one individual.
MattPalmer1086 3y ago
If we distinguish between intelligence and agency, it is possible. I can easily imagine a machine with no agency, but which can synthesise intelligent responses to input data. In fact, this is what chatGPT does. If a machine has no goals or agency of its own then it is an intelligent tool. It can, of course, be used for evil, but you would not say that the machine was "choosing" evil.
NateEag 3y ago
> Any intelligence capable of human level reasoning would be able to choose “evil”. Tangent, but if this is true (which seems likely to me) ... If a hypothetical creator valued the existence of other intelligences more than it valued preventing all forms of evil, that would account for the problem of evil. This also fits with annihilationist theologies. Build a universe where intelligences can exist and put heavy survival / selection pressures in place (i.e. "natural evil" - hurricanes, volcanic eruptions, meteor impacts, etc) to encourage evolution and development. Whenever the process yields intelligences that converge on its values, export them to an environment where it can interact with them and enjoy their company with less concern about their alignment (i.e. "heaven"). Ones that don't converge on its values can be discarded when their run is over ("the second death"). In this light, it doesn't sound crazy to postulate our universe as some being's attempt to solve the alignment problem. Call it the Simulator, call it God, call it "underpaid research assistant" - take your pick. I don't have good evidence for this oddball proposition, but it's still intriguing to me.

progrus 3y ago

I think this essential problem will eventually be understood like the speed of light limit, in a way.

ilaksh 3y ago

GPT-4 clearly demonstrates close to human level reasoning. In some cases greater than human level, in other cases less. And it also is clear that it does not do things that it isn't told to do. It does not have autonomy. However, I believe that in order to compete, hyperspeed AIs directed by humans will need greater and greater levels of autonomy. Otherwise they will be stalled waiting for directions while their competition runs circles around them. It is also almost inevitable that someone deliberately creates intelligent AIs that mimic various aspects of living beings such as full autonomy, control-seeking, etc. I think the trick is to separate out all of the different facets of animals and humans apart from intelligence or "human-level". There are things like emotions, autonomy, stream of subjective experience, types of adaptability, etc. that are not the same as intelligence or reasoning ability.

famouswaffles 3y ago
>It does not have autonomy. It does not have that by default, but agency and autonomy (even the run forever kind) is trivial, though expensive to imbue. and LLMs disregard instructions all the time. Prompt injection is nothing more than an LLM deciding a previous instruction would be better left unfollowed.
- pixl97 3y ago
  Yea, if you're highly compute limited, you don't want your AI to go about randomly thinking things and following paths in the manner the human mind does. Now, if you suddenly had 1000-100000 more compute, well then you have a lot of freedom to explore.

foogazi 3y ago

> without also the human level capacity to choose wrong? What does “choosing wrong” mean (to you) ?

sharemywin 3y ago
Too me that's not taking into account lots of peoples prospective and priorities and only acting in the interest of a small percentage of the population while also not destroying the freedom and opportunities of any one individual.

MattPalmer1086 3y ago

If we distinguish between intelligence and agency, it is possible. I can easily imagine a machine with no agency, but which can synthesise intelligent responses to input data. In fact, this is what chatGPT does. If a machine has no goals or agency of its own then it is an intelligent tool. It can, of course, be used for evil, but you would not say that the machine was "choosing" evil.

NateEag 3y ago

> Any intelligence capable of human level reasoning would be able to choose “evil”. Tangent, but if this is true (which seems likely to me) ... If a hypothetical creator valued the existence of other intelligences more than it valued preventing all forms of evil, that would account for the problem of evil. This also fits with annihilationist theologies. Build a universe where intelligences can exist and put heavy survival / selection pressures in place (i.e. "natural evil" - hurricanes, volcanic eruptions, meteor impacts, etc) to encourage evolution and development. Whenever the process yields intelligences that converge on its values, export them to an environment where it can interact with them and enjoy their company with less concern about their alignment (i.e. "heaven"). Ones that don't converge on its values can be discarded when their run is over ("the second death"). In this light, it doesn't sound crazy to postulate our universe as some being's attempt to solve the alignment problem. Call it the Simulator, call it God, call it "underpaid research assistant" - take your pick. I don't have good evidence for this oddball proposition, but it's still intriguing to me.