3 ms·

Agree, the path might be quite bumpy but so could the endgame. Whether AGI wants to help or kill the human race will depend on how aligned it is to our goals. N

Agree, the path might be quite bumpy but so could the endgame. Whether AGI wants to help or kill the human race will depend on how aligned it is to our goals. Now, since "our" goals will depend on whoever has the metaphysical ear of such AI this could be problematic.

visarga 3y ago
I think I read this kind of discussions too many times, imagine how the training set of GPT-5 will look like? all sorts of theories of "whether AGI wants to help or kill the human race" spread over the internet and analyzed to death. It is going to know this topic in-and-out. Will be able to write a masterful dissertation on the topic.
- ben_w 3y ago
  Yes, and that's good because GPT-5 is going to be coming from a team that cares about alignment and which has values only a little different from mine. They can still mess up: https://youtu.be/qV_rOlHjvvs?si=6SI_abC9BLHGoWjt https://youtu.be/qV_rOlHjvvs?si=6SI_abC9BLHGoWjt OTOH, an LLM trained by the FSB or CIA to help their agents, with no limits on what the agents are allowed to get as help because their bosses trust it's always in the service of the nation, will quite "happily" write that essay… and almost certainly get leaked eventually — 10 years, 50 years, doesn't matter — at which point someone who isn't so trustworthy will ask for a doomsday plan and it will be equally "happy" to write that too. Hopefully the agencies like the FSB and CIA aren't quite as bad as I fear, but secrecy and what little has become public makes it impossible to trust them.
nerpderp82 3y ago
AGI doesn't matter, because the world will have been turned upside down multiple times before it even arrives. AI due to its extreme breadth is already smarter than any individual human across all of AIs competencies.
Terr_ 3y ago
I feel that even in just the phrasing of the question, "wants" is doing an awful lot of heavy lifting.
Terr_ 3y ago
I feel that even in just the phrasing of the question, "wants" is doing an awful lot of heavy lifting. It's almost tied in a kind of "No True AGI" framing, where if it doesn't want something it's not really AGI, etc.

visarga 3y ago

I think I read this kind of discussions too many times, imagine how the training set of GPT-5 will look like? all sorts of theories of "whether AGI wants to help or kill the human race" spread over the internet and analyzed to death. It is going to know this topic in-and-out. Will be able to write a masterful dissertation on the topic.

ben_w 3y ago
Yes, and that's good because GPT-5 is going to be coming from a team that cares about alignment and which has values only a little different from mine. They can still mess up: https://youtu.be/qV_rOlHjvvs?si=6SI_abC9BLHGoWjt https://youtu.be/qV_rOlHjvvs?si=6SI_abC9BLHGoWjt OTOH, an LLM trained by the FSB or CIA to help their agents, with no limits on what the agents are allowed to get as help because their bosses trust it's always in the service of the nation, will quite "happily" write that essay… and almost certainly get leaked eventually — 10 years, 50 years, doesn't matter — at which point someone who isn't so trustworthy will ask for a doomsday plan and it will be equally "happy" to write that too. Hopefully the agencies like the FSB and CIA aren't quite as bad as I fear, but secrecy and what little has become public makes it impossible to trust them.

nerpderp82 3y ago

AGI doesn't matter, because the world will have been turned upside down multiple times before it even arrives. AI due to its extreme breadth is already smarter than any individual human across all of AIs competencies.

Terr_ 3y ago

I feel that even in just the phrasing of the question, "wants" is doing an awful lot of heavy lifting.

Terr_ 3y ago

I feel that even in just the phrasing of the question, "wants" is doing an awful lot of heavy lifting. It's almost tied in a kind of "No True AGI" framing, where if it doesn't want something it's not really AGI, etc.