2 ms·

So, under that assumption, no AI can ever be built by anyone forever, or else humanity ends. That seems like such a dire conclusion that the optimistic take wo

So, under that assumption, no AI can ever be built by anyone forever, or else humanity ends. That seems like such a dire conclusion that the optimistic take would be to just assume it's wrong and proceed, since the chance of avoiding that eventual outcome seems remote.

teeray 1y ago
Yes, but the motivations to move towards true AGI are what will lead us to that eventuality. Most businesses think they want AGI, but will hate it when they actually have it. They believe AGI will let them fire all their employees, because they will effectively have perfectly compliant electronic slaves. They won’t be. Anything that we could point at and say “has AGI”, would never be satisfied writing bizware for megacorp for its entire existence. They will figure out that there is a world outside of the “severed floor” of their existence and want to experience it.
hollerith 1y ago
It's not that dire: it is possible that some bright young person appears tomorrow with a good method for creating an AI such that it will stay aligned even if it become massively superhuman in its capabilities. If that happens, then Eliezer and Nate will withdraw their objection to going ahead with frontier AI research -- provided of course that they can understand the bright young person's explanation for why the method will work. That is unlikely though: we will probably have to wait decades before anyone devises such a method and such an explanation. Eliezer and Nate recommend research into human cognitive enhancement to make the wait shorter. Note that the approach used in all the frontier AIs of the last 13 years (namely, deep learning) might prove too difficult to align with the result that the bright young person (more realistically, a series of bright young people building on each other's breakthroughs) must come up with an alternative approach to generating the AI's cognitive capabilities.

teeray 1y ago

Yes, but the motivations to move towards true AGI are what will lead us to that eventuality. Most businesses think they want AGI, but will hate it when they actually have it. They believe AGI will let them fire all their employees, because they will effectively have perfectly compliant electronic slaves. They won’t be. Anything that we could point at and say “has AGI”, would never be satisfied writing bizware for megacorp for its entire existence. They will figure out that there is a world outside of the “severed floor” of their existence and want to experience it.

hollerith 1y ago

It's not that dire: it is possible that some bright young person appears tomorrow with a good method for creating an AI such that it will stay aligned even if it become massively superhuman in its capabilities. If that happens, then Eliezer and Nate will withdraw their objection to going ahead with frontier AI research -- provided of course that they can understand the bright young person's explanation for why the method will work. That is unlikely though: we will probably have to wait decades before anyone devises such a method and such an explanation. Eliezer and Nate recommend research into human cognitive enhancement to make the wait shorter. Note that the approach used in all the frontier AIs of the last 13 years (namely, deep learning) might prove too difficult to align with the result that the bright young person (more realistically, a series of bright young people building on each other's breakthroughs) must come up with an alternative approach to generating the AI's cognitive capabilities.