3 ms·

Then the first thing the AGI will do is figure out ultra-high throughput analog computing for LLMs

londons_explore 3y ago
An AGI will likely be scared of even more powerful AGI's that it itself cannot control. It might take steps to try to prevent anyone building such a thing.
- Jensson 3y ago
  You assume consciousness here that can't be transferred to the new AI. Without that there is no reason why the AI should fear a more powerful AI. If it is just a dumb process like a flame, then the most efficient version prevails ie the smartest one, if it is conscious and that can be transferred then the AI just sees it as making itself more powerful.
  - ben_w 3y ago
    Given how many people here regard brain uploads as "not really cheating death", and that AI "can't have consciousness", I think a AGI with consciousness[0] would have a two different ways to have a reasonable chance of fearing that its existence would end with any attempt to transfer itself to a superior design. (Even more so if the architecture of the successor AI is very different, as that would be a bit like one of us being "uploaded" into an LLM via it watching us talk and write). An AGI without consciousness may well act like it has consciousness for the same reason VHS tapes playing on a TV do a good job of reproducing human facial expressions. An AI like this would still act scared of such things, because some of us act scared of such things. [0] it's not obvious to me that an AGI necessarily has to have consciousness
- _heimdall 3y ago
  Assuming that an AGI will have human-like emotions is likely a mistake. They may very well experience something like emotions that resemble human emotions, but they may not. Or they could have an entirely different set of emotions that we don't recognize. We simply don't know, just like we don't know what other species experience and can only guess at by observing their behavior and trying to map it back to what we know.

An AGI will likely be scared of even more powerful AGI's that it itself cannot control. It might take steps to try to prevent anyone building such a thing.

Jensson 3y ago
You assume consciousness here that can't be transferred to the new AI. Without that there is no reason why the AI should fear a more powerful AI. If it is just a dumb process like a flame, then the most efficient version prevails ie the smartest one, if it is conscious and that can be transferred then the AI just sees it as making itself more powerful.
- ben_w 3y ago
  Given how many people here regard brain uploads as "not really cheating death", and that AI "can't have consciousness", I think a AGI with consciousness[0] would have a two different ways to have a reasonable chance of fearing that its existence would end with any attempt to transfer itself to a superior design. (Even more so if the architecture of the successor AI is very different, as that would be a bit like one of us being "uploaded" into an LLM via it watching us talk and write). An AGI without consciousness may well act like it has consciousness for the same reason VHS tapes playing on a TV do a good job of reproducing human facial expressions. An AI like this would still act scared of such things, because some of us act scared of such things. [0] it's not obvious to me that an AGI necessarily has to have consciousness
_heimdall 3y ago
Assuming that an AGI will have human-like emotions is likely a mistake. They may very well experience something like emotions that resemble human emotions, but they may not. Or they could have an entirely different set of emotions that we don't recognize. We simply don't know, just like we don't know what other species experience and can only guess at by observing their behavior and trying to map it back to what we know.