4 ms·

Yes, every point in favor of the possibility of AGI comes with an asterisk. That's not all that interesting. We need to be competent at reasoning under uncertai

Yes, every point in favor of the possibility of AGI comes with an asterisk. That's not all that interesting. We need to be competent at reasoning under uncertainty, something few people seem to be capable of. When the utility of a runaway AGI is infinitely negative, while the possibility of that outcome is substantially non-zero, rationality demands we act to prevent that outcome. >How would they force perfect alignment, though? In order to be effective, each of these individuals will need to work on different problems and focus on different information, which means they will start diverging I disagree that independence is required for effectiveness. Independence is useful, but it also comes with an inordinate coordination cost. Lack of independence implies low coordination costs, and the features of an artificial intelligence implies the ability to maximally utilize the abilities of the sub-components. Consider the 'thousand brains' hypothesis, that human intelligence is essentially the coordination of thousands of mini-brains. It stands to reason that the more powerful the mini-brains, along with the efficiency of coordination, implies a much more capable unified intelligence. Of course all that remains to be seen.

breuleux 3y ago
> Lack of independence implies low coordination costs Perhaps, but it's not obvious. Lack of independence implies more back-and-forth communication with the central coordinator, whereas independent agents could do more work before communication is required. It's a tradeoff. > the features of an artificial intelligence implies the ability to maximally utilize the abilities of the sub-components Does it? Can you elaborate? > It stands to reason that the more powerful the mini-brains, along with the efficiency of coordination, implies a much more capable unified intelligence. It also implies an easier alignment problem. If an intelligence can coordinate "mini-brains" fully reliably (a big if, by the way), presumably I can do something similar with a Python script or narrow AI. Decoupling capability from independence is ideal with respect to alignment, so I'm a bit less worried, if this is how it's going to work.
- hackinthebochs 3y ago
  >Does it? Can you elaborate? I don't intend to say anything controversial here. The consideration is the tradeoff between independence and tight constraints of the subcomponents. Independent entities have their own interests, as well as added computational and energetic costs involved in managing a whole entity. These are costs that can't be directed towards the overarching goal. On the other hand, tightly constrained components do not have this extra overhead and so their capacity can be fully directed towards the goal as determined by the control system. In terms of utilization of compute and energy towards the principle goal, a unified AI will be more efficient. >If an intelligence can coordinate "mini-brains" fully reliably (a big if, by the way), presumably I can do something similar with a Python script or narrow AI. This is plausible, and I'm totally in favor of exploiting narrow AI to maximal effect. If the only AI we ever had to worry about was narrow AI, I wouldn't have any issue aside from the mundane issues we get with the potential misuse of any new technology. But we know people (e.g. open AI) are explicitly aiming towards AGI so we need to be planning for this eventuality.

breuleux 3y ago

> Lack of independence implies low coordination costs Perhaps, but it's not obvious. Lack of independence implies more back-and-forth communication with the central coordinator, whereas independent agents could do more work before communication is required. It's a tradeoff. > the features of an artificial intelligence implies the ability to maximally utilize the abilities of the sub-components Does it? Can you elaborate? > It stands to reason that the more powerful the mini-brains, along with the efficiency of coordination, implies a much more capable unified intelligence. It also implies an easier alignment problem. If an intelligence can coordinate "mini-brains" fully reliably (a big if, by the way), presumably I can do something similar with a Python script or narrow AI. Decoupling capability from independence is ideal with respect to alignment, so I'm a bit less worried, if this is how it's going to work.

hackinthebochs 3y ago
>Does it? Can you elaborate? I don't intend to say anything controversial here. The consideration is the tradeoff between independence and tight constraints of the subcomponents. Independent entities have their own interests, as well as added computational and energetic costs involved in managing a whole entity. These are costs that can't be directed towards the overarching goal. On the other hand, tightly constrained components do not have this extra overhead and so their capacity can be fully directed towards the goal as determined by the control system. In terms of utilization of compute and energy towards the principle goal, a unified AI will be more efficient. >If an intelligence can coordinate "mini-brains" fully reliably (a big if, by the way), presumably I can do something similar with a Python script or narrow AI. This is plausible, and I'm totally in favor of exploiting narrow AI to maximal effect. If the only AI we ever had to worry about was narrow AI, I wouldn't have any issue aside from the mundane issues we get with the potential misuse of any new technology. But we know people (e.g. open AI) are explicitly aiming towards AGI so we need to be planning for this eventuality.