2 ms·
> it is perfectly possible, I dare say plausible, that there exists a class of non-intelligent systems that no physically realizable intelligence can control I
by emtel 2y ago
> it is perfectly possible, I dare say plausible, that there exists a class of non-intelligent systems that no physically realizable intelligence can control
I am not sure what this means. It's almost certain that no physically realizable intelligence could compute exactly when to have a butterfly flap its wings in order to cause a hurricane next month, so the weather may be "uncontrollable", but so what? That probably won't be a binding constraint on any agent's ability to achieve its goals.
- breuleux 2y agoYes, sorry, that was a bit vague. What I mean is that there are probably classes of systems that consume resources too efficiently for intelligent agents to compete. Because an intelligent agent aims to "achieve goals," the space of actions they can undertake is constrained: if they need to do A in order to achieve goal B, they must do A in such a way that it does not undermine B. If they need to build paperclips to bind documents, it would be counterproductive to start an uncontrollable reaction that transforms all matter in the universe, including documents, into paperclips. So, insofar that intelligent agents must self-limit to controllable methods to extract resources to achieve their goals, it is quite possible that the optimal resource extraction system lies outside of that set. The "thing" that is the very best at consuming the resources that an intelligent agent wishes to consume may be something they would never dare to create, and if it was created, either through hubris or happenstance, the agent would be unable to control it and would therefore be unable to achieve their goals.