4 ms·
The more likely AGI risk is indifference in pursuit of optimizing a reward function with values not aligned with us. Like the anthill destroyed by the construc
by gonehome 5y ago
The more likely AGI risk is indifference in pursuit of optimizing a reward function with values not aligned with us.
Like the anthill destroyed by the construction of a dam. The humans don’t hate the ants, aren’t putting them in zoos, they just don’t think of them in pursuit of their goals. In the AGI case it’s even possible for those goals to be “dumb” or pointless if we mess up the reward function or some unanticipated thing fits it (paper clips).
The risk with a self-improving AGI is that it is way smarter than us and it can think way faster. In my experience people have a hard time really visualizing this.
Imagine a human trying to trick a chimp (or an ant) into doing something so the human can pursue its goal. The gap between a human and an AGI would be even larger than that.
Imagine scaling up a regular human brain’s operations per second a billion times so it’s the same architecture but thinks that much faster (no longer constrained by bio energy or head size). Humans wouldn’t be able to respond because to the machine it’d be nearly like interacting with a static object, it’d be trivially able to think/act around you.
That’s why the research around AGI is about the goal alignment problem. Which is hard because humans are not perfectly aligned either. It’s also why “just unplug it” is dumb.
The optimistic outcome if the alignment problem is solved is enormous intellectual capacity to solve any problem we might have.
- nradov 5y agoThe more likely AGI risk is that we expend tremendous resources trying to develop it, only to find it doesn't work very well or do anything useful.
- gonehome 5y agoThere’s nothing that makes it physically impossible so it’s likely inevitable - just hard to know when. The risk of poorly aligned AGI as described is real, makes sense to try to work on the problem before we need it (when it’ll be too late).
- nradov 5y agoNo it's not a real risk, just a silly fantasy with no basis in scientific reality.
- gonehome 5y agoDismissive comments without any substance make you come across as dumb, not smart. If you have some actual position of substance, articulate it or just don’t bother replying at all.
- nradov 5y agoClaims about AGI without hard scientific backing make you come across as pseudo-intellectual, not smart. If you have actual evidence then articulate it or just don't bother replying at all. Claims made without evidence can be dismissed without evidence.
- gonehome 5y agoIf AGI is possible, then the described concerns about risk hold up. Either you think AGI is impossible for some reason or you think it is possible, but the goal alignment issues don’t matter. You haven’t given any explanation for either. Maybe you think something else? I wouldn’t know because you don’t say anything. Obviously there isn’t “hard scientific backing” since AGI doesn’t exist yet. There are still unknowns about what would be required. That doesn’t mean it’s impossible to think about what its existence could mean. It was possible to think about the consequences of flight before airplanes existed. I think AGI is possible because we’re surrounded by general intelligence and there’s nothing magical about brains or biology. It’s just the timeframe that’s unclear. We already see issues with poorly optimized reward functions on a smaller scale, both in ML models (classifying based on unintended things) and in humans (drug addiction).
- nradov 5y agoSo in other words you're just engaging in idle speculation.