4 ms·

Chess are very limited compared to the whole universe, AGI could for example invent self-replicating fusion-driven paperclip factories, there's nothing in the u

by ajuc 5y ago

Chess are very limited compared to the whole universe, AGI could for example invent self-replicating fusion-driven paperclip factories, there's nothing in the universe rules that makes this impossible we simply don't know how to do it (yet).

zwegner 5y ago
Or even without using any such new technology, it could at least take advantage of the fact that almost all human communication is mediated by computers, and coerce virtually the entire human race into helping it. I think the GP wasn't being nearly creative enough when thinking about the possibilities/dangers of AGI.
notahacker 5y ago
Sure, and God could smite all unbelievers, turn water into wine and bring about the Rapture: there are no limits to what purely hypothetical omniscient beings can do. But based on we actually know about the sort of AI systems humans actually know how to build and obtain useful results from, we'd end up with a computer which absorbed time series of paperclip sales figures, supply chain costs and marketing campaign data which recommends small paperclips for the Kazakh market, and there's no reason to believe a model trained to optimise paperclip production would know anything at all about how to generate fusion power. This article suggests we might even be able to analyse why the algorithm prioritised that obscure market after applying a lot more thought to the relevant data ourselves. (We can also reason that the AI model is more likely to be wrong about the small paperclips for the Kazakh market, because the permutations for the Kazakh market aren't as perfectly encapsulated in the underlying model and simulated by training on data as the permutations of chess)
- ajuc 5y ago
  The assumption was > suppose we invent some superhuman AGI for $100B in 2100 not > AI systems humans actually know how to build and obtain useful results from You're talking about specialized expert system that is possible today. BTW: AGI don't need to know anything about fusion power, just like Alpha Zero doesn't know anything about chess when it starts. Its supposed to learn the rules as it goes, and in case of AGI its chessboard is the universe. Chess are a pretty well balanced and stable environment, so there's no way to cheat, but when you give less well-defined environment to AI it routinely cheats. See for example AlphaStar working around the APM limit by "saving" APM for crucial fights and completely demolishing human players in these fights microing at 1000s actions per minute. Or evolutionary systems learning to walk by abusing numerical instability bugs in the physics engine. We know there are cheatcodes to reality (see industrial revolution), we know we haven't yet discovered all of them, and the assumption was AGI will be smarter than us. What you're doing is wishful thinking AGI won't find them cause it will be too busy with accounting.
  - Vetch 5y ago
    There is no evidence to suggest cheat codes in reality. The industrial revolution, your example, was a hard lesson against that. Attempts to cheat thermodynamic laws with perpetual motion machines were summarily shot down.
  - notahacker 5y ago
    Sure, but we're also talking about drawing inferences about AGI from how superintelligences based on ML processes actually behave rather than from stories. The current of the state of the art suggests an AGI might not need to know anything about fusion power to invent it, if it is equipped with a system to perfectly simulate the outcomes of experimenting with fusion power like Alpha Zero is with the games it plays. I'm not sure we can accidentally code one of those. The "cheat" argument suppports my argument, not yours. Rather than do something incredibly difficult and indirect like master advanced physics outside the scope of the simulations it can run to invent new power sources to legitimately produce very low cost paperclips, a system to maximise paperclip sales indicators is probably going to exploit gaps in the simulations it can run or data its already supplied with, like data entry errors in a region or logical holes in the supply chain pricing model, or the marketing simulation not including indications of languages spoken. So it might actually be worse than the management consultants at optimising paperclip revenues. On a related note, if you did want your AGI to invent fusion power but your fusion power emulation engine isn't up to scratch, it's likely to design reactors that are very performant inside the simulation but don't actually work, much like those evolutionary systems exploiting physics engine bugs when faced with the comparatively simple task of learning how to walk.

by ajuc 5y ago

zwegner 5y ago

Or even without using any such new technology, it could at least take advantage of the fact that almost all human communication is mediated by computers, and coerce virtually the entire human race into helping it. I think the GP wasn't being nearly creative enough when thinking about the possibilities/dangers of AGI.

notahacker 5y ago

Sure, and God could smite all unbelievers, turn water into wine and bring about the Rapture: there are no limits to what purely hypothetical omniscient beings can do. But based on we actually know about the sort of AI systems humans actually know how to build and obtain useful results from, we'd end up with a computer which absorbed time series of paperclip sales figures, supply chain costs and marketing campaign data which recommends small paperclips for the Kazakh market, and there's no reason to believe a model trained to optimise paperclip production would know anything at all about how to generate fusion power. This article suggests we might even be able to analyse why the algorithm prioritised that obscure market after applying a lot more thought to the relevant data ourselves. (We can also reason that the AI model is more likely to be wrong about the small paperclips for the Kazakh market, because the permutations for the Kazakh market aren't as perfectly encapsulated in the underlying model and simulated by training on data as the permutations of chess)

ajuc 5y ago
The assumption was > suppose we invent some superhuman AGI for $100B in 2100 not > AI systems humans actually know how to build and obtain useful results from You're talking about specialized expert system that is possible today. BTW: AGI don't need to know anything about fusion power, just like Alpha Zero doesn't know anything about chess when it starts. Its supposed to learn the rules as it goes, and in case of AGI its chessboard is the universe. Chess are a pretty well balanced and stable environment, so there's no way to cheat, but when you give less well-defined environment to AI it routinely cheats. See for example AlphaStar working around the APM limit by "saving" APM for crucial fights and completely demolishing human players in these fights microing at 1000s actions per minute. Or evolutionary systems learning to walk by abusing numerical instability bugs in the physics engine. We know there are cheatcodes to reality (see industrial revolution), we know we haven't yet discovered all of them, and the assumption was AGI will be smarter than us. What you're doing is wishful thinking AGI won't find them cause it will be too busy with accounting.
- Vetch 5y ago
  There is no evidence to suggest cheat codes in reality. The industrial revolution, your example, was a hard lesson against that. Attempts to cheat thermodynamic laws with perpetual motion machines were summarily shot down.
- notahacker 5y ago
  Sure, but we're also talking about drawing inferences about AGI from how superintelligences based on ML processes actually behave rather than from stories. The current of the state of the art suggests an AGI might not need to know anything about fusion power to invent it, if it is equipped with a system to perfectly simulate the outcomes of experimenting with fusion power like Alpha Zero is with the games it plays. I'm not sure we can accidentally code one of those. The "cheat" argument suppports my argument, not yours. Rather than do something incredibly difficult and indirect like master advanced physics outside the scope of the simulations it can run to invent new power sources to legitimately produce very low cost paperclips, a system to maximise paperclip sales indicators is probably going to exploit gaps in the simulations it can run or data its already supplied with, like data entry errors in a region or logical holes in the supply chain pricing model, or the marketing simulation not including indications of languages spoken. So it might actually be worse than the management consultants at optimising paperclip revenues. On a related note, if you did want your AGI to invent fusion power but your fusion power emulation engine isn't up to scratch, it's likely to design reactors that are very performant inside the simulation but don't actually work, much like those evolutionary systems exploiting physics engine bugs when faced with the comparatively simple task of learning how to walk.