6 ms·

> rhyming-philosophical-nonsense problems like “alignment” He takes this shot in passing and it is a great summary of how much he misunderstands AI safety. For

> rhyming-philosophical-nonsense problems like “alignment” He takes this shot in passing and it is a great summary of how much he misunderstands AI safety. For many current researchers, alignment is a mathematical problem, not a philosophical one. The whole screed is against a set of strawman "pseudo-traits" that are not required for the alignment problem to be an existential problem. To be clearer: the "boring" engineering problems he admits to in the beginning have the potential to become exponentially more difficult as machine learning deployments become more powerful, to the point of becoming a human threat. No STIILTB required.

ryan93 4y ago
How can you encode “good” and “bad” in math? The alignment shit is complete nonsense. If super AI exists the only way to prevent bad stuff from happening is to not give it access to weapons.
- dinosaurdynasty 4y ago
  If a super AI exists and can talk to people it will get access to weapons if it so desires.
  - malux85 4y ago
    Or build its own, much more efficient, weapons When there’s large intelligence differentials in war, the lower intelligence doesn’t even know they are at war. (Human tank vs ant hill)
- ALittleLight 4y ago
  Alignment is more about the AI doing what you want and not good or evil. Probably not a good idea to reach a premature conclusion like "this is complete nonsense" before understanding the basics.
- more_corn 4y ago
  Everything is a weapon if you put enough power behind it.
- XorNot 4y ago
  One issue is a halting problem: would an AI system ever allow a conclusion which leads to it's own suspension of process be an acceptable outcome? This isn't abstract: we need certain management agents to ensure their own continuity so they won't switch themselves off idiotically, but that means granting a priority to various continuance of function weights. Balancing a system so it has "cease function" outcomes it will accept but doesn't always take immediately isn't easy - ML systems are notorious cheaters on metrics. You wouldn't want a nuclear power plant manager to not SCRAM the reactor because it predicts losing power to itself will fail a "maximize uptime" metric. This also gets more abstract as well: cease function is a problem in decision trees for AIs because it terminates the tree. The value is either infinite or 0, because every other path you can continue exploring and improving the summed weight of outcomes - but if you predict no future possible decisions, what weight do you assign that? It's a potentially infinite series of future reward weights versus 0.
- hackinthebochs 4y ago
  Good/bad, that is positive/negative valence, can be encoded into its evaluation function such that its value landscape aligns with ours. There's nothing nonsensical about it.
benreesman 4y ago
Animals get up and go do stuff mostly because they need food and sex and things in order to stick around, so we're surrounded by organisms like that. Imperatives like hunger and mating instinct create spontaneous action. The ability to sample from a modeled probability distribution is often useful in those pursuits, whether it's a model of the spatial world or of a corpus of pick-up lines that work well on a Friday evening, and the hyper-scaled transformers have more than demonstrated that kind of modeling and sampling. Take it a step further and you've got AlphaZero or something: it's sampling from a modeled distribution of moves that win games. But there's still a `while`-loop somewhere saying: time to sample a move. That part is not novel or mysterious. There is no demonstrated technology that I'm aware of where the "alignment" needs to be with the big model: the alignment needs to be with the person writing the `while`-loop. If someone has a machine repeatedly sample from a distribution of moves that DESTROYS_ALL_HUMANS_BEEP_BOOP, then your beef is with that person, not the model. Now if someone trains an RL agent with a loss around chasing sex or fame or fortune, we might have an issue. But that's still sci-fi stuff AFAIK, and it's a little hard to take the mix of real stuff and sci-fi that passes for much-if-not-most "AI Safety" discussion seriously, especially when you consider that the `while`-loop authors stand to gain a great deal by focusing attention on the modeling part.
- iratewizard 4y ago
  It's more interesting for many to believe that AI could be so powerful that we run into fantastical problems like what you described. That's the only reason I've been able to come up with to explain why people think deeply fake looking deep fakes are approaching Azimovian levels.

ryan93 4y ago

How can you encode “good” and “bad” in math? The alignment shit is complete nonsense. If super AI exists the only way to prevent bad stuff from happening is to not give it access to weapons.

dinosaurdynasty 4y ago
If a super AI exists and can talk to people it will get access to weapons if it so desires.
- malux85 4y ago
  Or build its own, much more efficient, weapons When there’s large intelligence differentials in war, the lower intelligence doesn’t even know they are at war. (Human tank vs ant hill)
ALittleLight 4y ago
Alignment is more about the AI doing what you want and not good or evil. Probably not a good idea to reach a premature conclusion like "this is complete nonsense" before understanding the basics.
more_corn 4y ago
Everything is a weapon if you put enough power behind it.
XorNot 4y ago
One issue is a halting problem: would an AI system ever allow a conclusion which leads to it's own suspension of process be an acceptable outcome? This isn't abstract: we need certain management agents to ensure their own continuity so they won't switch themselves off idiotically, but that means granting a priority to various continuance of function weights. Balancing a system so it has "cease function" outcomes it will accept but doesn't always take immediately isn't easy - ML systems are notorious cheaters on metrics. You wouldn't want a nuclear power plant manager to not SCRAM the reactor because it predicts losing power to itself will fail a "maximize uptime" metric. This also gets more abstract as well: cease function is a problem in decision trees for AIs because it terminates the tree. The value is either infinite or 0, because every other path you can continue exploring and improving the summed weight of outcomes - but if you predict no future possible decisions, what weight do you assign that? It's a potentially infinite series of future reward weights versus 0.
hackinthebochs 4y ago
Good/bad, that is positive/negative valence, can be encoded into its evaluation function such that its value landscape aligns with ours. There's nothing nonsensical about it.

benreesman 4y ago

Animals get up and go do stuff mostly because they need food and sex and things in order to stick around, so we're surrounded by organisms like that. Imperatives like hunger and mating instinct create spontaneous action. The ability to sample from a modeled probability distribution is often useful in those pursuits, whether it's a model of the spatial world or of a corpus of pick-up lines that work well on a Friday evening, and the hyper-scaled transformers have more than demonstrated that kind of modeling and sampling. Take it a step further and you've got AlphaZero or something: it's sampling from a modeled distribution of moves that win games. But there's still a `while`-loop somewhere saying: time to sample a move. That part is not novel or mysterious. There is no demonstrated technology that I'm aware of where the "alignment" needs to be with the big model: the alignment needs to be with the person writing the `while`-loop. If someone has a machine repeatedly sample from a distribution of moves that DESTROYS_ALL_HUMANS_BEEP_BOOP, then your beef is with that person, not the model. Now if someone trains an RL agent with a loss around chasing sex or fame or fortune, we might have an issue. But that's still sci-fi stuff AFAIK, and it's a little hard to take the mix of real stuff and sci-fi that passes for much-if-not-most "AI Safety" discussion seriously, especially when you consider that the `while`-loop authors stand to gain a great deal by focusing attention on the modeling part.

iratewizard 4y ago
It's more interesting for many to believe that AI could be so powerful that we run into fantastical problems like what you described. That's the only reason I've been able to come up with to explain why people think deeply fake looking deep fakes are approaching Azimovian levels.