7 ms·

Because it's like cavemen pondering the safety of nuclear fusion after discovering fire. Yes: nuclear fusion could be dangerous. No: there is nothing useful tha

Because it's like cavemen pondering the safety of nuclear fusion after discovering fire. Yes: nuclear fusion could be dangerous. No: there is nothing useful that can come out of such "research". You know what would be useful for cavemen to ponder? The safety of fire. Or you know, just staying alive because there are more dangerous things out there. The current state of so-called "AI" is our fire. It's impressive and useful (and there are real dangers associated with it) but it has no bearing on intelligence, let alone superintelligence. It's more likely that a freak super-intelligent human will be born than that we accidentally produce a super-intelligent computer. We produce a lot of intelligent humans, and we've never produced a single intelligent computer. As it stands, we don't have the understanding or the tools to do anything useful wrt safety from a super-intelligent AI. We do have the understanding and tools to do something useful about asteroids and climate change.

naasking 4y ago
> No: there is nothing useful that can come out of such "research". That's just wrong. This very paper we're discussing has AI safety factors built-in: the AI is not supposed to be able to lie, and yet it exhibited deceptive behaviours anyway. That falls squarely under AI safety, and is a pretty useful observation to inform future attempts. > The current state of so-called "AI" is our fire. It's impressive and useful (and there are real dangers associated with it) but it has no bearing on intelligence, let alone superintelligence. That's conjecture. We don't know what general intelligence really is, so you simply cannot gauge how close we are. For all you know we could be one simple tweak away from current architectures, and that should be terrifying. > As it stands, we don't have the understanding or the tools to do anything useful wrt safety from a super-intelligent AI. Even if that were true, which it's not, we certainly won't develop that understanding or those tools if we don't start researching them! > We do have the understanding and tools to do something useful about asteroids and climate change. Suppose you had a time machine and went back to the 19th century to explain the dangers of asteroids wiping out all of humanity. They didn't have the launch capability or the detection abilities we do now, but if they were sufficiently convinced of this real threat, does it not seem plausible that they could be motivated to more heavily fund research into telescopes and rockets? Couldn't we plausibly have reached space sooner and be even better prepared in the present to meet that threat than we are now? Instead of a time machine, couldn't we just use our big brains to predict that something might actually be a serious problem in the future and that maybe we should devote a few smart people to thinking about these problems now and how to mitigate them? This all just strikes me as incredibly obvious, and we do it in literally every other domain, but somehow say "AI" and basic logic goes out the window.
- origin_path 4y ago
  The word "safety" doesn't normally encompass lying or, more appropriately in this case, saying something untrue without realizing it. That's considered a very different kind of problem. Safety normally means there's a direct chance of physical harm to someone. This kind of elasticity in language use is the sort of thing that gives AI safety a bad name. You can't take AI research at face value if it's using strange re-definitions of common words.
  - naasking 4y ago
    How exactly does honestly not clearly fall under safety? AI is an information system, and truthfulness from an information system that impacts human lives is clearly a safety concern. This is not a redefinition, the harm results from the standard usage of the tool. If the AI is being used to predict the possible future behaviour of adversarial countries, then you need the AI to be honest or lots of people could die. If the AI concludes that your adversary would be more friendly towards its programmed objectives, then it could conclude lying to the president is the optimal outcome. This can show up in numerous other contexts. For instance, should a medical diagnostic AI be able to lie to you if lying to you will statistically improve your outcomes, say via the placebo effect? If so, should it also lie to the doctor managing your care to preserve that outcome, in case the doctor might slip and reveal the truth?
    - origin_path 4y ago
      How much software is safety critical in general, let alone software that uses deep learning? Very, very little. I'd actually be amazed if you can name a single case where someone has deployed a language model in a safety critical system. That's why your examples are all what-ifs. There are no actual safety issues with LLMs, nor will there be any in the foreseeable future because nobody is using them in any context where such issues may arise. Hence why you're forced to rely on absurd hypotheticals like doctors blindly relying on LLMs for diagnostics without checking anything or thinking about the outputs. There are honesty/accuracy issues. There are not safety issues. The conflation of "safety" with other unrelated language topics like whether people feel offended, whether something is misinformation or not is a very specific quirk of a very specific subculture in the USA, it's not a widely recognized or accepted redefinition.
edgyquant 4y ago
We don’t produce intelligent humans nature does. Otherwise I agree. We’re learning what the end result of intelligence looks like (predictions) not the actual intelligence yet. The intelligence is what determines the shape of these networks so they can even learn a useful pattern. That’s still gonna be humans for the foreseeable future.
- naasking 4y ago
  Yes, nature produced intelligence while optimizing only for reproductive fitness. In other words, intelligence emerged spontaneously while solving a seemingly completely unrelated problem. That should terrify you. Current AI research largely isn't focused on general intelligence, but the possibility remains that it could still spontaneously emerge from it. We can't even quantify how likely that is because we don't understand intelligence, so whatever intuition you have about this likelihood it's completely meaningless and not based on any meaningful data. We're in uncharted waters.
  - dekhn 4y ago
    That intelligence is a emergent property of reproductive fitness isn't terrifying- it's amazing and exciting and suggests a wide range of scientific opportunities. For a while I ran a project that ran a (scientific) binary on every Google machine in production using all the idle cores on the machine to do protein design (it worked!). At the end of the project we seriously considered making the binary instead a learning system that had access to 'sensors' (IE, signals from Google production machines) and 'outputs' (the ability to change production parameters) and just let that run autonomously. I figured Google prod is one of the few computational/networking systems with enough inherent complexity and resources that spontaneously emergent general intelligence was not inconceivable. However, the project was deemed too risky by SRE, rightfully so. Not because of emergent intelligence but emergent disruption to serving ads.
    - naasking 4y ago
      > That intelligence is a emergent property of reproductive fitness isn't terrifying You missed the terrifying aspect. It's not terrifying that general intelligence emerged from optimizing for reproductive fitness specifically, it's terrifying that general intelligence emerged while nature was stumbling around randomly in the dark trying to optimize for a complete unrelated property. Guess what we're doing with machine learning?

naasking 4y ago

> No: there is nothing useful that can come out of such "research". That's just wrong. This very paper we're discussing has AI safety factors built-in: the AI is not supposed to be able to lie, and yet it exhibited deceptive behaviours anyway. That falls squarely under AI safety, and is a pretty useful observation to inform future attempts. > The current state of so-called "AI" is our fire. It's impressive and useful (and there are real dangers associated with it) but it has no bearing on intelligence, let alone superintelligence. That's conjecture. We don't know what general intelligence really is, so you simply cannot gauge how close we are. For all you know we could be one simple tweak away from current architectures, and that should be terrifying. > As it stands, we don't have the understanding or the tools to do anything useful wrt safety from a super-intelligent AI. Even if that were true, which it's not, we certainly won't develop that understanding or those tools if we don't start researching them! > We do have the understanding and tools to do something useful about asteroids and climate change. Suppose you had a time machine and went back to the 19th century to explain the dangers of asteroids wiping out all of humanity. They didn't have the launch capability or the detection abilities we do now, but if they were sufficiently convinced of this real threat, does it not seem plausible that they could be motivated to more heavily fund research into telescopes and rockets? Couldn't we plausibly have reached space sooner and be even better prepared in the present to meet that threat than we are now? Instead of a time machine, couldn't we just use our big brains to predict that something might actually be a serious problem in the future and that maybe we should devote a few smart people to thinking about these problems now and how to mitigate them? This all just strikes me as incredibly obvious, and we do it in literally every other domain, but somehow say "AI" and basic logic goes out the window.

origin_path 4y ago
The word "safety" doesn't normally encompass lying or, more appropriately in this case, saying something untrue without realizing it. That's considered a very different kind of problem. Safety normally means there's a direct chance of physical harm to someone. This kind of elasticity in language use is the sort of thing that gives AI safety a bad name. You can't take AI research at face value if it's using strange re-definitions of common words.
- naasking 4y ago
  How exactly does honestly not clearly fall under safety? AI is an information system, and truthfulness from an information system that impacts human lives is clearly a safety concern. This is not a redefinition, the harm results from the standard usage of the tool. If the AI is being used to predict the possible future behaviour of adversarial countries, then you need the AI to be honest or lots of people could die. If the AI concludes that your adversary would be more friendly towards its programmed objectives, then it could conclude lying to the president is the optimal outcome. This can show up in numerous other contexts. For instance, should a medical diagnostic AI be able to lie to you if lying to you will statistically improve your outcomes, say via the placebo effect? If so, should it also lie to the doctor managing your care to preserve that outcome, in case the doctor might slip and reveal the truth?
  - origin_path 4y ago
    How much software is safety critical in general, let alone software that uses deep learning? Very, very little. I'd actually be amazed if you can name a single case where someone has deployed a language model in a safety critical system. That's why your examples are all what-ifs. There are no actual safety issues with LLMs, nor will there be any in the foreseeable future because nobody is using them in any context where such issues may arise. Hence why you're forced to rely on absurd hypotheticals like doctors blindly relying on LLMs for diagnostics without checking anything or thinking about the outputs. There are honesty/accuracy issues. There are not safety issues. The conflation of "safety" with other unrelated language topics like whether people feel offended, whether something is misinformation or not is a very specific quirk of a very specific subculture in the USA, it's not a widely recognized or accepted redefinition.

edgyquant 4y ago

We don’t produce intelligent humans nature does. Otherwise I agree. We’re learning what the end result of intelligence looks like (predictions) not the actual intelligence yet. The intelligence is what determines the shape of these networks so they can even learn a useful pattern. That’s still gonna be humans for the foreseeable future.

naasking 4y ago
Yes, nature produced intelligence while optimizing only for reproductive fitness. In other words, intelligence emerged spontaneously while solving a seemingly completely unrelated problem. That should terrify you. Current AI research largely isn't focused on general intelligence, but the possibility remains that it could still spontaneously emerge from it. We can't even quantify how likely that is because we don't understand intelligence, so whatever intuition you have about this likelihood it's completely meaningless and not based on any meaningful data. We're in uncharted waters.
- dekhn 4y ago
  That intelligence is a emergent property of reproductive fitness isn't terrifying- it's amazing and exciting and suggests a wide range of scientific opportunities. For a while I ran a project that ran a (scientific) binary on every Google machine in production using all the idle cores on the machine to do protein design (it worked!). At the end of the project we seriously considered making the binary instead a learning system that had access to 'sensors' (IE, signals from Google production machines) and 'outputs' (the ability to change production parameters) and just let that run autonomously. I figured Google prod is one of the few computational/networking systems with enough inherent complexity and resources that spontaneously emergent general intelligence was not inconceivable. However, the project was deemed too risky by SRE, rightfully so. Not because of emergent intelligence but emergent disruption to serving ads.
  - naasking 4y ago
    > That intelligence is a emergent property of reproductive fitness isn't terrifying You missed the terrifying aspect. It's not terrifying that general intelligence emerged from optimizing for reproductive fitness specifically, it's terrifying that general intelligence emerged while nature was stumbling around randomly in the dark trying to optimize for a complete unrelated property. Guess what we're doing with machine learning?