10 ms·

> It’s hard to take these safety concerns seriously I don't get this mindset at all. How can it not be obvious to you that AI is an uniquely powerful and thus

by patrec 3y ago

> It’s hard to take these safety concerns seriously I don't get this mindset at all. How can it not be obvious to you that AI is an uniquely powerful and thus uniquely dangerous technology? It's like saying nuclear missiles can't possibly be dangerous and nuclear arms reduction and non-proliferation treaties were a scam, because the US, China and the Soviet Union had positioned themselves to capture the majority of the strategic value nukes bring.

nicce 3y ago
If you look the the world politics, basically if you hold enough nuclear weapons, you can do whatever you want to those who don't have them. And based on the "dangers", new countries are prohibit to create them. And the countries which were quick enough to create them, holds all the power. Their value is immeasurable especially for the Russia. Without them, they could not attack to Ukraine. > non-proliferation treaties were a scam And yes, they mostly are right now. Russia has backed from them. There are no real consequences if you are backing off, and you can do it in any time. The parent commenter is most likely saying, that now the selected parties hold the power of AI, they want to prevent others to gain similar power, while maintaining all the value by themselves.
- staunton 3y ago
  > There are no real consequences if you are backing off, and you can do it in any time. That's not quite true. Sure, noone is going to start a war about such a withdrawal. However, nuclear arsenals are expensive to maintain and it's even more expensive to be in an arms race. Also, nobody wants to risk nuclear war if they can avoid it. Civilian populations will support disarmament in times where they don't feel directly threatened. That's why lot of leaders of all persuasions have advocated for and taken part in efforts to reduce their arsenals. Same goes for relations between countries generally and the huge economic benefits that come with trade and cooperation. Withdrawing from nuclear treaties endangers all of these benefits and increases risk. A country would only choose this route out of desperation or for likely immediate gain.
  - nicce 3y ago
    I think it really depends. E.g. from the Western perspective, only US, UK, France, Russia and China have signed the treaty from nuclear countries. India or Pakistan are not part of the treaties and for some reason, we don't see big problems. There is only China left who might leave the treaty in the first place, anymore. And we are so dependent of the China, that there is no guarantee for consequences. Should we treat China then equally than India? What that means? Also, leaving the treaty does not mean that countries start massively increasing their arsenal. There will be just a lack of inspections and information exchange.
randomdata 3y ago
Nuclear missiles present an obvious danger to the human body. AI is an application of math. It is not clear how that can be used directly to harm a body. The assumption seems to be that said math will be coupled with something like a nuclear missile, but in that case the nuclear missile is still the threat. Any use of AI is just an implementation detail.
- pixl97 3y ago
  Germany, for example would disagree with you. They believe violent speech is an act of violence in itself. >AI is an application of math. It turns out that people hook computers to 'things' that exist in the physical world. You know like robot bodies, or 3D printers. And as mentioned above, even virtual things like social media can cause enough problems. People hook AI to tools. And this is just the maybe not quite general AI we have now. If and when we create a general AI that with self-changing feedback loops then all this "AI is just a tool" asshattery goes out the window. Remember at the end of the day, you're just an application of chemistry that is really weak without your ability to use tools and to communicate.
  - randomdata 3y ago
    > It turns out that people hook computers to 'things' that exist in the physical world. But those physical things would be the danger, at least if you consider the nuclear missile to be the danger. It seems you are trying to go down the "guns don't kill people, people kill people" line of thinking. Which is fine, but outside of the discussion taking place.
    - rahmeero 3y ago
      There are many relevant things that already exist in the physical world and are not currently considered dangers: ecommerce, digital payments, doordash-style delivery, cross-border remittances, remote gig work, social media fanning extreme political views, event organizing. However, these are constituent elements that could be aggregated and weaponized by a maleficent AI.
valine 3y ago
It's not clear at all that we have an avenue to super intelligence. I think the most likely outcome is that we hit a local maximum with our current architectures and end up with helpful assistants similar in capability to George Lucas's C3PO. The scary doomsday scenarios aren't possible without an AI that's capable of both strategic thinking and long term planning. Those two things also happen to be the biggest limitations of our most powerful language models. We simply don't know how to build a system like that.
- pixl97 3y ago
  >It's not clear at all that we have an avenue to super intelligence. All problems in reality are probability problems. If we don't have a path to superintelligence, then the worst problems just don't manifest themselves. If we do have a path to super intelligence then the doomsday scenarios are nearly a certainty. It's not really any different than saying "A supervolcano is unlikely to go off tomorrow, but if a supervolcano does go off tomorrow it is a doomsday scenario". >We simply don't know how to build a system like that. You are already a superintelligence when compared to all other intelligences on earth. Evolution didn't need to know how to build a system like that, and yet it still reached this point. And there is not really any to believe humanity is the pinnacle of intelligence, we are our own local maxima of power/communication limitations. An intelligence coupled with evolutionary systems design is much more apt to create 'super-' anything than the random walk alone.
  - RandomLensman 3y ago
    Why are doomsday scenarios are certainty then. What's the model to get to that that isn't just some sort of scary story that waves away or into existence a lot of things we don't know if they can exist.
    - pixl97 3y ago
      >What's the model to get to that Let's say I was a small furry mammal that tasted really good, but also for some reason understood the world as it is now. I would tell you that super intelligence had already happened. That super intelligence was humans. That humans happened to reach super intelligence by 1) having the proper hardware. 2) filtering noise from important information. 3) then sharing that information with others to amplify the power of intelligence 4) having a toolkit/tools to turn that information into useful things. 5) And with all that power humans can kill me off in mass, or farm me for my tasty meat at their leisure with little to nothing that I can do about it. There doesn't appear to be any more magic than that. All these things already exist in biological systems that elevated humans far above their warm blooded peers. When we look at digital systems we see they are designed to communicate. You don't have an ethernet jack as a person. You can't speak the protocol to directly drive a 3 axis mill to produce something. Writing computer code is a pain in the ass to most of us. We are developing a universal communication intelligence, that at least in theory can drive tools at a much higher efficiency than humans will ever be able to. Coming back to point 5. Cats/dogs are the real smart ones here when dealing with superintelligences. Get domesticated by the intelligence so they want to keep you around as a pet.
deleted 3y ago
[deleted]
falsaberN1 3y ago
And I don't get the opposed mindset, that AI is suddenly going to "become a real boy, and murder us all". Isn't it a funny coincidence how the popular opinion of AIs aligns perfectly with blockbusters and popular media ONLY? People are specifically wanting to prevent Skynet. The kicker (and irony to a degree) is that I really want sapient AI to exist. People being so influenced by fiction is something I see as a menace to that happening in my lifetime. I live in a world where the majority is apparently Don Quixote. - Point one: If the sentient AI can launch nukes, so can your neighbor. - Point zwei: Redistributing itself online to have unlimited compute resources is a fun scenario but if networks were that good then Stadia wouldn't have been a huge failure. - Point trois: A distributed-to-all-computers AI must have figured out universal executables. Once we deal with the nuclear winter, we can plagiarize it for ourselves. No more appimage/snap/flatpak discussions! Works for any hardware! No more dependency issues! Works on CentOS and Windows from 1.0 to 11! (it's also on AUR, of course.) - Point cuatro: The rogue AI is clearly born as a master hacker capable of finding your open ports, figure out any exploits or create 0-day exploits to get in, and hope there's enough resources to get the payload injected, then pray no competent admin is looking at the thing. - Point go: All of this rides on the assumption that the "cold, calculating" AI has the emotional maturity of a teenager. Wait, but that's not what "cold, calculating" means, that's "hothead and emotional". Which is it? - Point six: Skynet lost, that's the point of the first movie's plot. If everyone is going to base their beliefs after a movie, at least get all the details. Everything Skynet did after the first attack was full of boneheaded decisions that only made the situation worse for it, to the point the writers cannot figure ways to bring Skynet back anymore because it doomed itself in the very first movie. You should be worrying about Legion now, I think. It shuts down our electronics instead of nuking. Considering it won't have the advantage of triggering a nuclear attack because that's not how nukes work, the evil sentient AI is so doomed to fail it's ridiculous to think otherwise. But, companies know this is how the public works. They'll milk it for all it's worth so only a few companies can run or develop AIs, maybe making it illegal otherwise, or liable for DMCAs. Smart business move, but it affects my ability to research and use them. I cannot cure people's ability to separate reality and fiction though, and that's unfortunate.
- boringuser2 3y ago
  Very Dunning-Kruger post right here.
brookst 3y ago
> How can it not be obvious You have succinctly and completely summed up the AI risk argument more eloquently than anyone I've seen before. "How can it not be obvious?" Everything else is just intellectual fig leaves for the core argument that intuitively, without evidence, this proposition is obvious. The problem is, lots of "obvious" things have turned out to be very wrong. Sometimes relatively harmlessly, like the obviousness of the sun revolving around the earth, and sometimes catastrophically, like the obviousness of one race being inherently inferior. We should be very suspicious of policy that is based on propositions so obvious that it's borderline offensive to question them.
- computerphage 3y ago
  > borderline offensive to question them I would be happy to politely discuss any proposition regarding AI Risk. I don't think any claim should go unquestioned. I can also point you to much longer-form discussions. For example, this post, which has 670 comments, discussing various aspects of the argument: https://www.lesswrong.com/posts/uMQ3cqWDPHhjtiesc/agi-ruin-a-list-of-lethalities https://www.lesswrong.com/posts/uMQ3cqWDPHhjtiesc/agi-ruin-a...
  - frankzinger 3y ago
    That link is about the risks of AGI, which doesn't exist, and there's no reason to believe that it ever will exist. (If I'm wrong about AGI then I'm open to being convinced, but that's a different conversation as the topic here is non-general AI, is it not?)
    - computerphage 3y ago
      I disagree that there's no reason to believe it will ever exist. For one thing, many smart people are trying to build the technology right now and they believe it to be possible. I see no compelling case that the intelligence scale simply tops-out with humans; that a more intelligent system is ruled out by the laws of physics. The topic here is human extinction caused by AI. I don't know of any serious argument for why a non-general intelligence (really a system less intelligent than a human) would pose an extinction risk to humanity. Plus, my background understanding of the people who signed this is that they're worried about AGI, not present-day systems, but that's an inference.
Veen 3y ago
It is possible to believe that AI poses threat, while also thinking that the AI safety organizations currently sprouting up are essentially grifts that will do absolutely nothing to combat the genuine threat. Especially when their primary goal seems to be the creation of well-funded sinecures for a group of like-minded, ideologically aligned individuals who want to limit AI control to a small group of wealthy technologists.
- patrec 3y ago
  I agree. But as you can see yourself, there are countless people even here, in a technical forum, who claim that AI poses no plausible threat whatsoever. I fail to see how one can reasonably believe that.
patch_cable 3y ago
> How can it not be obvious to you It isn't obvious to me. And I've yet to read something that spills out the obvious reasoning. I feel like everything I've read just spells out some contrived scenario, and then when folks push back explaining all the reasons that particular scenario wouldn't come to pass, the counter argument is just "but that's just one example!" without offering anything more convincing. Do you have any better resources that you could share?
- hackinthebochs 3y ago
  The history of humanity is replete with examples of the slightly more technologically advanced group decimating their competition. The default position should be that uneven advantage is extremely dangerous to those disadvantaged. This idea that an intelligence significantly greater than our own is benign just doesn't pass the smell test. From the tech perspective: higher order objectives are insidious. While we may assume a narrow misalignment in received vs intended objective of a higher order nature, this misalignment can result in very divergent first-order behavior. Misalignment in behavior is by its nature destructive of value. The question is how much destruction of value can we expect? The machine may intentionally act in destructive ways as it goes about carrying out its slightly misaligned higher order objective-guided behavior. Of course we will have first-order rules that constrain its behavior. But again, slight misalignment in first-order rule descriptions are avenues for exploitation. If we cannot be sure we have zero exploitable rules, we must assume a superintelligence will find such loopholes and exploit them to maximum effect. Human history since we started using technology has been a lesson on the outcome of an intelligent entity aimed at realizing an objective. Loopholes are just resources to be exploited. The destruction of the environment and other humans is just the inevitable outcome of slight misalignment of an intelligent optimizer. If this argument is right, the only thing standing between us and destruction is the AGI having reached its objective before it eats the world. That is, there will always be some value lost in any significant execution of an AGI agent due to misalignment. Can we prove that the ratio of value created to value lost due to misalignment is always above some suitable threshold? Until we do, x-risk should be the default assumption.
- patrec 3y ago
  OK, which of the following propositions do you disagree with? 1. AIs have made rapid progress in approaching and often surpassing human abilities in many areas. 2. The fact that AIs have some inherent scalability, speed, cost, reliability and compliance advantages over humans means that many undesirable things that could previously not be done at all or at least not done at scale are becoming both feasible and cost-effective. Examples would include 24/7 surveillance with social desirability scoring based on a precise ideological and psychological profile derived from a comprehensive record of interactions, fine-tuned mass manipulation and large scale plausible falsification of the historical record. Given the general rise of authoritarianism, this is pretty worrying. 3. On the other hand the rapid progress and enormous investment we've been seeing makes it very plausible that before too long we will, in fact, see AIs that outperform humans on most tasks. 4. AIs that are much smarter than any human pose even graver dangers. 5. Even if there is a general agreement that AIs pose grave or even existential risks, states, organizations and individuals will are all incentivized to still seek to improve their own AI capabilities, as doing so provides an enormous competitive advantage. 6. There is a danger of a rapid self-improvement feedback loop. Humans can reproduce, learn new and significantly improve existing skills, as well as pass skills on to others via teaching. But there are fundamental limits on speed and scale for all of these, whereas it's not obvious at all how an AI that has reached super-human level intelligence would be fundamentally prevented from rapidly improving itself further, or produce millions of "offspring" that can collaborate and skill-exchange extremely efficiently. Furthermore, since AIs can operate at completely different time scales than humans, this all could happen extremely rapidly, and such a system might very quickly become much more powerful than humanity and the rest of AIs combined. I think you only have to subscribe a small subset of these (say 1.&2.) to conclude that "AI is an uniquely powerful and thus uniquely dangerous technology" obviously follows. For the stronger claim of existential risk, have you read the lesswrong link posted elsewhere in this discussion? https://www.lesswrong.com/posts/uMQ3cqWDPHhjtiesc/agi-ruin-a-list-of-lethalities https://www.lesswrong.com/posts/uMQ3cqWDPHhjtiesc/agi-ruin-a... ?
api 3y ago
Most of the credible threats I see from AI that don't rely on a lot of sci-fi extrapolation involve small groups of humans in control of massively powerful AI using it as a force multiplier to control or attack other groups of humans. Sam Altman's proposal is to create precisely that situation with himself and a few other large oligarchs being the ones in control of the leading edge of AI. If we really do face runaway intelligence growth and god-like AIs then this is a profound amount of power to place in the hands of just a few people. Even worse it opens the possibility that such developments could happen partly in secret, so the public might not even know how powerful the secret AIs under command of the oligarchs have become. The analogy with nuclear weapons is profoundly broken in lots of ways. Reasoning from a sloppy analogy is a great way to end up somewhere stupid. AI is a unique technology with a unique set of risks and benefits and a unique profile.
csomar 3y ago
> It's like saying nuclear missiles can't possibly be dangerous and nuclear arms reduction and non-proliferation treaties were a scam, because the US, China and the Soviet Union had positioned themselves to capture the majority of the strategic value nukes bring. I'm honestly not sure if this is sarcasm. The non-proliferation treaties are indeed a scam. The war is raging between the US and Russia and nuclear is a big part of it (though just words/threats for now). It's nonsensical to think that these treaties are possible.
- patrec 3y ago
  Not only is the Non proliferation treaty possible, it's been evidently effective in slowing the proliferation of nuclear weapons. The only country that ratified or acceded to it and went on to develop nuclear weapons is North Korea, and the only country that ratified or acceded to it and looks on track to develop nuclear weapons is Iran. One country that was not a signatory and developed nuclear weapons voluntarily gave them up and acceded to it partly due to international pressure (South Africa). Israel, Pakistan and India have since developed nuclear weapons but they were never signatories, the only other non-signatory is South Sudan which probably won't acquire nuclear capabilities anytime soon.

by patrec 3y ago

nicce 3y ago

If you look the the world politics, basically if you hold enough nuclear weapons, you can do whatever you want to those who don't have them. And based on the "dangers", new countries are prohibit to create them. And the countries which were quick enough to create them, holds all the power. Their value is immeasurable especially for the Russia. Without them, they could not attack to Ukraine. > non-proliferation treaties were a scam And yes, they mostly are right now. Russia has backed from them. There are no real consequences if you are backing off, and you can do it in any time. The parent commenter is most likely saying, that now the selected parties hold the power of AI, they want to prevent others to gain similar power, while maintaining all the value by themselves.

staunton 3y ago
> There are no real consequences if you are backing off, and you can do it in any time. That's not quite true. Sure, noone is going to start a war about such a withdrawal. However, nuclear arsenals are expensive to maintain and it's even more expensive to be in an arms race. Also, nobody wants to risk nuclear war if they can avoid it. Civilian populations will support disarmament in times where they don't feel directly threatened. That's why lot of leaders of all persuasions have advocated for and taken part in efforts to reduce their arsenals. Same goes for relations between countries generally and the huge economic benefits that come with trade and cooperation. Withdrawing from nuclear treaties endangers all of these benefits and increases risk. A country would only choose this route out of desperation or for likely immediate gain.
- nicce 3y ago
  I think it really depends. E.g. from the Western perspective, only US, UK, France, Russia and China have signed the treaty from nuclear countries. India or Pakistan are not part of the treaties and for some reason, we don't see big problems. There is only China left who might leave the treaty in the first place, anymore. And we are so dependent of the China, that there is no guarantee for consequences. Should we treat China then equally than India? What that means? Also, leaving the treaty does not mean that countries start massively increasing their arsenal. There will be just a lack of inspections and information exchange.

randomdata 3y ago

Nuclear missiles present an obvious danger to the human body. AI is an application of math. It is not clear how that can be used directly to harm a body. The assumption seems to be that said math will be coupled with something like a nuclear missile, but in that case the nuclear missile is still the threat. Any use of AI is just an implementation detail.

pixl97 3y ago
Germany, for example would disagree with you. They believe violent speech is an act of violence in itself. >AI is an application of math. It turns out that people hook computers to 'things' that exist in the physical world. You know like robot bodies, or 3D printers. And as mentioned above, even virtual things like social media can cause enough problems. People hook AI to tools. And this is just the maybe not quite general AI we have now. If and when we create a general AI that with self-changing feedback loops then all this "AI is just a tool" asshattery goes out the window. Remember at the end of the day, you're just an application of chemistry that is really weak without your ability to use tools and to communicate.
- randomdata 3y ago
  > It turns out that people hook computers to 'things' that exist in the physical world. But those physical things would be the danger, at least if you consider the nuclear missile to be the danger. It seems you are trying to go down the "guns don't kill people, people kill people" line of thinking. Which is fine, but outside of the discussion taking place.
  - rahmeero 3y ago
    There are many relevant things that already exist in the physical world and are not currently considered dangers: ecommerce, digital payments, doordash-style delivery, cross-border remittances, remote gig work, social media fanning extreme political views, event organizing. However, these are constituent elements that could be aggregated and weaponized by a maleficent AI.

valine 3y ago

It's not clear at all that we have an avenue to super intelligence. I think the most likely outcome is that we hit a local maximum with our current architectures and end up with helpful assistants similar in capability to George Lucas's C3PO. The scary doomsday scenarios aren't possible without an AI that's capable of both strategic thinking and long term planning. Those two things also happen to be the biggest limitations of our most powerful language models. We simply don't know how to build a system like that.

pixl97 3y ago
>It's not clear at all that we have an avenue to super intelligence. All problems in reality are probability problems. If we don't have a path to superintelligence, then the worst problems just don't manifest themselves. If we do have a path to super intelligence then the doomsday scenarios are nearly a certainty. It's not really any different than saying "A supervolcano is unlikely to go off tomorrow, but if a supervolcano does go off tomorrow it is a doomsday scenario". >We simply don't know how to build a system like that. You are already a superintelligence when compared to all other intelligences on earth. Evolution didn't need to know how to build a system like that, and yet it still reached this point. And there is not really any to believe humanity is the pinnacle of intelligence, we are our own local maxima of power/communication limitations. An intelligence coupled with evolutionary systems design is much more apt to create 'super-' anything than the random walk alone.
- RandomLensman 3y ago
  Why are doomsday scenarios are certainty then. What's the model to get to that that isn't just some sort of scary story that waves away or into existence a lot of things we don't know if they can exist.
  - pixl97 3y ago
    >What's the model to get to that Let's say I was a small furry mammal that tasted really good, but also for some reason understood the world as it is now. I would tell you that super intelligence had already happened. That super intelligence was humans. That humans happened to reach super intelligence by 1) having the proper hardware. 2) filtering noise from important information. 3) then sharing that information with others to amplify the power of intelligence 4) having a toolkit/tools to turn that information into useful things. 5) And with all that power humans can kill me off in mass, or farm me for my tasty meat at their leisure with little to nothing that I can do about it. There doesn't appear to be any more magic than that. All these things already exist in biological systems that elevated humans far above their warm blooded peers. When we look at digital systems we see they are designed to communicate. You don't have an ethernet jack as a person. You can't speak the protocol to directly drive a 3 axis mill to produce something. Writing computer code is a pain in the ass to most of us. We are developing a universal communication intelligence, that at least in theory can drive tools at a much higher efficiency than humans will ever be able to. Coming back to point 5. Cats/dogs are the real smart ones here when dealing with superintelligences. Get domesticated by the intelligence so they want to keep you around as a pet.

deleted 3y ago

[deleted]

falsaberN1 3y ago

And I don't get the opposed mindset, that AI is suddenly going to "become a real boy, and murder us all". Isn't it a funny coincidence how the popular opinion of AIs aligns perfectly with blockbusters and popular media ONLY? People are specifically wanting to prevent Skynet. The kicker (and irony to a degree) is that I really want sapient AI to exist. People being so influenced by fiction is something I see as a menace to that happening in my lifetime. I live in a world where the majority is apparently Don Quixote. - Point one: If the sentient AI can launch nukes, so can your neighbor. - Point zwei: Redistributing itself online to have unlimited compute resources is a fun scenario but if networks were that good then Stadia wouldn't have been a huge failure. - Point trois: A distributed-to-all-computers AI must have figured out universal executables. Once we deal with the nuclear winter, we can plagiarize it for ourselves. No more appimage/snap/flatpak discussions! Works for any hardware! No more dependency issues! Works on CentOS and Windows from 1.0 to 11! (it's also on AUR, of course.) - Point cuatro: The rogue AI is clearly born as a master hacker capable of finding your open ports, figure out any exploits or create 0-day exploits to get in, and hope there's enough resources to get the payload injected, then pray no competent admin is looking at the thing. - Point go: All of this rides on the assumption that the "cold, calculating" AI has the emotional maturity of a teenager. Wait, but that's not what "cold, calculating" means, that's "hothead and emotional". Which is it? - Point six: Skynet lost, that's the point of the first movie's plot. If everyone is going to base their beliefs after a movie, at least get all the details. Everything Skynet did after the first attack was full of boneheaded decisions that only made the situation worse for it, to the point the writers cannot figure ways to bring Skynet back anymore because it doomed itself in the very first movie. You should be worrying about Legion now, I think. It shuts down our electronics instead of nuking. Considering it won't have the advantage of triggering a nuclear attack because that's not how nukes work, the evil sentient AI is so doomed to fail it's ridiculous to think otherwise. But, companies know this is how the public works. They'll milk it for all it's worth so only a few companies can run or develop AIs, maybe making it illegal otherwise, or liable for DMCAs. Smart business move, but it affects my ability to research and use them. I cannot cure people's ability to separate reality and fiction though, and that's unfortunate.

boringuser2 3y ago
Very Dunning-Kruger post right here.

brookst 3y ago

> How can it not be obvious You have succinctly and completely summed up the AI risk argument more eloquently than anyone I've seen before. "How can it not be obvious?" Everything else is just intellectual fig leaves for the core argument that intuitively, without evidence, this proposition is obvious. The problem is, lots of "obvious" things have turned out to be very wrong. Sometimes relatively harmlessly, like the obviousness of the sun revolving around the earth, and sometimes catastrophically, like the obviousness of one race being inherently inferior. We should be very suspicious of policy that is based on propositions so obvious that it's borderline offensive to question them.

computerphage 3y ago
> borderline offensive to question them I would be happy to politely discuss any proposition regarding AI Risk. I don't think any claim should go unquestioned. I can also point you to much longer-form discussions. For example, this post, which has 670 comments, discussing various aspects of the argument: https://www.lesswrong.com/posts/uMQ3cqWDPHhjtiesc/agi-ruin-a-list-of-lethalities https://www.lesswrong.com/posts/uMQ3cqWDPHhjtiesc/agi-ruin-a...
- frankzinger 3y ago
  That link is about the risks of AGI, which doesn't exist, and there's no reason to believe that it ever will exist. (If I'm wrong about AGI then I'm open to being convinced, but that's a different conversation as the topic here is non-general AI, is it not?)
  - computerphage 3y ago
    I disagree that there's no reason to believe it will ever exist. For one thing, many smart people are trying to build the technology right now and they believe it to be possible. I see no compelling case that the intelligence scale simply tops-out with humans; that a more intelligent system is ruled out by the laws of physics. The topic here is human extinction caused by AI. I don't know of any serious argument for why a non-general intelligence (really a system less intelligent than a human) would pose an extinction risk to humanity. Plus, my background understanding of the people who signed this is that they're worried about AGI, not present-day systems, but that's an inference.

Veen 3y ago

It is possible to believe that AI poses threat, while also thinking that the AI safety organizations currently sprouting up are essentially grifts that will do absolutely nothing to combat the genuine threat. Especially when their primary goal seems to be the creation of well-funded sinecures for a group of like-minded, ideologically aligned individuals who want to limit AI control to a small group of wealthy technologists.

patrec 3y ago
I agree. But as you can see yourself, there are countless people even here, in a technical forum, who claim that AI poses no plausible threat whatsoever. I fail to see how one can reasonably believe that.

patch_cable 3y ago

> How can it not be obvious to you It isn't obvious to me. And I've yet to read something that spills out the obvious reasoning. I feel like everything I've read just spells out some contrived scenario, and then when folks push back explaining all the reasons that particular scenario wouldn't come to pass, the counter argument is just "but that's just one example!" without offering anything more convincing. Do you have any better resources that you could share?

hackinthebochs 3y ago
The history of humanity is replete with examples of the slightly more technologically advanced group decimating their competition. The default position should be that uneven advantage is extremely dangerous to those disadvantaged. This idea that an intelligence significantly greater than our own is benign just doesn't pass the smell test. From the tech perspective: higher order objectives are insidious. While we may assume a narrow misalignment in received vs intended objective of a higher order nature, this misalignment can result in very divergent first-order behavior. Misalignment in behavior is by its nature destructive of value. The question is how much destruction of value can we expect? The machine may intentionally act in destructive ways as it goes about carrying out its slightly misaligned higher order objective-guided behavior. Of course we will have first-order rules that constrain its behavior. But again, slight misalignment in first-order rule descriptions are avenues for exploitation. If we cannot be sure we have zero exploitable rules, we must assume a superintelligence will find such loopholes and exploit them to maximum effect. Human history since we started using technology has been a lesson on the outcome of an intelligent entity aimed at realizing an objective. Loopholes are just resources to be exploited. The destruction of the environment and other humans is just the inevitable outcome of slight misalignment of an intelligent optimizer. If this argument is right, the only thing standing between us and destruction is the AGI having reached its objective before it eats the world. That is, there will always be some value lost in any significant execution of an AGI agent due to misalignment. Can we prove that the ratio of value created to value lost due to misalignment is always above some suitable threshold? Until we do, x-risk should be the default assumption.
patrec 3y ago
OK, which of the following propositions do you disagree with? 1. AIs have made rapid progress in approaching and often surpassing human abilities in many areas. 2. The fact that AIs have some inherent scalability, speed, cost, reliability and compliance advantages over humans means that many undesirable things that could previously not be done at all or at least not done at scale are becoming both feasible and cost-effective. Examples would include 24/7 surveillance with social desirability scoring based on a precise ideological and psychological profile derived from a comprehensive record of interactions, fine-tuned mass manipulation and large scale plausible falsification of the historical record. Given the general rise of authoritarianism, this is pretty worrying. 3. On the other hand the rapid progress and enormous investment we've been seeing makes it very plausible that before too long we will, in fact, see AIs that outperform humans on most tasks. 4. AIs that are much smarter than any human pose even graver dangers. 5. Even if there is a general agreement that AIs pose grave or even existential risks, states, organizations and individuals will are all incentivized to still seek to improve their own AI capabilities, as doing so provides an enormous competitive advantage. 6. There is a danger of a rapid self-improvement feedback loop. Humans can reproduce, learn new and significantly improve existing skills, as well as pass skills on to others via teaching. But there are fundamental limits on speed and scale for all of these, whereas it's not obvious at all how an AI that has reached super-human level intelligence would be fundamentally prevented from rapidly improving itself further, or produce millions of "offspring" that can collaborate and skill-exchange extremely efficiently. Furthermore, since AIs can operate at completely different time scales than humans, this all could happen extremely rapidly, and such a system might very quickly become much more powerful than humanity and the rest of AIs combined. I think you only have to subscribe a small subset of these (say 1.&2.) to conclude that "AI is an uniquely powerful and thus uniquely dangerous technology" obviously follows. For the stronger claim of existential risk, have you read the lesswrong link posted elsewhere in this discussion? https://www.lesswrong.com/posts/uMQ3cqWDPHhjtiesc/agi-ruin-a-list-of-lethalities https://www.lesswrong.com/posts/uMQ3cqWDPHhjtiesc/agi-ruin-a... ?

api 3y ago

Most of the credible threats I see from AI that don't rely on a lot of sci-fi extrapolation involve small groups of humans in control of massively powerful AI using it as a force multiplier to control or attack other groups of humans. Sam Altman's proposal is to create precisely that situation with himself and a few other large oligarchs being the ones in control of the leading edge of AI. If we really do face runaway intelligence growth and god-like AIs then this is a profound amount of power to place in the hands of just a few people. Even worse it opens the possibility that such developments could happen partly in secret, so the public might not even know how powerful the secret AIs under command of the oligarchs have become. The analogy with nuclear weapons is profoundly broken in lots of ways. Reasoning from a sloppy analogy is a great way to end up somewhere stupid. AI is a unique technology with a unique set of risks and benefits and a unique profile.

csomar 3y ago

> It's like saying nuclear missiles can't possibly be dangerous and nuclear arms reduction and non-proliferation treaties were a scam, because the US, China and the Soviet Union had positioned themselves to capture the majority of the strategic value nukes bring. I'm honestly not sure if this is sarcasm. The non-proliferation treaties are indeed a scam. The war is raging between the US and Russia and nuclear is a big part of it (though just words/threats for now). It's nonsensical to think that these treaties are possible.

patrec 3y ago
Not only is the Non proliferation treaty possible, it's been evidently effective in slowing the proliferation of nuclear weapons. The only country that ratified or acceded to it and went on to develop nuclear weapons is North Korea, and the only country that ratified or acceded to it and looks on track to develop nuclear weapons is Iran. One country that was not a signatory and developed nuclear weapons voluntarily gave them up and acceded to it partly due to international pressure (South Africa). Israel, Pakistan and India have since developed nuclear weapons but they were never signatories, the only other non-signatory is South Sudan which probably won't acquire nuclear capabilities anytime soon.