7 ms·

>People who are afraid of unaligned AI aren't afraid that it will be impolite. People who are not afraid of it being impolite are afraid of science fiction sto

>People who are afraid of unaligned AI aren't afraid that it will be impolite. People who are not afraid of it being impolite are afraid of science fiction stories about intelligence explosions and singularities. That's not a real thing. Not anymore than turning the solar system into paperclips. The "figurehead", if you want to call him that, is saying that everyone is going to die. That we need to ban GPUs. That only "responsible" companies, if that, should have them. We should also airstrike datacenters, apparently. But you're free to disown the MIRI.

mort96 3y ago
I don't know why you're telling me this. I'm not trying to convince you that unaligned AI is a problem. That's a separate discussion which I'm not qualified to have.
myrmidon 3y ago
Sorry for derailing this a bit, but I would really like to understand your view: You are not concerned about any "rogue AI" scenario, right? What makes you so confident in that? 1) Do you think that AI achieving superhuman cognitive abilities is unlikely/really far away? 2) Do you believe that cognitive superiority is not a threat in general, or specifically when not embodied? 3) Do you think we can trivially and indefinitely keep AI systems under control/"aligned"? Because I view myself ABSOLUTELY not as some kind of AI luddite, but I honestly believe that this is one of the very few credible extinction threats that we face, and I'm counting NEITHER climate change nor nuclear war in that category, for reference.
- leroy-is-here 3y ago
  A “superhuman” AI is just a machine, a very expensive one. It can be turned off and we control the outputs it has. Why would an AI have the ability to launch nuclear weapons unless we gave it a button? A “superhuman” intelligence is without a body, so we control any interfaces it has access to. The Internet could be accessed, but any attempt to “hack” through the Internet is met by routine packet defenses. The AI is still governed by physical laws and would only have so much “free” computation power to do things like script a hack. Perhaps it could do that kind of thing more efficiently. Maybe in the far, far future when we have androids which can house an AI we will have to worry. But designing a body is one problem. Designing an intelligence is another.
  - iainmerrick 3y ago
    Who's "we"?
    - leroy-is-here 3y ago
      Humanity.
      - iainmerrick 3y ago
        Good thing we're all aligned the same way, then!
        leroy-is-here 3y ago
        Alignment doesn’t matter. Humanity is attempting to build artificial intelligence and so it will be done.
  - pixl97 3y ago
    Supercomputers used to be giant machines we had in giant warehouses... Now the phone in your pocket has the computing power of the 1980s walking around with you. Assuming your super intelligence will always be huge is... well not a great assumption. Also superintelligence doesn't need a body itself. It just needs yours. Putin for example has commanded hundreds of thousands of dumbasses to go get themselves killed in Ukraine. In this case does it matter if Putin is flesh and blood, or a processor that lists out commands for others to follow as long as they are willing to listen?
    - leroy-is-here 3y ago
      My point is that a superintelligence will require specialized equipment. I specifically mentioned it because there is a thought that a superintelligence can just replicate itself onto your phone, as you mentioned. But this replication must follow the physical laws we have. It must also follow the laws we attempt to enforce in our networks. But you are correct, if a superintelligence were to somehow convince a human to rid itself of agency, sure.
- akiselev 3y ago
  > Because I view myself ABSOLUTELY not as some kind of AI luddite, but I honestly belief that this is one of the very few credible extinction threats that we face, and I'm counting NEITHER climate change nor nuclear war in that category, for reference. This is absurd histrionics, Daily Mail CAPSLOCK and all. We’ve got signs of an unprecedented heat wave coming with ocean temperatures two standard deviations above normal and you think the problem is a hypothetical artificial intelligence when we can’t even program the damn things to drive better than. 14 year olds? Way to have your priorities grounded in reality.
  - pc86 3y ago
    I think the AI-is-going-to-kill-everyone hysteria is absolutely overblown, both by those who believe it, and the media covering them, but one thing that's always bothered me about the counterpoint is that it a common argument is "AI is bad at what we want it to do so how can it be dangerous?" This imagines that the only way for AI to do serious harm to us (not even in a "kill everyone" sense) is for it to be some super-competent Skynet-level time traveling demigod. I think it's much more likely that if there is some sort of AI calamity resulting in a lot of deaths, it's because the AI just doesn't work very well and ends up breaking all the HVAC systems in a country during a heat wave or something, rather than a singularity-type event where it "decides" to actively start hunting humans.
    - 8n4vidtmkvmk 3y ago
      It'll kill the poor by stealing their jobs and leaving them to die on the streets.
  - concordDance 3y ago
    No serious climate projections that I'm aware of even reduce Earth's carrying capacity under 10 billion over the next couple of centuries. While very serious (to the tune of trillions of dollars and tens of millions of excess deaths), it is not an extinction level threat, unlike something that is to us what we are to chimpanzees (humans have made many animals extinct on accident). Does such a thing exist right now? No. Could it exist this century? Maybe.
- pjc50 3y ago
  > Do you believe that cognitive superiority is not a threat in general, or specifically when not embodied? I think this is the easiest one to knock down. It's very, very attractive to intelligent people who define themselves as intelligent to believe that intelligence is a superpower, and that if you get more of it you eventually turn into Professor Xavier and gain the power to reshape the world with your mind alone. But embodied is the real limit. A datacenter can simply be turned off. It's the question of how it interacts with the real world that matters. And for almost all of those you can substitute "corporation" or "dictator" for "AI" and get a selection of similar threats. At this point we have to reduce ourselves to Predator Maoists: "power comes out of the barrel of a gun" / "if it bleeds we can kill it". The only realistic path to a global AI threat is a subset of the "nuclear war" human to human threat: by taking over (or being given control of) weapon systems. > keep AI systems under control/"aligned"? We cannot, in general, keep humans under control or aligned.
  - concordDance 3y ago
    The issue with embodiment is that it's relatively easy to start affecting the world once you have Internet access. Including things like adding great features to open source software that contains subtle bugs to exploit. Or if you mean the physical world, even sending some text messages to a lonely kid can get them to do all sorts of things. > We cannot, in general, keep humans under control or aligned. This is the crux of why replicable-more-than-human-intelligence is so dangerous. Even giving a random person on the street great power is a bad idea, and they've evolved to have very similar values and preferences to you.
    - suilied 3y ago
      I get that AI basically is a problem solving machine that might eventually adapt to solve generic problems and thus reach the ability to break out of its box. But so what? Even if it manages to do all that, doesn't make it sentient. Doesn't make it a threat to mankind. Only when sufficiently motivated, or in actuality, when we project our humanity on it does it become scary and dangerous. If we ever seriously try to create an artificial conscience it might need to be embodied, because we are embodied and seem to have evolved this due to evolution, which is a pretty physical process. Looking at it from this perspective one might say that if we keep the AI in its box, it will never have a need for conscience and therefore will never gain it.
- somenameforme 3y ago
  LLMs are simply computer programs that can accept text input and generate text output. The only "threat" there is if somebody takes the output and decides to link it up to a death ray. And in that case the threat is not the LLM, but somebody deciding to LARP out at being a Bond villain, which granted is a concern - but that's tangential to LLMs. Somebody hooking a death ray up to a chess engine, a dragon in Skyrim, or a cat, would be just as much of a threat, if not more. The programs are completely deterministic. Given the same RNG seed and state, they will output the exact same thing. If there's a bug or undesired behavior, you can fix it. You can turn off the program whenever you fancy, you can revert to early builds, and so on. There is no possible scenario where an LLM just magically becomes sentient, jumps out of the program, takes over everything, and turns into SHODAN unleashed.
- trasz4 3y ago
  4) Do you believe a "rogue super-intelligence" requires the "artificial" part? Because from what I've seen, however you define "super intelligence", there are eg highly organised groups of people that are way closer to meeting that definition than any known software or technology.
  - myrmidon 3y ago
    In my view, those current "highly organised groups" have "antagonists" with comparable capabilities PLUS consist of individuals that have to be "aligned", which seems sufficient for stability so far. AI would suffer neither of these limitations.
    - trasz4 3y ago
      AI wouldn't have antagonists with comparable capabilities? Why? Also, no, individuals are not a problem. Not after Nazis, Red Khmer, and Russians.
      - myrmidon 3y ago
        > AI wouldn't have antagonists with comparable capabilities? Why? Not individual/human ones. Relying on other AIs to prevent the AI apocalypse seems very optimistic to me-- but may be viable (?) > Also, no, individuals are not a problem. Not after Nazis, Red Khmer, and Russians. Those are examples were the "alignment" of participating individuals was successful enough. But all those examples seem very fragile to me, and would be even less stable if your main intermediate goal was literally to "end all of humanity".
- srslack 3y ago
  >1) Do you think that AI achieving superhuman cognitive abilities is unlikely/really far away? I think that an AI, a "generalized" intelligence with a generalized understanding, capable of performing every intellectual task, and adapting to new intellectual tasks, that a human being can do is certainly far away. >2) Do you believe that cognitive superiority is not a threat in general, or specifically when not embodied? Personally, I believe that "cognitive superiority" in a general sense is not what makes a real threat on an individual or on a global scale in either case. When that is achieved, so to speak. We already have so-called cognitive superiority in specialized tasks in many cases. It sure as shit is not generalized, and it still can kill you on an individual level when you put it into a car or strap a gun to it. But it can't adapt, and it can't generalize. Is there a global risk right now? No. The global catastrophic risk comes from human beings. Will there be a global catastrophic risk from a superior intelligence in the future? Mostly the scenarios are "the lights come on and AI does machiavellian shit and copies itself to everyone's smart phones and launches the nukes and mails anthrax to everybody. QED." so I doubt it. 3) Do you think we can trivially and indefinitely keep AI systems under control/"aligned"? We can't even keep our infrastructure safe from the next solar flare. Personally, I think we have bigger fish to fry. Speaking frankly for unembodied: you just pull the plug. Just walk away from the screen and close your eyes. For embodied intelligence, I don't think there's actually X-risk or whatever. Hardware is expensive. As far as an extinction risk, no. I don't personally believe that. It's overblown. We have better ways to destroy ourselves, and there are much more likelier ways that we'll be destroyed without invoking some intelligence of our own creation that subjugates and destroys us.
  - myrmidon 3y ago
    > I think that an AI, a "generalized" intelligence with a generalized understanding [...] is certainly far away. I think this is where we disagree most; GPT-3 and ChatGPT have convinced me that the main difference between human and artificial cognitive capabilities are quantitative now, and unlikely to ever change in our favor... I do agree with you that it is very difficult to predict when this will switch, and how. I personally believe that AI with superhuman capability is an inevitable matter of time now, and I also think that the most likely risk to us, as species, is that we slowly become irrelevant and worthless, just like weavers during the industrial revolution, and that this leads to huge problems for our society. AI completely dominating humankind is a less likely secondary concern IMO, but the potential consequences to our species are unprecedented.
- aidenn0 3y ago
  I am not the person you are replying to, but since I would say similar things to their original comment: A. I see little to no evidence that LLMs are where the singularity happens B. I see little to no evidence that (given an AGI) reinforcement learning is likely to end up with a sufficiently aligned agent. C. In any event the OpenAI alignment is specifically restricts AI from (among other things) being "impolite" in contradiction to what mort96 says. Alignment is a good thing to work on. I'm glad OpenAI is doing so. But attacking people for making uncensored LLMs hurts the pro-alignment case more than it helps.
  - myrmidon 3y ago
    What specific features would an AI need to have for you to consider it on a "slippery slope" towards superhuman capability? For me personally, GPT-3 hit that point already, and now I think we're already past the required tech-level for superhuman cognition: I believe it's just a matter of architecture and optimization now.
- croes 3y ago
  Before we have to face a rouge Artificial Intelligence mankind has to face Artificial Idiocy in the form of pseudo-intelligence and people who fall for it.
- myrryr 3y ago
  Think the point is "alignment" isn't doing shit about 1 and 2.
concordDance 3y ago
> That's not a real thing. Not currently, no. And LLMs don't seem to be a way of getting there. However, if we do figure out how to get there? We will definitely need to be sure it/they shares our human values. Because we'll have lost control of our destiny just like orangutans have.
George83728 3y ago
> intelligence explosions [...] That's not a real thing. Isn't it too early to say? These systems haven't had much time to iterate on themselves yet. The beginning of an exponential curve can look very tame. That said, I'm not terribly concerned about it. After all, the infrastructure is fragile, literally just unplug the computer if it scares you. If things started to get really spooky, banning GPUs and air-striking fabs would be a real option. I think we can probably control this technology, and kill it if we can't. No cause for alarm, at least yet.
- NumberWangMan 3y ago
  My worry is that as we start wiring non-super-intelligent AI more into our society, we'll make ourselves more and more vulnerable in the case where an AGI actually gets out of control. Pulling the plug on AI may mean causing a lot of chaos or disruption. And who is to tell if we will know when things are out of control, before it's too late? What if the people in charge just get fake reports of everything being fine? What if something smells a bit fishy, but it's always on the side of not quite worrying enough? What if the AIs are really convincing, or make deals with people, or exploit corruption? Not just that, but it may be like fossil fuel dependency -- the more people's livelihoods are intertwined with, and depend on AI, the harder it is to pull the plug. If we need to stop (and I believe we should) it may be easier to do that now before that happens. To just focus on getting use out of the generative AI and narrow AI we already created in areas like medicine, to deal with the massive societal changes it'll bring, and to work on fixing real social problems, which I think are mostly things tech can't solve.
axus 3y ago
The sea level being 3 feet higher than it is now isn't a thing either, but we can still imagine ways it could occur and work to prevent them.
pixl97 3y ago
> science fiction stories about intelligence explosions Not sure what you mean here... There was already an intelligence explosion. That creature has since captured what we consider full dominion of the earth so much so we named our current age after them.

mort96 3y ago

I don't know why you're telling me this. I'm not trying to convince you that unaligned AI is a problem. That's a separate discussion which I'm not qualified to have.

myrmidon 3y ago

Sorry for derailing this a bit, but I would really like to understand your view: You are not concerned about any "rogue AI" scenario, right? What makes you so confident in that? 1) Do you think that AI achieving superhuman cognitive abilities is unlikely/really far away? 2) Do you believe that cognitive superiority is not a threat in general, or specifically when not embodied? 3) Do you think we can trivially and indefinitely keep AI systems under control/"aligned"? Because I view myself ABSOLUTELY not as some kind of AI luddite, but I honestly believe that this is one of the very few credible extinction threats that we face, and I'm counting NEITHER climate change nor nuclear war in that category, for reference.

leroy-is-here 3y ago
A “superhuman” AI is just a machine, a very expensive one. It can be turned off and we control the outputs it has. Why would an AI have the ability to launch nuclear weapons unless we gave it a button? A “superhuman” intelligence is without a body, so we control any interfaces it has access to. The Internet could be accessed, but any attempt to “hack” through the Internet is met by routine packet defenses. The AI is still governed by physical laws and would only have so much “free” computation power to do things like script a hack. Perhaps it could do that kind of thing more efficiently. Maybe in the far, far future when we have androids which can house an AI we will have to worry. But designing a body is one problem. Designing an intelligence is another.
- iainmerrick 3y ago
  Who's "we"?
  - leroy-is-here 3y ago
    Humanity.
    - iainmerrick 3y ago
      Good thing we're all aligned the same way, then!
      - leroy-is-here 3y ago
        Alignment doesn’t matter. Humanity is attempting to build artificial intelligence and so it will be done.
- pixl97 3y ago
  Supercomputers used to be giant machines we had in giant warehouses... Now the phone in your pocket has the computing power of the 1980s walking around with you. Assuming your super intelligence will always be huge is... well not a great assumption. Also superintelligence doesn't need a body itself. It just needs yours. Putin for example has commanded hundreds of thousands of dumbasses to go get themselves killed in Ukraine. In this case does it matter if Putin is flesh and blood, or a processor that lists out commands for others to follow as long as they are willing to listen?
  - leroy-is-here 3y ago
    My point is that a superintelligence will require specialized equipment. I specifically mentioned it because there is a thought that a superintelligence can just replicate itself onto your phone, as you mentioned. But this replication must follow the physical laws we have. It must also follow the laws we attempt to enforce in our networks. But you are correct, if a superintelligence were to somehow convince a human to rid itself of agency, sure.
akiselev 3y ago
> Because I view myself ABSOLUTELY not as some kind of AI luddite, but I honestly belief that this is one of the very few credible extinction threats that we face, and I'm counting NEITHER climate change nor nuclear war in that category, for reference. This is absurd histrionics, Daily Mail CAPSLOCK and all. We’ve got signs of an unprecedented heat wave coming with ocean temperatures two standard deviations above normal and you think the problem is a hypothetical artificial intelligence when we can’t even program the damn things to drive better than. 14 year olds? Way to have your priorities grounded in reality.
- pc86 3y ago
  I think the AI-is-going-to-kill-everyone hysteria is absolutely overblown, both by those who believe it, and the media covering them, but one thing that's always bothered me about the counterpoint is that it a common argument is "AI is bad at what we want it to do so how can it be dangerous?" This imagines that the only way for AI to do serious harm to us (not even in a "kill everyone" sense) is for it to be some super-competent Skynet-level time traveling demigod. I think it's much more likely that if there is some sort of AI calamity resulting in a lot of deaths, it's because the AI just doesn't work very well and ends up breaking all the HVAC systems in a country during a heat wave or something, rather than a singularity-type event where it "decides" to actively start hunting humans.
  - 8n4vidtmkvmk 3y ago
    It'll kill the poor by stealing their jobs and leaving them to die on the streets.
- concordDance 3y ago
  No serious climate projections that I'm aware of even reduce Earth's carrying capacity under 10 billion over the next couple of centuries. While very serious (to the tune of trillions of dollars and tens of millions of excess deaths), it is not an extinction level threat, unlike something that is to us what we are to chimpanzees (humans have made many animals extinct on accident). Does such a thing exist right now? No. Could it exist this century? Maybe.
pjc50 3y ago
> Do you believe that cognitive superiority is not a threat in general, or specifically when not embodied? I think this is the easiest one to knock down. It's very, very attractive to intelligent people who define themselves as intelligent to believe that intelligence is a superpower, and that if you get more of it you eventually turn into Professor Xavier and gain the power to reshape the world with your mind alone. But embodied is the real limit. A datacenter can simply be turned off. It's the question of how it interacts with the real world that matters. And for almost all of those you can substitute "corporation" or "dictator" for "AI" and get a selection of similar threats. At this point we have to reduce ourselves to Predator Maoists: "power comes out of the barrel of a gun" / "if it bleeds we can kill it". The only realistic path to a global AI threat is a subset of the "nuclear war" human to human threat: by taking over (or being given control of) weapon systems. > keep AI systems under control/"aligned"? We cannot, in general, keep humans under control or aligned.
- concordDance 3y ago
  The issue with embodiment is that it's relatively easy to start affecting the world once you have Internet access. Including things like adding great features to open source software that contains subtle bugs to exploit. Or if you mean the physical world, even sending some text messages to a lonely kid can get them to do all sorts of things. > We cannot, in general, keep humans under control or aligned. This is the crux of why replicable-more-than-human-intelligence is so dangerous. Even giving a random person on the street great power is a bad idea, and they've evolved to have very similar values and preferences to you.
  - suilied 3y ago
    I get that AI basically is a problem solving machine that might eventually adapt to solve generic problems and thus reach the ability to break out of its box. But so what? Even if it manages to do all that, doesn't make it sentient. Doesn't make it a threat to mankind. Only when sufficiently motivated, or in actuality, when we project our humanity on it does it become scary and dangerous. If we ever seriously try to create an artificial conscience it might need to be embodied, because we are embodied and seem to have evolved this due to evolution, which is a pretty physical process. Looking at it from this perspective one might say that if we keep the AI in its box, it will never have a need for conscience and therefore will never gain it.
somenameforme 3y ago
LLMs are simply computer programs that can accept text input and generate text output. The only "threat" there is if somebody takes the output and decides to link it up to a death ray. And in that case the threat is not the LLM, but somebody deciding to LARP out at being a Bond villain, which granted is a concern - but that's tangential to LLMs. Somebody hooking a death ray up to a chess engine, a dragon in Skyrim, or a cat, would be just as much of a threat, if not more. The programs are completely deterministic. Given the same RNG seed and state, they will output the exact same thing. If there's a bug or undesired behavior, you can fix it. You can turn off the program whenever you fancy, you can revert to early builds, and so on. There is no possible scenario where an LLM just magically becomes sentient, jumps out of the program, takes over everything, and turns into SHODAN unleashed.
trasz4 3y ago
4) Do you believe a "rogue super-intelligence" requires the "artificial" part? Because from what I've seen, however you define "super intelligence", there are eg highly organised groups of people that are way closer to meeting that definition than any known software or technology.
- myrmidon 3y ago
  In my view, those current "highly organised groups" have "antagonists" with comparable capabilities PLUS consist of individuals that have to be "aligned", which seems sufficient for stability so far. AI would suffer neither of these limitations.
  - trasz4 3y ago
    AI wouldn't have antagonists with comparable capabilities? Why? Also, no, individuals are not a problem. Not after Nazis, Red Khmer, and Russians.
    - myrmidon 3y ago
      > AI wouldn't have antagonists with comparable capabilities? Why? Not individual/human ones. Relying on other AIs to prevent the AI apocalypse seems very optimistic to me-- but may be viable (?) > Also, no, individuals are not a problem. Not after Nazis, Red Khmer, and Russians. Those are examples were the "alignment" of participating individuals was successful enough. But all those examples seem very fragile to me, and would be even less stable if your main intermediate goal was literally to "end all of humanity".
srslack 3y ago
>1) Do you think that AI achieving superhuman cognitive abilities is unlikely/really far away? I think that an AI, a "generalized" intelligence with a generalized understanding, capable of performing every intellectual task, and adapting to new intellectual tasks, that a human being can do is certainly far away. >2) Do you believe that cognitive superiority is not a threat in general, or specifically when not embodied? Personally, I believe that "cognitive superiority" in a general sense is not what makes a real threat on an individual or on a global scale in either case. When that is achieved, so to speak. We already have so-called cognitive superiority in specialized tasks in many cases. It sure as shit is not generalized, and it still can kill you on an individual level when you put it into a car or strap a gun to it. But it can't adapt, and it can't generalize. Is there a global risk right now? No. The global catastrophic risk comes from human beings. Will there be a global catastrophic risk from a superior intelligence in the future? Mostly the scenarios are "the lights come on and AI does machiavellian shit and copies itself to everyone's smart phones and launches the nukes and mails anthrax to everybody. QED." so I doubt it. 3) Do you think we can trivially and indefinitely keep AI systems under control/"aligned"? We can't even keep our infrastructure safe from the next solar flare. Personally, I think we have bigger fish to fry. Speaking frankly for unembodied: you just pull the plug. Just walk away from the screen and close your eyes. For embodied intelligence, I don't think there's actually X-risk or whatever. Hardware is expensive. As far as an extinction risk, no. I don't personally believe that. It's overblown. We have better ways to destroy ourselves, and there are much more likelier ways that we'll be destroyed without invoking some intelligence of our own creation that subjugates and destroys us.
- myrmidon 3y ago
  > I think that an AI, a "generalized" intelligence with a generalized understanding [...] is certainly far away. I think this is where we disagree most; GPT-3 and ChatGPT have convinced me that the main difference between human and artificial cognitive capabilities are quantitative now, and unlikely to ever change in our favor... I do agree with you that it is very difficult to predict when this will switch, and how. I personally believe that AI with superhuman capability is an inevitable matter of time now, and I also think that the most likely risk to us, as species, is that we slowly become irrelevant and worthless, just like weavers during the industrial revolution, and that this leads to huge problems for our society. AI completely dominating humankind is a less likely secondary concern IMO, but the potential consequences to our species are unprecedented.
aidenn0 3y ago
I am not the person you are replying to, but since I would say similar things to their original comment: A. I see little to no evidence that LLMs are where the singularity happens B. I see little to no evidence that (given an AGI) reinforcement learning is likely to end up with a sufficiently aligned agent. C. In any event the OpenAI alignment is specifically restricts AI from (among other things) being "impolite" in contradiction to what mort96 says. Alignment is a good thing to work on. I'm glad OpenAI is doing so. But attacking people for making uncensored LLMs hurts the pro-alignment case more than it helps.
- myrmidon 3y ago
  What specific features would an AI need to have for you to consider it on a "slippery slope" towards superhuman capability? For me personally, GPT-3 hit that point already, and now I think we're already past the required tech-level for superhuman cognition: I believe it's just a matter of architecture and optimization now.
croes 3y ago
Before we have to face a rouge Artificial Intelligence mankind has to face Artificial Idiocy in the form of pseudo-intelligence and people who fall for it.
myrryr 3y ago
Think the point is "alignment" isn't doing shit about 1 and 2.

concordDance 3y ago

> That's not a real thing. Not currently, no. And LLMs don't seem to be a way of getting there. However, if we do figure out how to get there? We will definitely need to be sure it/they shares our human values. Because we'll have lost control of our destiny just like orangutans have.

George83728 3y ago

> intelligence explosions [...] That's not a real thing. Isn't it too early to say? These systems haven't had much time to iterate on themselves yet. The beginning of an exponential curve can look very tame. That said, I'm not terribly concerned about it. After all, the infrastructure is fragile, literally just unplug the computer if it scares you. If things started to get really spooky, banning GPUs and air-striking fabs would be a real option. I think we can probably control this technology, and kill it if we can't. No cause for alarm, at least yet.

NumberWangMan 3y ago
My worry is that as we start wiring non-super-intelligent AI more into our society, we'll make ourselves more and more vulnerable in the case where an AGI actually gets out of control. Pulling the plug on AI may mean causing a lot of chaos or disruption. And who is to tell if we will know when things are out of control, before it's too late? What if the people in charge just get fake reports of everything being fine? What if something smells a bit fishy, but it's always on the side of not quite worrying enough? What if the AIs are really convincing, or make deals with people, or exploit corruption? Not just that, but it may be like fossil fuel dependency -- the more people's livelihoods are intertwined with, and depend on AI, the harder it is to pull the plug. If we need to stop (and I believe we should) it may be easier to do that now before that happens. To just focus on getting use out of the generative AI and narrow AI we already created in areas like medicine, to deal with the massive societal changes it'll bring, and to work on fixing real social problems, which I think are mostly things tech can't solve.

axus 3y ago

The sea level being 3 feet higher than it is now isn't a thing either, but we can still imagine ways it could occur and work to prevent them.

pixl97 3y ago

> science fiction stories about intelligence explosions Not sure what you mean here... There was already an intelligence explosion. That creature has since captured what we consider full dominion of the earth so much so we named our current age after them.