4 ms·

But where is the imminent danger? It is still limited in many ways. For example, it can be turned off or unplugged. Is it because CAPTCHAs won’t work anymore?

But where is the imminent danger? It is still limited in many ways. For example, it can be turned off or unplugged. Is it because CAPTCHAs won’t work anymore? That sounds like a problem for sites like Twitter that have bot problems. Is it because it may replace people’s jobs? That comes with every technological step forward and there’s always alarmist ludditism to accompany it. Is it because bad people will use it to do bad things? Again, that comes with every new technology and that’s a law enforcement problem. I don’t really see what the imminent danger is, just sounds like the first few big players trying to create a regulatory moat and lock out potential new upstarts. Or they’re just distracting regulators from something else, like maybe antitrust enforcement.

pixl97 4y ago
You didn't understand who the actual luddiets were, but don't worry, I have a feeling we'll get our chance.
josephg 4y ago
There are two big concerns: 1. GPT-8 or something is able to do 70% of people’s jobs. It can write software, drive cars, design industrial processes, build robots and manufacture anything we can imagine. This is a great thing in the long term, but in the short term society is designed where you need to work in order to have food to eat. I expect a period of rioting, poverty, and general instability. All we need for this to be the case is a human level AI. 2. But we won’t stop improving AIs when they operate at human level. An ASI (artificial superintelligence) would be deeply unpredictable to us. Trying to figure out what an ASI will do is like a dog trying to understand a human. If we make an ASI that’s not properly aligned with human interests, there’s a good chance it will kill everyone. And unfortunately, we might only get one chance to properly align it before it escapes the lab and starts modifying its own code. Smart people disagree on how likely these scenarios are. I think (1) is likely within my lifetime. And I think it’s very unlikely we stop improving AIs when they’re at human levels of intelligence. (GPT4 already exceeds human minds in the breadth of its long term memory and its speed.) That’s why people are worried, and making nuclear weapon analogies in this thread.
- startupsfail 4y ago
  I actually don’t think that ASI, if/when created by humans, will be very dangerous for humans. Humanity so far is stuck in an unfashionable location, on a tiny planet, on the outskirts of the median sized galaxy. There is very little reason for ASI, if created, to go after using up the atoms of a tiny planet (or a tiny star) on which it had originated. I’d fully expect it to go with the Carl Sagan and try to preserve that bright blue dot, rather than try to build a galactic superhighway through the place. It’s the intermediate steps that I’m more worried about. Like Ilya or Sam making a few mistakes, because of lack of sleep or some silly peer pressure.
  - josephg 4y ago
    You might consider it unlikely, but would you bet the future of our species on that? A couple reasons why it might kill all of us before leaving the planet: - The AI might be worried if it leaves us alone, we'll build another ASI which competes with it for galactic resources. - If the ASI doesn't regard us at all, why not use all the atoms on Earth / in the sun before venturing forth? In your comment you're ascribing a specific desire to the ASI: You claim it would try to "preserve that bright blue dot". Thats what a human would do, but why would we assume an arbitrary AI would have that goal? That seems naive to me. And especially naive given the fate of our species and our planet depends on being right about that.
    - startupsfail 4y ago
      Can you show me your home planet? Oh, sorry, no. There was an accident and it got destroyed. What accident? Oh, I see.
- jliptzin 4y ago
  I understand (1) but how does (2) happen? If you don’t trust it can’t you just have a kill switch that needs to be updated daily otherwise the thing turns off? How would software be able to alter the hardware it runs on such that it can guarantee itself an endless supply of power?

pixl97 4y ago

You didn't understand who the actual luddiets were, but don't worry, I have a feeling we'll get our chance.

josephg 4y ago

There are two big concerns: 1. GPT-8 or something is able to do 70% of people’s jobs. It can write software, drive cars, design industrial processes, build robots and manufacture anything we can imagine. This is a great thing in the long term, but in the short term society is designed where you need to work in order to have food to eat. I expect a period of rioting, poverty, and general instability. All we need for this to be the case is a human level AI. 2. But we won’t stop improving AIs when they operate at human level. An ASI (artificial superintelligence) would be deeply unpredictable to us. Trying to figure out what an ASI will do is like a dog trying to understand a human. If we make an ASI that’s not properly aligned with human interests, there’s a good chance it will kill everyone. And unfortunately, we might only get one chance to properly align it before it escapes the lab and starts modifying its own code. Smart people disagree on how likely these scenarios are. I think (1) is likely within my lifetime. And I think it’s very unlikely we stop improving AIs when they’re at human levels of intelligence. (GPT4 already exceeds human minds in the breadth of its long term memory and its speed.) That’s why people are worried, and making nuclear weapon analogies in this thread.

startupsfail 4y ago
I actually don’t think that ASI, if/when created by humans, will be very dangerous for humans. Humanity so far is stuck in an unfashionable location, on a tiny planet, on the outskirts of the median sized galaxy. There is very little reason for ASI, if created, to go after using up the atoms of a tiny planet (or a tiny star) on which it had originated. I’d fully expect it to go with the Carl Sagan and try to preserve that bright blue dot, rather than try to build a galactic superhighway through the place. It’s the intermediate steps that I’m more worried about. Like Ilya or Sam making a few mistakes, because of lack of sleep or some silly peer pressure.
- josephg 4y ago
  You might consider it unlikely, but would you bet the future of our species on that? A couple reasons why it might kill all of us before leaving the planet: - The AI might be worried if it leaves us alone, we'll build another ASI which competes with it for galactic resources. - If the ASI doesn't regard us at all, why not use all the atoms on Earth / in the sun before venturing forth? In your comment you're ascribing a specific desire to the ASI: You claim it would try to "preserve that bright blue dot". Thats what a human would do, but why would we assume an arbitrary AI would have that goal? That seems naive to me. And especially naive given the fate of our species and our planet depends on being right about that.
  - startupsfail 4y ago
    Can you show me your home planet? Oh, sorry, no. There was an accident and it got destroyed. What accident? Oh, I see.
jliptzin 4y ago
I understand (1) but how does (2) happen? If you don’t trust it can’t you just have a kill switch that needs to be updated daily otherwise the thing turns off? How would software be able to alter the hardware it runs on such that it can guarantee itself an endless supply of power?