5 ms·

I understand but agentic/learning general intelligence has not been shown to exist, except ourselves. I’d say this is like worrying about deadly quantum laser w

I understand but agentic/learning general intelligence has not been shown to exist, except ourselves. I’d say this is like worrying about deadly quantum laser weapons that will consume the planet when we are still in the AK47 phase. Edit: it could still be true though. I guess I like some more handholding and pre-chewing before giving governments and large corporations more ropes.

mitthrowaway2 3y ago
That hand-holding exists elsewhere, but I get the sense that this particular document is very short on purpose.
the8472 3y ago
The difference to doomsday weapons is that we can build the weapons first and then worry about using them. With an AGI building one alone might be sufficient. It could become smart enough to unbox itself during a training run.
- SanderNL 3y ago
  The LHC could have produced a black hole that swallowed the earth, so you’re also too late if that were the case. Lots of tech is like that, but it turns out we don’t get anything done if we keep fearmongering.
ethanbond 3y ago
Or it’s like worrying about an arms race toward civilization-ending arsenals after seeing the Trinity test… which… was the correct response. We don’t know it’s possible to build superintelligences but so far we don’t have a good reason to think we can’t and we have complete certainty that humans will spend immense, immense resources getting as close as they can as fast as they can. Very different from the lasers.
- SanderNL 3y ago
  My point is that there is no Trinity test. Trinity showed nuclear devices were real. GPT shows language models are useful and can potentially be part of what might be an AGI but nobody showed how, not even in theory. You don’t have to build it, just show how it could be super intelligent.
  - ethanbond 3y ago
    Language models' potential role in AGI is a red herring. GPT shows that there's enough "there" to draw billions of dollars and thousands of the smartest researchers on the planet into the general problem space of creating intelligent machines. I suppose the closer analog would be to Szilard's chain reaction work which prompted the Manhattan Project. There might be another AI winter or 4 or 10 or 100 before superintelligence, but they will almost certainly be winters and not ends of the road. If there is any route to superintelligence and we don't annihilate ourselves by some other method first, we will get there. I don't get what the conceptual challenge with superintelligence is. Do you believe that the brain that happens to exist inside the naked apes of Earth is the absolute pinnacle of possible intelligence?
    - SanderNL 3y ago
      > I don't get what the conceptual challenge with superintelligence is. Do you believe that the brain that happens to exist inside the naked apes of Earth is the absolute pinnacle of possible intelligence? Well, first of all I do not believe that, but now you mentioned it I assume you do not believe we are the pinnacle of intelligence. Any proof? Because anybody can present literally all of Earth's life as proof that we are. It is completely unknown to us if and how any great improvements are possible. This quest shares some aspect of the quest for extraterrestial intelligence in that way. It should be out there, because it's hard to think of reasons it can't be, but yet the skies are empty. They could be on our doorstep or the universe could be devoid of intelligent life and anything in between. There's literally no way to tell at this point. I guess what I'm saying is that it's a bit early to starting beating drums.
pixl97 3y ago
>except ourselves. Good argument, lets ignore the (human) elephant in the room! >worrying about deadly quantum laser weapons If humans were shooting smaller less deadly quantum lasers out of their eyes I'd be very fucking worried that we'd make a much more powerful artificial version. Tell me why do you think humans are the pinnacle of intelligence? What was the evolutionary requirement that somehow pushed us to this level? You simply cannot answer that last question. Humans have a tiny power budget. We have a fit out of the birth canal limitation that cuts down on our brain size. We have a "don't starve to death" evolutionary pressure that was the biggest culling factor of all up to about 150 years ago. The idea that we couldn't build a better intelligence optimized system than nature is foreign to me, nature simply was not trying to achieve that goal.
- revelio 3y ago
  AI also has a power budget. It has to fit in a datacenter. Inconveniently for AI, that power budget is controlled by us.
  - pixl97 3y ago
    Do you remember the days the computing power that's in your pocket took up an entire floor of a building? Because I do. If that is somehow the only barrier between humanity and annihilation, then things don't bode well for us.
    - revelio 3y ago
      Sure. Despite all that progress, computers still have an off switch and power efficiency still matters. It actually matters more now than in the past.
      - pixl97 3y ago
        What you're arguing is "what is the minimum viable power envelope for a super intelligence". Currently that answer is "quite a lot". But for the sake of cutting out a lot of argument lets say you have a cellphone sized device that runs on battery power for 24 hours that can support a general intelligence. Lets say, again for arguments sake, there are millions of devices like this distributed in the population. Do you mind telling me how exactly you turn that off? Now we're lucky in the sense we don't have that today. AI still requires data centers inputting massive amounts of power and huge cooling bills. Maybe we'll forever require AI to take stupid large amounts of power. But at the same time, a cray super computer required stupid amounts of power and space, and your cellphone has leaps and bounds more computing power than that.
sebzim4500 3y ago
> I’d say this is like worrying about deadly quantum laser weapons that will consume the planet when we are still in the AK47 phase. Directed energy weapons will almost certainly exist eventually, to some extent they already do. The reason why it makes more sense to worry about AGI than laser weapons is that when you try to make a laser weapon but fail slightly not much happens: either you miss the target or it doesn't fire. When you try to make an aligned superintelligence and slightly fail you potentially end up with an unaligned superintelligence, hence the panic.

mitthrowaway2 3y ago

That hand-holding exists elsewhere, but I get the sense that this particular document is very short on purpose.

the8472 3y ago

The difference to doomsday weapons is that we can build the weapons first and then worry about using them. With an AGI building one alone might be sufficient. It could become smart enough to unbox itself during a training run.

SanderNL 3y ago
The LHC could have produced a black hole that swallowed the earth, so you’re also too late if that were the case. Lots of tech is like that, but it turns out we don’t get anything done if we keep fearmongering.

ethanbond 3y ago

Or it’s like worrying about an arms race toward civilization-ending arsenals after seeing the Trinity test… which… was the correct response. We don’t know it’s possible to build superintelligences but so far we don’t have a good reason to think we can’t and we have complete certainty that humans will spend immense, immense resources getting as close as they can as fast as they can. Very different from the lasers.

SanderNL 3y ago
My point is that there is no Trinity test. Trinity showed nuclear devices were real. GPT shows language models are useful and can potentially be part of what might be an AGI but nobody showed how, not even in theory. You don’t have to build it, just show how it could be super intelligent.
- ethanbond 3y ago
  Language models' potential role in AGI is a red herring. GPT shows that there's enough "there" to draw billions of dollars and thousands of the smartest researchers on the planet into the general problem space of creating intelligent machines. I suppose the closer analog would be to Szilard's chain reaction work which prompted the Manhattan Project. There might be another AI winter or 4 or 10 or 100 before superintelligence, but they will almost certainly be winters and not ends of the road. If there is any route to superintelligence and we don't annihilate ourselves by some other method first, we will get there. I don't get what the conceptual challenge with superintelligence is. Do you believe that the brain that happens to exist inside the naked apes of Earth is the absolute pinnacle of possible intelligence?
  - SanderNL 3y ago
    > I don't get what the conceptual challenge with superintelligence is. Do you believe that the brain that happens to exist inside the naked apes of Earth is the absolute pinnacle of possible intelligence? Well, first of all I do not believe that, but now you mentioned it I assume you do not believe we are the pinnacle of intelligence. Any proof? Because anybody can present literally all of Earth's life as proof that we are. It is completely unknown to us if and how any great improvements are possible. This quest shares some aspect of the quest for extraterrestial intelligence in that way. It should be out there, because it's hard to think of reasons it can't be, but yet the skies are empty. They could be on our doorstep or the universe could be devoid of intelligent life and anything in between. There's literally no way to tell at this point. I guess what I'm saying is that it's a bit early to starting beating drums.

pixl97 3y ago

>except ourselves. Good argument, lets ignore the (human) elephant in the room! >worrying about deadly quantum laser weapons If humans were shooting smaller less deadly quantum lasers out of their eyes I'd be very fucking worried that we'd make a much more powerful artificial version. Tell me why do you think humans are the pinnacle of intelligence? What was the evolutionary requirement that somehow pushed us to this level? You simply cannot answer that last question. Humans have a tiny power budget. We have a fit out of the birth canal limitation that cuts down on our brain size. We have a "don't starve to death" evolutionary pressure that was the biggest culling factor of all up to about 150 years ago. The idea that we couldn't build a better intelligence optimized system than nature is foreign to me, nature simply was not trying to achieve that goal.

revelio 3y ago
AI also has a power budget. It has to fit in a datacenter. Inconveniently for AI, that power budget is controlled by us.
- pixl97 3y ago
  Do you remember the days the computing power that's in your pocket took up an entire floor of a building? Because I do. If that is somehow the only barrier between humanity and annihilation, then things don't bode well for us.
  - revelio 3y ago
    Sure. Despite all that progress, computers still have an off switch and power efficiency still matters. It actually matters more now than in the past.
    - pixl97 3y ago
      What you're arguing is "what is the minimum viable power envelope for a super intelligence". Currently that answer is "quite a lot". But for the sake of cutting out a lot of argument lets say you have a cellphone sized device that runs on battery power for 24 hours that can support a general intelligence. Lets say, again for arguments sake, there are millions of devices like this distributed in the population. Do you mind telling me how exactly you turn that off? Now we're lucky in the sense we don't have that today. AI still requires data centers inputting massive amounts of power and huge cooling bills. Maybe we'll forever require AI to take stupid large amounts of power. But at the same time, a cray super computer required stupid amounts of power and space, and your cellphone has leaps and bounds more computing power than that.

sebzim4500 3y ago

> I’d say this is like worrying about deadly quantum laser weapons that will consume the planet when we are still in the AK47 phase. Directed energy weapons will almost certainly exist eventually, to some extent they already do. The reason why it makes more sense to worry about AGI than laser weapons is that when you try to make a laser weapon but fail slightly not much happens: either you miss the target or it doesn't fire. When you try to make an aligned superintelligence and slightly fail you potentially end up with an unaligned superintelligence, hence the panic.