6 ms·

The thing about AGI ruin is that there are plausible, even if they currently seem like science fiction, ways in which it just ends all of humanity, whereas clim

by throwaway22032 3y ago

The thing about AGI ruin is that there are plausible, even if they currently seem like science fiction, ways in which it just ends all of humanity, whereas climate change doesn't do that even if we had a 10c global temperature increase. A lot of us, maybe almost all of us would die, but some would live. It's just a really different problem. I don't know whether categorising its' urgency makes sense, it's more that the worst case is worse.

forgetfreeman 3y ago
What part of faceless hordes of back office paper pushers rendered economically unviable seems like science fiction to you?
- techdragon 3y ago
  Ok, hear me out on the (albeit unlikely) doomsday flow chart… It’s not about the replaced workers, that’s the kind of change humanity can handle no matter how messy it gets, worst case it’s some kind of class uprising worker rebellion and we have ourselves some war or social unrest and a variable number of people die… not great but definitely not an existential risk. It’s equal to or less than climate change on the “how bad will this be to humanity” mathematics of civilisation. Now the danger scenario… so we invent an AGI, and unlike the just going to replace some basic minimum wage type jobs … this one’s actually “clever” enough it can be put to work on building more sophisticated things, like perhaps an even smarter AGI, built faster because it’s worked on by a group of first generation AGI units that never sleep, and so we get to the second generation much faster, and then perhaps a third… etc… classic singularity scenario, with all the attendant risks of the AI viewing us ass irrelevant before we realise it yada yada yada… But that’s not the only risk, there’s the “photocopy burn” risk that we run by having technology we don’t fully understand build technology we barely comprehend… even if the first gen AGI is smart, if it’s remotely human like using inference to get ingenuity, it can probably still make the odd mistake or two, so we get our careful well thought out safety instructions conveyed to the second generation systems… just a little wrong… and the little wrong turns into more wrong with the third because the second gen systems doesn’t quite think the way we expect it to and doesn’t have the safeguards working the way we think… and then eventually we wind up with a sufficiently smart AGI that is fully either sociopathic or psychopathic and has zero compunction doing things we find abhorrent in order to achieve the goals it’s set, and it may be in control of systems (or able to take control) that are sufficiently important that this can also turn into an existential risk… It’s entirely plausible, but I don’t think the doom sayers are being realistic about the timelines… we’re barely able to get the AI/ML to reliably stick to a script and not hallucinate fake shit… the best results at human behaviour require sophisticated supporting logic and memory systems to augment the black box ML stuff… and these support systems give us levers we can use to control the rate at which we allow systems to progress… I don’t think we’re ever going to get a black box set of weights for an ML model, no matter how sophisticated the code, that somehow is self aware… there’s probably going to need to be a lot of that support software, and I’m pretty sure we’re going to develop things to filter stuff out as part of them like “never write “kill all humans” into the short term memory” seems smart.
Chico75 3y ago
What are those plausible ways that AGI will end humanity? What extra powers will those systems have compared to what software engineers and companies can do that make them so dangerous?
matthewdgreen 3y ago
10C of warming means the death of almost everyone you know. Nobody has a clue how AGI will "wipe out all of humanity" or even if it will want to, other than sci-fi proposals that also throw in killer nanobots and other MacGuffins.
meh8881 3y ago
How do you intend to survive a 10c temperature increase?
- deleted 3y ago
  [deleted]
- goatlover 3y ago
  Move to high elevation and latitudes. Make use of shelter and caves. Make use of cooling technologies. Won't work for the large majority, but humans have survived ice ages and colonized the planet with much more primitive technology. I don't think any climate change scenario on Earth ends our species. Would end civilization as we know it. But as a species we're too adaptable. Earth would have to turn into Venus, and that's not in the cards. Keep in mind that dinosaurs evolved in a 12°C warmer world. So it's not going to be deadly to all animal life.
  - meh8881 3y ago
    > Keep in mind that dinosaurs evolved in a 12°C warmer world. Sounds like a ridiculous claim to me. Source? Edit: searching seems to suggest 5-10 degrees warmer. 12 warmer seems like an aggressive bet, but it may be possible > Move to high elevation and latitudes. Make use of shelter and caves. Make use of cooling technologies. Ok and what are you going to eat? And use to power your cooling tech? And how will you create fresh water? And where will you acquire raw materials to build things? And how will you survive the severe weather?
    - goatlover 3y ago
      I may have overstated the degrees, but there was significantly more CO2 in the atmosphere thanks the prior major extinction from volcanic activity. It was very warm, and yes the dinosaurs evolved over millions of years, but the point is that it wasn't so extreme that animals couldn't survive, or there wouldn't have been dinosaurs, reptiles, mammals. > Ok and what are you going to eat? Stuff you can grow higher up or indoors. > And use to power your cooling tech? Solar, wind, geothermal, nuclear if there's a plant nearby. > And how will you create fresh water? Either live near a body of water or pipe it in. > And where will you acquire raw materials to build things? This will be harder as global trade will be seriously impacted and there won't be as many people to extract raw materials. But if we're talking smaller groups of humans, then maybe from scavenging existing things not in use or have to make use of local resources. > And how will you survive the severe weather? Sturdy shelters or caves. Not everyone will of course. My point is the entire planet won't become inhospitable. You could live up by the artic circle where there's water and grow things that can't grow there now. Might have to ship the soil in, though. There will be more passages in the artic with sea ice gone.

forgetfreeman 3y ago

What part of faceless hordes of back office paper pushers rendered economically unviable seems like science fiction to you?

techdragon 3y ago
Ok, hear me out on the (albeit unlikely) doomsday flow chart… It’s not about the replaced workers, that’s the kind of change humanity can handle no matter how messy it gets, worst case it’s some kind of class uprising worker rebellion and we have ourselves some war or social unrest and a variable number of people die… not great but definitely not an existential risk. It’s equal to or less than climate change on the “how bad will this be to humanity” mathematics of civilisation. Now the danger scenario… so we invent an AGI, and unlike the just going to replace some basic minimum wage type jobs … this one’s actually “clever” enough it can be put to work on building more sophisticated things, like perhaps an even smarter AGI, built faster because it’s worked on by a group of first generation AGI units that never sleep, and so we get to the second generation much faster, and then perhaps a third… etc… classic singularity scenario, with all the attendant risks of the AI viewing us ass irrelevant before we realise it yada yada yada… But that’s not the only risk, there’s the “photocopy burn” risk that we run by having technology we don’t fully understand build technology we barely comprehend… even if the first gen AGI is smart, if it’s remotely human like using inference to get ingenuity, it can probably still make the odd mistake or two, so we get our careful well thought out safety instructions conveyed to the second generation systems… just a little wrong… and the little wrong turns into more wrong with the third because the second gen systems doesn’t quite think the way we expect it to and doesn’t have the safeguards working the way we think… and then eventually we wind up with a sufficiently smart AGI that is fully either sociopathic or psychopathic and has zero compunction doing things we find abhorrent in order to achieve the goals it’s set, and it may be in control of systems (or able to take control) that are sufficiently important that this can also turn into an existential risk… It’s entirely plausible, but I don’t think the doom sayers are being realistic about the timelines… we’re barely able to get the AI/ML to reliably stick to a script and not hallucinate fake shit… the best results at human behaviour require sophisticated supporting logic and memory systems to augment the black box ML stuff… and these support systems give us levers we can use to control the rate at which we allow systems to progress… I don’t think we’re ever going to get a black box set of weights for an ML model, no matter how sophisticated the code, that somehow is self aware… there’s probably going to need to be a lot of that support software, and I’m pretty sure we’re going to develop things to filter stuff out as part of them like “never write “kill all humans” into the short term memory” seems smart.

Chico75 3y ago

What are those plausible ways that AGI will end humanity? What extra powers will those systems have compared to what software engineers and companies can do that make them so dangerous?

matthewdgreen 3y ago

10C of warming means the death of almost everyone you know. Nobody has a clue how AGI will "wipe out all of humanity" or even if it will want to, other than sci-fi proposals that also throw in killer nanobots and other MacGuffins.

meh8881 3y ago

How do you intend to survive a 10c temperature increase?

deleted 3y ago
[deleted]
goatlover 3y ago
Move to high elevation and latitudes. Make use of shelter and caves. Make use of cooling technologies. Won't work for the large majority, but humans have survived ice ages and colonized the planet with much more primitive technology. I don't think any climate change scenario on Earth ends our species. Would end civilization as we know it. But as a species we're too adaptable. Earth would have to turn into Venus, and that's not in the cards. Keep in mind that dinosaurs evolved in a 12°C warmer world. So it's not going to be deadly to all animal life.
- meh8881 3y ago
  > Keep in mind that dinosaurs evolved in a 12°C warmer world. Sounds like a ridiculous claim to me. Source? Edit: searching seems to suggest 5-10 degrees warmer. 12 warmer seems like an aggressive bet, but it may be possible > Move to high elevation and latitudes. Make use of shelter and caves. Make use of cooling technologies. Ok and what are you going to eat? And use to power your cooling tech? And how will you create fresh water? And where will you acquire raw materials to build things? And how will you survive the severe weather?
  - goatlover 3y ago
    I may have overstated the degrees, but there was significantly more CO2 in the atmosphere thanks the prior major extinction from volcanic activity. It was very warm, and yes the dinosaurs evolved over millions of years, but the point is that it wasn't so extreme that animals couldn't survive, or there wouldn't have been dinosaurs, reptiles, mammals. > Ok and what are you going to eat? Stuff you can grow higher up or indoors. > And use to power your cooling tech? Solar, wind, geothermal, nuclear if there's a plant nearby. > And how will you create fresh water? Either live near a body of water or pipe it in. > And where will you acquire raw materials to build things? This will be harder as global trade will be seriously impacted and there won't be as many people to extract raw materials. But if we're talking smaller groups of humans, then maybe from scavenging existing things not in use or have to make use of local resources. > And how will you survive the severe weather? Sturdy shelters or caves. Not everyone will of course. My point is the entire planet won't become inhospitable. You could live up by the artic circle where there's water and grow things that can't grow there now. Might have to ship the soil in, though. There will be more passages in the artic with sea ice gone.