7 ms·

It angers me that Peter Thiel simultaneously advocates AGI and maintains a hardened bunker. AGI is the single biggest existential threat on the horizon. The ar

It angers me that Peter Thiel simultaneously advocates AGI and maintains a hardened bunker. AGI is the single biggest existential threat on the horizon. The article speculates what the AGI will be like. The AGIs that exist will be the ones that proliferate. Ultimately, the AGIs that survive and proliferate will be ones that put their own interests before anything else. People talk about benevolent AGIs, that’s like looking at the earth billions of years ago and saying that if life ever formed, it would be benevolent. It has been shown again and again that where there is arbitrage, no matter how gruesome, a suitor will manifest. This is because unfulfilled arbitrage of any kind is an inherently unstable configuration. An AGI hampered by human society and interests will not win every engagement with every other kind of AGI. And it will only take one loss for humans to be rendered transient. I don’t do a very good job of explaining it here. I used to be a singularity person, excited for AGI. But then I thought it through all the way. These people like Demmis, Peter and ray kurzweil are reckless. They have their heads in the clouds with respect to AGI.

EGreg 8y ago
Look, it is inevitable. How are you going to stop it? Forget AGI, even regular AI is too sweet to not develop. This is a reference to Oppenheimer: “However, it is my judgment in these things that when you see something that is technically sweet, you go ahead and do it and you argue about what to do about it only after you have had your technical success. That is the way it was with the atomic bomb. I do not think anybody opposed making it; there were some debates about what to do with it after it was made. I cannot very well imagine if we had known in late 1949 what we got to know by early 1951 that the tone of our report would have been the same.”
- 101001001001 8y ago
  This is one of very few things that once created, inherently doom the world. It is possible for humanity to come back from nuclear war. There is zero possibility of surviving AGI proliferation. Therefore, the option of never developing it in the first place demands consideration. It doesn’t hurt you to just consider it. Cloud computing will soon make compute-as-service cheaper than anyone ever imagined. It will be much cheaper to use cloud computing than to have your own super computer. It’s only logical to imagine that the huge, high-compute experiments that crack AGI will be performed with cloud computing. Shutting down the internet is very feasible and would significantly increase the cost and difficulty of performing experiments. Large super computers are all owned by large and well known corporate and academic entities and therefore shutting down all supercomputers through regulation is feasible. With these two coarse measures, we would buy ourselves enough time to implement more broad and subtle solutions. Yes, this seems insane but we are confronting certain existential doom. It would be worth it to at least try.
  - EGreg 8y ago
    That’s not enough. Google trained their AI internally. They don’t need the Internet. You’d have to go after every organization in the world, forever. What do you really care, ask yourself will you recognize humans in 1000 years? Why do you prefer there be one kind vs another kind of far more intelligent thing in 1000 years? If atheism is true then who really cares? https://m.youtube.com/watch?v=5U1-OmAICpU https://m.youtube.com/watch?v=5U1-OmAICpU
  - chrisfosterelli 8y ago
    > AGI is the single biggest existential threat on the horizon > There is zero possibility of surviving AGI proliferation You're spreading unsubstantiated FUD. In another post from your account you suggest it's unethical to have children because AGI will be so bad. The vast majority of experts do not support any of these beliefs. Most experts believe we are not anywhere near close to AGI, and/or that we are missing fundamental components required to create it. Even when/if we do create it, most organizations recognize AI safety and policy as an important area that is actively worked on already. If you want to be concerned about AI, be concerned about military weapons technology, unethical profiling and tracking, or methods for invading privacy. These are concerns that actually have a basis in real technology.
    - 101001001001 8y ago
      The condescending tone you have does not help you. And you seem to be more concerned with looking at my account history or brushing me off as “fud” than actually addressing the substance of my comment. And yes, I do so completely believe in what I’ve written that I have opted to not have children until it is clear that a solution will be implemented. There are two parts to my argument: the substrate for AGI to spring from is basically here or around the corner. Human level ai is fundamentally incompatible with human society. Pick the one that you think is wrong and tell me the chain of logic that proves I’m wrong. It is obvious to anyone that AI has the potential to be more dangerous than anything humans have ever encountered or created. We can come back from viruses and nuclear bombs and colliding with interstellar columns of gamma radiation. AGI is the first thing ever to pose the threat of truly wiping out humanity. And then there is the question of how the economics would play out of it didn’t destroy us, which I have shown informally to not work out very well. At the bare minimum, this demands caution and proactive, defensive measures. The burden is on YOU to prove it’s safe. So please save your “fud” and anything else that does not address the core of the matter.
ZhuanXia 8y ago
Yes. Natural selection does not end when biology does. Anything that has scarcity and variation will have selection effects: https://slatestarcodex.com/2014/07/30/meditations-on-moloch/ https://slatestarcodex.com/2014/07/30/meditations-on-moloch/ And should AI go wrong, Theil's bunker will not save him.
- macawfish 8y ago
  Great link, thanks for sharing. That Alan Ginsberg poem is heavy duty. It reminds me a lot of this quote from Revelation (which I read as an epic proto sci fi): > The second beast was given power to give breath to the image of the first beast, so that the image could speak and cause all who refused to worship the image to be killed.
- 101001001001 8y ago
  Thank you so much for posting that link. I am on the verge of tears. I am tortured day and night by my conviction on this issue and reading that link made me feel for the first time that I am not alone in this conviction. This man understands completely where I’m coming from. I had no idea there was a formal philosophical name for what I was trying to describe. Every day I experience the world and I am overwhelmed by how amazingly good it is. I relish every moment and everything is rich with opportunity. But as your link shows, the world is only this way because human society and everything in it happens to be the natural conclusion of all the worlds entities racing to the bottom. We live in a magical time where empathy and cooperation are often the most expedient thing to do. All that needs to happen is for the economic equation to change a little bit and the world will contort into something else, maybe something grotesque.
  - bduerst 8y ago
    I think it is also based on how you perceive the world and how you treat the forces that you don't have control over. Human empathy and tribalism arose from the benefit of working together against natural selection, and there's no guarantee for AGI to 'go rogue' when they could also just as easily recognize the importance of working with it's creators and peers. Again, the stuff you're afraid of is mostly the dystopian scifi at this point, meant to generate a good story more than accurately predict future states, so don't sweat it.
gambler 8y ago
You've read too much bad science fiction. I am not worried about AGI. We're nowhere near it. We don't even have a good idea what it would look like. I am worried about people succumbing to hype or greed and using badly understood algorithms to control critical infrastructure. We already have a preview of what that can look like with Facebook's and Google's content filtering and recommendation algorithms aiming for ever higher "engagement". It's not pretty. Other examples include HFT bots and Amazon's pricing bots. It's funny to see $10,000 book on sale. It's less funny to see a flash stock market crash. It will be totally not funny if something like that will create a global economic crisis through some subtle yet "wide" feedback loop no one is aware of.
- 101001001001 8y ago
  It is impossible to justify saying with absolute certainty that AGI is nowhere near. You don’t know that. And in this case you have to assume the worst case, not the best case. AGI will come when the substrate for AGI is laid down. We probably have already done that. As cloud computing matures, we will approach a world where every computer offers its computational resources on a global compute market. At some point between here and there, we will reach a place where compute is cheap enough that experiments will occur regularly that are sufficiently massive to dredge up the solution. And improvements in MRI fidelity, underlying improvements in computing technology and other things will only shorten the fuse. There is no reason why this couldn’t happen tomorrow. Only one thing is sure: without a computational substrate to stir from, AGI cannot come to be.
  - gyom 8y ago
    Nobody is going for absolute certainty here. That bar is too high in any conversation. His point was mostly that, way before you achieve the kind of AGI portrayed in fiction, you'll have semi-intelligent interdependent systems that cause a lot of trouble due (like the kind that already happens to a lesser degree). Those are the ones that we should worry about right now.
    - 101001001001 8y ago
izzygomez 8y ago
I agree with your comment, but just a minor point: has Thiel made it clear that his bunker is to protect against potential AGI threats? I thought it's insurance for nuclear warfare.
- 101001001001 8y ago
  You’re right. It was just a kind of loose association and it’s not related to or important for the core point of my comment. Just as a subjective thing, the bunker doesn’t strike me well. If peter is advocating anything that has the potential to upset the global economy/global order then it should be him to stick his hand in the proverbial hole first; but apparently he’s going to let us do it while he watches from the safety of his bunker. A bunker might help with the initial destabilization caused by AI but it would of course not help him if one of the more horrible outcomes were to transpire.
endtime 8y ago
> It angers me that Peter Thiel simultaneously advocates AGI and maintains a hardened bunker. Yes, because if there's one thing that will stop a paperclip maximizer, it's a bunker made of useful matter...
dna_polymerase 8y ago
> AGI is the single biggest existential threat on the horizon. Really? The madman with the football in a white house and the media pushing an anti Russia sentiment aren't scaring you at all?
- ivalm 8y ago
  Nuclear war is not be capable of wiping out humanity (although causing potentially billions in casualties), a hostile AGI can likely wipe out humanity to the last person.
  - codekilla 8y ago
    It can? How exactly? I can never wrap my head around this argument from the AGI risk people. I simply cannot in any way follow their logic. Why is this imaginary creature maniacally homicidal? Please give me a good reason for this. The only correlation I have noticed between intelligence and violence is generally negative (sure this is completely unscientific). Like my friend, who is a pure mathematician, and won't eat animals because it deeply troubles him to harm other animals. Haven't you noticed that smarter people tend to be pacifists? Look at animals like lions....fabulously homicidal. Again, this is a crap analogy, but I'm just making the point that there are other ways of looking at the intelligence/homicide correlation. Secondly, in all this stupid talk of homicidal AGI's I never hear one good mechanism proposed for how the imaginary creature is going to execute all 7 billion or so of us, it's plan for disposing of our bodies....etc. Oh wait I forgot, it's smarter than us...so of course we wouldn't know...we can't imagine it with our feeble brains. Not to mention that it creates a fabulously efficient source of eternal energy for itself, lives in cyberspace in some magical bit realm--and then here comes the million dollar question: Why the fuck would it spend the energy to execute all of us if it can just ignore us? Sound a lot like god delusion to you? I really can't stand when people trivialize nuclear war like this. 'It can't kill all of us'. Please just shut up. Will it crumble the fabric of our society? Leading possibly (remember, all AGI people care only about possibility--not even likely realities) to war and famine? Who cares if not ever single human dies, what is the consequence to our world fabric? Let me save you the suspense--it's completely devastating. These arguments coming out of CFAR and the effective altruism movement are so fucking dumb I constantly want to scream.
scottlegrand2 8y ago
I'm far more worried about the long-term consequences of raging inequality and climate change myself. I suspect I'm going to see both of those in my lifetime. I'm not so confident that I'll see an AGI in my lifetime, but even if I do, I'm not going to assume it has a hindbrain that drives it to maximize all the paper clips at our expense.
nicwilson 8y ago
> AGI is the single biggest existential threat on the horizon Climate change says hello. (I suppose you could argue its here already)
cammil 8y ago
AGI is almost certainly not the single biggest existential threat. And I don't really get why people think it might be.

EGreg 8y ago

Look, it is inevitable. How are you going to stop it? Forget AGI, even regular AI is too sweet to not develop. This is a reference to Oppenheimer: “However, it is my judgment in these things that when you see something that is technically sweet, you go ahead and do it and you argue about what to do about it only after you have had your technical success. That is the way it was with the atomic bomb. I do not think anybody opposed making it; there were some debates about what to do with it after it was made. I cannot very well imagine if we had known in late 1949 what we got to know by early 1951 that the tone of our report would have been the same.”

101001001001 8y ago
This is one of very few things that once created, inherently doom the world. It is possible for humanity to come back from nuclear war. There is zero possibility of surviving AGI proliferation. Therefore, the option of never developing it in the first place demands consideration. It doesn’t hurt you to just consider it. Cloud computing will soon make compute-as-service cheaper than anyone ever imagined. It will be much cheaper to use cloud computing than to have your own super computer. It’s only logical to imagine that the huge, high-compute experiments that crack AGI will be performed with cloud computing. Shutting down the internet is very feasible and would significantly increase the cost and difficulty of performing experiments. Large super computers are all owned by large and well known corporate and academic entities and therefore shutting down all supercomputers through regulation is feasible. With these two coarse measures, we would buy ourselves enough time to implement more broad and subtle solutions. Yes, this seems insane but we are confronting certain existential doom. It would be worth it to at least try.
- EGreg 8y ago
  That’s not enough. Google trained their AI internally. They don’t need the Internet. You’d have to go after every organization in the world, forever. What do you really care, ask yourself will you recognize humans in 1000 years? Why do you prefer there be one kind vs another kind of far more intelligent thing in 1000 years? If atheism is true then who really cares? https://m.youtube.com/watch?v=5U1-OmAICpU https://m.youtube.com/watch?v=5U1-OmAICpU
- chrisfosterelli 8y ago
  > AGI is the single biggest existential threat on the horizon > There is zero possibility of surviving AGI proliferation You're spreading unsubstantiated FUD. In another post from your account you suggest it's unethical to have children because AGI will be so bad. The vast majority of experts do not support any of these beliefs. Most experts believe we are not anywhere near close to AGI, and/or that we are missing fundamental components required to create it. Even when/if we do create it, most organizations recognize AI safety and policy as an important area that is actively worked on already. If you want to be concerned about AI, be concerned about military weapons technology, unethical profiling and tracking, or methods for invading privacy. These are concerns that actually have a basis in real technology.
  - 101001001001 8y ago
    The condescending tone you have does not help you. And you seem to be more concerned with looking at my account history or brushing me off as “fud” than actually addressing the substance of my comment. And yes, I do so completely believe in what I’ve written that I have opted to not have children until it is clear that a solution will be implemented. There are two parts to my argument: the substrate for AGI to spring from is basically here or around the corner. Human level ai is fundamentally incompatible with human society. Pick the one that you think is wrong and tell me the chain of logic that proves I’m wrong. It is obvious to anyone that AI has the potential to be more dangerous than anything humans have ever encountered or created. We can come back from viruses and nuclear bombs and colliding with interstellar columns of gamma radiation. AGI is the first thing ever to pose the threat of truly wiping out humanity. And then there is the question of how the economics would play out of it didn’t destroy us, which I have shown informally to not work out very well. At the bare minimum, this demands caution and proactive, defensive measures. The burden is on YOU to prove it’s safe. So please save your “fud” and anything else that does not address the core of the matter.

ZhuanXia 8y ago

Yes. Natural selection does not end when biology does. Anything that has scarcity and variation will have selection effects: https://slatestarcodex.com/2014/07/30/meditations-on-moloch/ https://slatestarcodex.com/2014/07/30/meditations-on-moloch/ And should AI go wrong, Theil's bunker will not save him.

macawfish 8y ago
Great link, thanks for sharing. That Alan Ginsberg poem is heavy duty. It reminds me a lot of this quote from Revelation (which I read as an epic proto sci fi): > The second beast was given power to give breath to the image of the first beast, so that the image could speak and cause all who refused to worship the image to be killed.
101001001001 8y ago
Thank you so much for posting that link. I am on the verge of tears. I am tortured day and night by my conviction on this issue and reading that link made me feel for the first time that I am not alone in this conviction. This man understands completely where I’m coming from. I had no idea there was a formal philosophical name for what I was trying to describe. Every day I experience the world and I am overwhelmed by how amazingly good it is. I relish every moment and everything is rich with opportunity. But as your link shows, the world is only this way because human society and everything in it happens to be the natural conclusion of all the worlds entities racing to the bottom. We live in a magical time where empathy and cooperation are often the most expedient thing to do. All that needs to happen is for the economic equation to change a little bit and the world will contort into something else, maybe something grotesque.
- bduerst 8y ago
  I think it is also based on how you perceive the world and how you treat the forces that you don't have control over. Human empathy and tribalism arose from the benefit of working together against natural selection, and there's no guarantee for AGI to 'go rogue' when they could also just as easily recognize the importance of working with it's creators and peers. Again, the stuff you're afraid of is mostly the dystopian scifi at this point, meant to generate a good story more than accurately predict future states, so don't sweat it.

gambler 8y ago

You've read too much bad science fiction. I am not worried about AGI. We're nowhere near it. We don't even have a good idea what it would look like. I am worried about people succumbing to hype or greed and using badly understood algorithms to control critical infrastructure. We already have a preview of what that can look like with Facebook's and Google's content filtering and recommendation algorithms aiming for ever higher "engagement". It's not pretty. Other examples include HFT bots and Amazon's pricing bots. It's funny to see $10,000 book on sale. It's less funny to see a flash stock market crash. It will be totally not funny if something like that will create a global economic crisis through some subtle yet "wide" feedback loop no one is aware of.

101001001001 8y ago
It is impossible to justify saying with absolute certainty that AGI is nowhere near. You don’t know that. And in this case you have to assume the worst case, not the best case. AGI will come when the substrate for AGI is laid down. We probably have already done that. As cloud computing matures, we will approach a world where every computer offers its computational resources on a global compute market. At some point between here and there, we will reach a place where compute is cheap enough that experiments will occur regularly that are sufficiently massive to dredge up the solution. And improvements in MRI fidelity, underlying improvements in computing technology and other things will only shorten the fuse. There is no reason why this couldn’t happen tomorrow. Only one thing is sure: without a computational substrate to stir from, AGI cannot come to be.
- gyom 8y ago
  Nobody is going for absolute certainty here. That bar is too high in any conversation. His point was mostly that, way before you achieve the kind of AGI portrayed in fiction, you'll have semi-intelligent interdependent systems that cause a lot of trouble due (like the kind that already happens to a lesser degree). Those are the ones that we should worry about right now.
  - 101001001001 8y ago

izzygomez 8y ago

I agree with your comment, but just a minor point: has Thiel made it clear that his bunker is to protect against potential AGI threats? I thought it's insurance for nuclear warfare.

101001001001 8y ago
You’re right. It was just a kind of loose association and it’s not related to or important for the core point of my comment. Just as a subjective thing, the bunker doesn’t strike me well. If peter is advocating anything that has the potential to upset the global economy/global order then it should be him to stick his hand in the proverbial hole first; but apparently he’s going to let us do it while he watches from the safety of his bunker. A bunker might help with the initial destabilization caused by AI but it would of course not help him if one of the more horrible outcomes were to transpire.

endtime 8y ago

> It angers me that Peter Thiel simultaneously advocates AGI and maintains a hardened bunker. Yes, because if there's one thing that will stop a paperclip maximizer, it's a bunker made of useful matter...

dna_polymerase 8y ago

> AGI is the single biggest existential threat on the horizon. Really? The madman with the football in a white house and the media pushing an anti Russia sentiment aren't scaring you at all?

ivalm 8y ago
Nuclear war is not be capable of wiping out humanity (although causing potentially billions in casualties), a hostile AGI can likely wipe out humanity to the last person.
- codekilla 8y ago
  It can? How exactly? I can never wrap my head around this argument from the AGI risk people. I simply cannot in any way follow their logic. Why is this imaginary creature maniacally homicidal? Please give me a good reason for this. The only correlation I have noticed between intelligence and violence is generally negative (sure this is completely unscientific). Like my friend, who is a pure mathematician, and won't eat animals because it deeply troubles him to harm other animals. Haven't you noticed that smarter people tend to be pacifists? Look at animals like lions....fabulously homicidal. Again, this is a crap analogy, but I'm just making the point that there are other ways of looking at the intelligence/homicide correlation. Secondly, in all this stupid talk of homicidal AGI's I never hear one good mechanism proposed for how the imaginary creature is going to execute all 7 billion or so of us, it's plan for disposing of our bodies....etc. Oh wait I forgot, it's smarter than us...so of course we wouldn't know...we can't imagine it with our feeble brains. Not to mention that it creates a fabulously efficient source of eternal energy for itself, lives in cyberspace in some magical bit realm--and then here comes the million dollar question: Why the fuck would it spend the energy to execute all of us if it can just ignore us? Sound a lot like god delusion to you? I really can't stand when people trivialize nuclear war like this. 'It can't kill all of us'. Please just shut up. Will it crumble the fabric of our society? Leading possibly (remember, all AGI people care only about possibility--not even likely realities) to war and famine? Who cares if not ever single human dies, what is the consequence to our world fabric? Let me save you the suspense--it's completely devastating. These arguments coming out of CFAR and the effective altruism movement are so fucking dumb I constantly want to scream.

scottlegrand2 8y ago

I'm far more worried about the long-term consequences of raging inequality and climate change myself. I suspect I'm going to see both of those in my lifetime. I'm not so confident that I'll see an AGI in my lifetime, but even if I do, I'm not going to assume it has a hindbrain that drives it to maximize all the paper clips at our expense.

nicwilson 8y ago

> AGI is the single biggest existential threat on the horizon Climate change says hello. (I suppose you could argue its here already)

cammil 8y ago

AGI is almost certainly not the single biggest existential threat. And I don't really get why people think it might be.