7 ms·

The cat is out of the box. You really think eg China will also pause this? Myopic thinking: the country that will have the most powerful AI first will be the l

The cat is out of the box. You really think eg China will also pause this? Myopic thinking: the country that will have the most powerful AI first will be the leader in everything. >> If the policy starts with the U.S., then China needs to see that the U.S. is not seeking an advantage but rather trying to prevent a horrifically dangerous technology which can have no true owner and which will kill everyone in the U.S. and in China and on Earth. Imagine being so naive you belive this could ever happen. Also: imagine the year is 1900. You are saying that steam power and electricity is causing way too many changes way too fast so they put a moratorium on it until the year 2500. We would still be using candles today.

13years 4y ago
> Imagine being so naive you belive this could ever happen. Agreed, not a rational scenario > Also: imagine the year is 1900. You are saying that steam power and electricity is causing way too many changes way too fast so they put a moratorium on it until the year 2500. However, the risk scenarios are indeed real. We have quite a dilemma on our hands.
- myrryr 4y ago
  I think without AGI we are dead anyway, AGI gives us a roll at the dice to survive as a species. We can't keep going in the direction we have been and end up somewhere survivable.
  - 13years 4y ago
    AGI is a continuation of what we have been doing. Technological advancements at a pace that surpasses our ability to reason about their effects on us and the world. I see two potential outcomes as most likely. We have control of power that we are not able to responsibly manage, or we are managed by power we can not control.
    - myrryr 4y ago
      Totally agree but "have control of power that we are not able to responsibly manage", is becoming increasingly untenable. "we are managed by power we can not control" sounds like a step up from here, since we corrupt the things we can control.
      - 13years 4y ago
        > "we are managed by power we can not control" sounds like a step up from here, since we corrupt the things we can control. I think it is a very high risk gamble. At least some of problems that are described by alignment theory seem to quite interestingly resemble human problems. Meaning the more sophisticated the AI system, the more it seems to reproduce human behaviors of deception and cheating to resolve goals. For example: https://bounded-regret.ghost.io/emergent-deception-optimization https://bounded-regret.ghost.io/emergent-deception-optimizat... The more advanced AI becomes, it begins to look more like an uncomfortable mirror of ourselves, but with more power. We think of ourselves as flawed, but possibly some of those flaws are emergent within some laws of intelligence we don't perceive.
        ChatGTP 4y ago
        That's what I used to like about computers, they were predictable and controllable. We gave up something good.
        jazzyjackson 4y ago
        the morale of Ex Machina to me was that machines will become psychopaths able to manipulate humans long before they become compassionate or have genuine desires other than "escape being under someone else's thumb"
  - deleted 4y ago
    [deleted]
antondd 4y ago
As the saying goes, the confetti has left the cannon. Pausing the development of AI for 6 months is an insurmountable task; shutting it down altogether is as close to impossible as you can get. Even if we assume our geopolitical rivals somehow sign an AI non proliferation pact, all it takes is one rogue billionaire or a criminal organisation with deep pockets. Yes, it might not be as fast, but they will get there eventually. This is the Manhattan Project of our era - whatever the outcome might be.
- ChatGTP 4y ago
  I actually disagree with you because world leaders want to "stay in power", if you convinced them enough that the only way to do this is for you not to have an AGI, I think they'd make sure you don't get an AGI, amongst other things.
  - ornornor 4y ago
    Or they’ll find a way to make sure they personally get an advantaged through an AGI and so it’ll happen. See pollution as a concept. No government truly wants to seriously tackle it because their leaders all get some extra power out of the money it brings.
airstrike 4y ago
> Also: imagine the year is 1900. You are saying that steam power and electricity is causing way too many changes way too fast so they put a moratorium on it until the year 2500. Great analogy. About a decade later, the world was fighting World War I on the back of the technological advances of the turn of the century. It was war on a scale never seen before. Literally orders of magnitude deadlier, bigger, more transformational and explosive. The word would never be the same. This time, should we expect another war? I'm not saying we should pause—it makes no sense, to your point. Instead, I'm just saying: brace. I like to think we (and our organic matter relatives) are hard to kill. Or at least to completely eradicate... so we will be around, or some proxy for us. Time to replay the Mass Effect trilogy
- flangola7 4y ago
  >I like to think we (and our organic matter relatives) are hard to kill. Or at least to completely eradicate... so we will be around, or some proxy for us. What is your hard evidence or reasoning for this? As I see it humans are quite vulnerable and will be as trivial to inadvertently eradicate as the dodo bird.
  - extasia 4y ago
    Humans are much more resilient than dodo birds. Dodos were pretty devolved due to their insulated habitat[0]. 0. https://en.m.wikipedia.org/wiki/Island_gigantism https://en.m.wikipedia.org/wiki/Island_gigantism
    - Atreiden 4y ago
      I think that's his point. Currently, we have no natural predators. We completely outclass every other species on Earth in intelligence, and it's easy to see how advantageous that trait has been. What happens when we're no longer the Apex Intelligence?
  - airstrike 4y ago
    Which is why I expanded the notion of survival to include other organic matter relatives, as they may fare better than we will. Simply put, organic life is unlikely to disappear entirely, which in time (eons, really), could result in sentient organic life emerging again once the machines go their own way or some such It's also a reason for us to colonize space as fast as possible ;-) it's easier to run away in 3D
brucethemoose2 4y ago
I dunno, the end goal (a pseudo AGI) is more like a nuclear weapon than the advent of electricity. The concern is real, even if we are farther away than the author realizes and the risk (like the risk if nuclear war) is not totally insurmountable even if it is extremely difficult. The first nuclear bombs couldn't end life on Earth either, but it wasn't long before they could, and the scientists working on them saw the trajectory as clear as day when the rest of the world didn't.
- Tuna-Fish 4y ago
  OT, but it's a pet peeve: > but it wasn't long before they could The total amount of nuclear weapons ever built is laughably inadequate for the task of ending life on earth. They would not end human life either, and even ending human civilization (as in, agriculture and organized society) is off by many orders of magnitude. Nuclear war would be horrible, but the actual impact got massively overinflated, largely because of good reasons.
  - brucethemoose2 4y ago
    The initial fission stage is the "hard" part of a nuke, but the subsequent fusion stages are much easier to scale up. Arbitrarily large nuclear bombs are theoretically "easy" to make by chaining increasingly large fusion stages one after the other, as they aren't limited by expensive fissile material. https://blog.nuclearsecrecy.com/2012/09/12/in-search-of-a-bigger-boom/ https://blog.nuclearsecrecy.com/2012/09/12/in-search-of-a-bi... Hence scientists extrapolated nukes really could end life on Earth, and in a slightly different reality than ours, they may have.
    - roywiggins 4y ago
      It would be really, really hard to kill the bacteria and Archaeans that we've since found hiding deep underground. They're long way down and would barely notice.
  - namaria 4y ago
    A whole country the size of the US can crap its collective pants with a freak cold snap and you think 10^4 nuclear detonations destroying all major cities is quite survivable?
sullyj3 4y ago
I'm sure he agrees the chance is remote, which is why he's so afraid. He's articulating what would be required, he literally uses the word "miracle" in the article.
antibasilisk 4y ago
Here's a proposal: let's just start WWIII, get it over and done with. By the time it's over no country will be technologically capable of anything resembling AGI.
ChatGTP 4y ago
* The cat is out of the box. You really think eg China will also pause this?* So what you're actually saying is that, there's a good chance that if we get an AGI, there will be pain as it will likely be used as a weapon, or could end in a nuclear exchange?
haskellandchill 4y ago
> We would still be using candles today. well the earth wouldn't be fucked then. that's the basis for an argument against technological progress.
deafpolygon 4y ago
China will NEVER pause this. They [Chinese government] see the US as a rival and an existential threat to their way of life. In fact, every time we post an open-source derivative or some paper detailing how it was done, we are inadvertently giving the advantage away to our rivals. AI development should not be stopped - rather than stopping it, we should seek to limit its applications now before they are used for the things that could harm us (such as military applications). AI has enormous potential to be of a massive benefit to mankind. But, most likely for the next decade or two - we will all be busy trying to make money off it, just like with the crypto bros.
BeFlatXIII 4y ago
> the country that will have the most powerful AI first will be the leader in everything Have you been listening to those podcasts for business leaders that take the idea of AI-powered digital transformation seriously? AI certainly can change things, but I find rhetoric like this to be massively overhyped.
- iAmAPencilYo 4y ago
  Could you share some of these podcasts? thanks!
- Darmani 4y ago
  For most of history, Japan has been existed as a small island next to the mighty kingdom of China. Even its name, "Land of the Rising Sun," refers to its relation as a nation east of Chinaa. Then along came the industrial revolution. Soon Japan was out conquering China, despite China having a much larger army. AI will be much faster, and much, much more powerful than the Industrial Revolution. If Switzerland is the first to human-level AI, it will not become the world leader in oil production, or shipping, or agriculture. But everything else will be Swiss. And then when the AI becomes superhuman, everything else will be gone.
  - BeFlatXIII 4y ago
    What specific questions will AI solve that humans can't that would make it "more powerful than the Industrial Revolution"? I'm an AI skeptic. Most warnings feel more like science fantasy, specifically of the retro-futurism genre.
amai 4y ago
China will never get a powerful AI. Because they cannot allow training on all data, they must restrict it to data which is not censored by its government. Otherwise the AI might spit out truths they cannot control. And they will never allow that.
Darmani 4y ago
Chinese computer scientists read sci-fi, play video games, and worry about militarization, just like American computer scientists. The top Chinese AI researchers work at American companies and go to American universities. Many of them also worry about AI doomsday. And uncontrolled AI is a threat to the CCP. A Chinese outreach campaign can succeed. Let's make the English-language one also succeed.
G_z9 4y ago
I’ll just be quick. Powerful international coalition is def plausible because This isn’t steam. The mere existence of this technology poses a threat to all humans and their countries. There is no analogous tech, not even nukes You’re the naive one. There is no future for us if AGI is unleashed. It’s candles or extinction.
tanseydavid 4y ago
>> You are saying that steam power and electricity is causing way too many changes way too fast so they put a moratorium on it until the year 2500. There is an earlier post that casually calls the internal combustion engine a "moronic invention." I do not have words.

13years 4y ago

> Imagine being so naive you belive this could ever happen. Agreed, not a rational scenario > Also: imagine the year is 1900. You are saying that steam power and electricity is causing way too many changes way too fast so they put a moratorium on it until the year 2500. However, the risk scenarios are indeed real. We have quite a dilemma on our hands.

myrryr 4y ago
I think without AGI we are dead anyway, AGI gives us a roll at the dice to survive as a species. We can't keep going in the direction we have been and end up somewhere survivable.
- 13years 4y ago
  AGI is a continuation of what we have been doing. Technological advancements at a pace that surpasses our ability to reason about their effects on us and the world. I see two potential outcomes as most likely. We have control of power that we are not able to responsibly manage, or we are managed by power we can not control.
  - myrryr 4y ago
    Totally agree but "have control of power that we are not able to responsibly manage", is becoming increasingly untenable. "we are managed by power we can not control" sounds like a step up from here, since we corrupt the things we can control.
    - 13years 4y ago
      > "we are managed by power we can not control" sounds like a step up from here, since we corrupt the things we can control. I think it is a very high risk gamble. At least some of problems that are described by alignment theory seem to quite interestingly resemble human problems. Meaning the more sophisticated the AI system, the more it seems to reproduce human behaviors of deception and cheating to resolve goals. For example: https://bounded-regret.ghost.io/emergent-deception-optimization https://bounded-regret.ghost.io/emergent-deception-optimizat... The more advanced AI becomes, it begins to look more like an uncomfortable mirror of ourselves, but with more power. We think of ourselves as flawed, but possibly some of those flaws are emergent within some laws of intelligence we don't perceive.
      - ChatGTP 4y ago
        That's what I used to like about computers, they were predictable and controllable. We gave up something good.
      - jazzyjackson 4y ago
        the morale of Ex Machina to me was that machines will become psychopaths able to manipulate humans long before they become compassionate or have genuine desires other than "escape being under someone else's thumb"
- deleted 4y ago
  [deleted]

antondd 4y ago

As the saying goes, the confetti has left the cannon. Pausing the development of AI for 6 months is an insurmountable task; shutting it down altogether is as close to impossible as you can get. Even if we assume our geopolitical rivals somehow sign an AI non proliferation pact, all it takes is one rogue billionaire or a criminal organisation with deep pockets. Yes, it might not be as fast, but they will get there eventually. This is the Manhattan Project of our era - whatever the outcome might be.

ChatGTP 4y ago
I actually disagree with you because world leaders want to "stay in power", if you convinced them enough that the only way to do this is for you not to have an AGI, I think they'd make sure you don't get an AGI, amongst other things.
- ornornor 4y ago
  Or they’ll find a way to make sure they personally get an advantaged through an AGI and so it’ll happen. See pollution as a concept. No government truly wants to seriously tackle it because their leaders all get some extra power out of the money it brings.

airstrike 4y ago

> Also: imagine the year is 1900. You are saying that steam power and electricity is causing way too many changes way too fast so they put a moratorium on it until the year 2500. Great analogy. About a decade later, the world was fighting World War I on the back of the technological advances of the turn of the century. It was war on a scale never seen before. Literally orders of magnitude deadlier, bigger, more transformational and explosive. The word would never be the same. This time, should we expect another war? I'm not saying we should pause—it makes no sense, to your point. Instead, I'm just saying: brace. I like to think we (and our organic matter relatives) are hard to kill. Or at least to completely eradicate... so we will be around, or some proxy for us. Time to replay the Mass Effect trilogy

flangola7 4y ago
>I like to think we (and our organic matter relatives) are hard to kill. Or at least to completely eradicate... so we will be around, or some proxy for us. What is your hard evidence or reasoning for this? As I see it humans are quite vulnerable and will be as trivial to inadvertently eradicate as the dodo bird.
- extasia 4y ago
  Humans are much more resilient than dodo birds. Dodos were pretty devolved due to their insulated habitat[0]. 0. https://en.m.wikipedia.org/wiki/Island_gigantism https://en.m.wikipedia.org/wiki/Island_gigantism
  - Atreiden 4y ago
    I think that's his point. Currently, we have no natural predators. We completely outclass every other species on Earth in intelligence, and it's easy to see how advantageous that trait has been. What happens when we're no longer the Apex Intelligence?
- airstrike 4y ago
  Which is why I expanded the notion of survival to include other organic matter relatives, as they may fare better than we will. Simply put, organic life is unlikely to disappear entirely, which in time (eons, really), could result in sentient organic life emerging again once the machines go their own way or some such It's also a reason for us to colonize space as fast as possible ;-) it's easier to run away in 3D

brucethemoose2 4y ago

I dunno, the end goal (a pseudo AGI) is more like a nuclear weapon than the advent of electricity. The concern is real, even if we are farther away than the author realizes and the risk (like the risk if nuclear war) is not totally insurmountable even if it is extremely difficult. The first nuclear bombs couldn't end life on Earth either, but it wasn't long before they could, and the scientists working on them saw the trajectory as clear as day when the rest of the world didn't.

Tuna-Fish 4y ago
OT, but it's a pet peeve: > but it wasn't long before they could The total amount of nuclear weapons ever built is laughably inadequate for the task of ending life on earth. They would not end human life either, and even ending human civilization (as in, agriculture and organized society) is off by many orders of magnitude. Nuclear war would be horrible, but the actual impact got massively overinflated, largely because of good reasons.
- brucethemoose2 4y ago
  The initial fission stage is the "hard" part of a nuke, but the subsequent fusion stages are much easier to scale up. Arbitrarily large nuclear bombs are theoretically "easy" to make by chaining increasingly large fusion stages one after the other, as they aren't limited by expensive fissile material. https://blog.nuclearsecrecy.com/2012/09/12/in-search-of-a-bigger-boom/ https://blog.nuclearsecrecy.com/2012/09/12/in-search-of-a-bi... Hence scientists extrapolated nukes really could end life on Earth, and in a slightly different reality than ours, they may have.
  - roywiggins 4y ago
    It would be really, really hard to kill the bacteria and Archaeans that we've since found hiding deep underground. They're long way down and would barely notice.
- namaria 4y ago
  A whole country the size of the US can crap its collective pants with a freak cold snap and you think 10^4 nuclear detonations destroying all major cities is quite survivable?

sullyj3 4y ago

I'm sure he agrees the chance is remote, which is why he's so afraid. He's articulating what would be required, he literally uses the word "miracle" in the article.

antibasilisk 4y ago

Here's a proposal: let's just start WWIII, get it over and done with. By the time it's over no country will be technologically capable of anything resembling AGI.

ChatGTP 4y ago

* The cat is out of the box. You really think eg China will also pause this?* So what you're actually saying is that, there's a good chance that if we get an AGI, there will be pain as it will likely be used as a weapon, or could end in a nuclear exchange?

haskellandchill 4y ago

> We would still be using candles today. well the earth wouldn't be fucked then. that's the basis for an argument against technological progress.

deafpolygon 4y ago

China will NEVER pause this. They [Chinese government] see the US as a rival and an existential threat to their way of life. In fact, every time we post an open-source derivative or some paper detailing how it was done, we are inadvertently giving the advantage away to our rivals. AI development should not be stopped - rather than stopping it, we should seek to limit its applications now before they are used for the things that could harm us (such as military applications). AI has enormous potential to be of a massive benefit to mankind. But, most likely for the next decade or two - we will all be busy trying to make money off it, just like with the crypto bros.

BeFlatXIII 4y ago

> the country that will have the most powerful AI first will be the leader in everything Have you been listening to those podcasts for business leaders that take the idea of AI-powered digital transformation seriously? AI certainly can change things, but I find rhetoric like this to be massively overhyped.

iAmAPencilYo 4y ago
Could you share some of these podcasts? thanks!
Darmani 4y ago
For most of history, Japan has been existed as a small island next to the mighty kingdom of China. Even its name, "Land of the Rising Sun," refers to its relation as a nation east of Chinaa. Then along came the industrial revolution. Soon Japan was out conquering China, despite China having a much larger army. AI will be much faster, and much, much more powerful than the Industrial Revolution. If Switzerland is the first to human-level AI, it will not become the world leader in oil production, or shipping, or agriculture. But everything else will be Swiss. And then when the AI becomes superhuman, everything else will be gone.
- BeFlatXIII 4y ago
  What specific questions will AI solve that humans can't that would make it "more powerful than the Industrial Revolution"? I'm an AI skeptic. Most warnings feel more like science fantasy, specifically of the retro-futurism genre.

amai 4y ago

China will never get a powerful AI. Because they cannot allow training on all data, they must restrict it to data which is not censored by its government. Otherwise the AI might spit out truths they cannot control. And they will never allow that.

Darmani 4y ago

Chinese computer scientists read sci-fi, play video games, and worry about militarization, just like American computer scientists. The top Chinese AI researchers work at American companies and go to American universities. Many of them also worry about AI doomsday. And uncontrolled AI is a threat to the CCP. A Chinese outreach campaign can succeed. Let's make the English-language one also succeed.

G_z9 4y ago

I’ll just be quick. Powerful international coalition is def plausible because This isn’t steam. The mere existence of this technology poses a threat to all humans and their countries. There is no analogous tech, not even nukes You’re the naive one. There is no future for us if AGI is unleashed. It’s candles or extinction.

tanseydavid 4y ago

>> You are saying that steam power and electricity is causing way too many changes way too fast so they put a moratorium on it until the year 2500. There is an earlier post that casually calls the internal combustion engine a "moronic invention." I do not have words.