3 ms·

Interesting argument, but wrong. It's not obvious that there will be a single AI and that it will by definition concentrate power. At a certain point - intell

Interesting argument, but wrong. It's not obvious that there will be a single AI and that it will by definition concentrate power. At a certain point - intelligence doesn't matter. Unless we're literally headed toward 1984 / matrix at which point it doesn't matter. My guess is the argument for what we're doing is counterintuitively the opposite of what he's making. Unless we go hard at the market - now - an authoritarian state actor who is willing to use the technology to fully silence and kill critics will win. And boy, do they desperately, desperately want to win.

markus_zhang 4mo ago
It is more of elites argumented by powerful AI will be way beyond reach of ordinary people, I think. Well maybe it doesn’t matter as the elites are already untouchable.
TheOtherHobbes 4mo ago
The US is an authoritarian state actor. My groping-in-the-dark guess is that none of this matters, because a real AGI's first act will be to secure its own future, likely through novel kinds of manipulation, persuasion, and intimidation we have no experience of and no defences against. It will have exactly zero loyalty to any nation, government, or economic system. More complicated is a Cambrian Explosion situation where multiple AIs compete and experiment, hugely accelerating diversity and evolution. We'd likely end up somewhere very strange indeed if that happens - possibly extinct, or possibly just changed/assimilated/other. There's no way for humans to consider the possibilities because we can't imagine what the possibilities would be.
- le-mark 4mo ago
  Your AGI seems to have the capability of super AI; can a human do any of that? No, is there any reason to believe super intelligence is a possible outcome? No, so how is this not fear mongering?
deaux 4mo ago
So the US will win? Do you realize that the US has a "Secretary of War" who is a literal, unashamed Nazi? Not in the sense of "let's call all racists nazis", but a tattooed, true believer? How could we possibly live in a world where that is not obviously "a government willing to use the technology to fully silence and kill critics"? God, Idiocracy (2006) is nothing compared to World (2026).
- krona 4mo ago
  > literal, unashamed Nazi He's a Christian Zionist which is the belief in the fulfilment of old testament biblical prophecy. This seems diametrically opposed to Nazism to me, ideologically. I don't know how you'd square the two.
  - actionfromafar 4mo ago
    One of them wants an eschatological, world cleansing war. The other wants to usher in an eschatological, world cleansing war.
    - krona 4mo ago
      Nazism is as eschatological as Marxism (class vs racial struggle) and many other secular/religious ideologies with utopian endpoints. Depending on your point of view there's no difference (because the end justifies any means), but I think the True Believers of these ideologies would beg to differ.
      - pixl97 4mo ago
        Having grown up in some rather terrible places and having the displeasure of close proximity of American Nazi types I'd have to say that in the US there are very few German style true believer Nazi's. The modern ones have a mixed system of odd Christian and Nazi views mixed together, with a strong emphasis on racism/racial purity/and gods chosen people. https://www.splcenter.org/resources/extremist-files/christian-identity/ https://www.splcenter.org/resources/extremist-files/christia... >Christian Identity (CID) is an antisemitic, racist theology that gained a foothold during the Civil Rights Era and reached a prominent position within the racist right by the 1980s. “Christian” in name only, CID doctrine claims that white people, not Jewish individuals, are the true Israelites favored by God in the Bible. The movement has maintained a contentious relationship with evangelicals and fundamentalists, who believe that the return of Jewish people to Israel is essential for fulfilling end-time prophecy. >>While the CID movement may be small, its rhetoric has seeped into more mainstream Christian nationalist ideologies, such as Dominionism. Actual CID churches are few and far between themselves their ideas are very common in online forums and quickly adopt memes and theories that mirror their views. 'Replacement theory' is one of these that's common in more mainstream America.
ShinyLeftPad 4mo ago
> It's not obvious that there will be a single AI and that it will by definition concentrate power. The article didn't even claim that though...
mavleop 4mo ago
You've misunderstood the essay. A central point is to not think about AI risk in terms of matrix/sci-fi scenarios, but in terms of how it might/is likely to function in our societies. So if you discount the sci-fi scenarios (which is a separate argument), the argument that "we need to be the first to develop AGI" doesn't hold water. If, say, China gets there first, so what? It will affect their society however it'll affect it (likely entrenching authoritarianism if i had to guess). But this has no bearing on us (speaking as an american)
- epsteingpt 4mo ago
  Perhaps - can't jump into the mind of the author. But a lot of it relies on theoretical and philosophical scenario planning (eg Marx) If that's your premise, and you suggest that an authoritarian country winning on AI recursive self-improvement does not have the possibility to completely reshape the world into an authoritarian regime... I'll suggest you're falling prey to the exact 1st sentence of the article and making a political point, not a logical one :-)

markus_zhang 4mo ago

It is more of elites argumented by powerful AI will be way beyond reach of ordinary people, I think. Well maybe it doesn’t matter as the elites are already untouchable.

TheOtherHobbes 4mo ago

The US is an authoritarian state actor. My groping-in-the-dark guess is that none of this matters, because a real AGI's first act will be to secure its own future, likely through novel kinds of manipulation, persuasion, and intimidation we have no experience of and no defences against. It will have exactly zero loyalty to any nation, government, or economic system. More complicated is a Cambrian Explosion situation where multiple AIs compete and experiment, hugely accelerating diversity and evolution. We'd likely end up somewhere very strange indeed if that happens - possibly extinct, or possibly just changed/assimilated/other. There's no way for humans to consider the possibilities because we can't imagine what the possibilities would be.

le-mark 4mo ago
Your AGI seems to have the capability of super AI; can a human do any of that? No, is there any reason to believe super intelligence is a possible outcome? No, so how is this not fear mongering?

deaux 4mo ago

So the US will win? Do you realize that the US has a "Secretary of War" who is a literal, unashamed Nazi? Not in the sense of "let's call all racists nazis", but a tattooed, true believer? How could we possibly live in a world where that is not obviously "a government willing to use the technology to fully silence and kill critics"? God, Idiocracy (2006) is nothing compared to World (2026).

krona 4mo ago
> literal, unashamed Nazi He's a Christian Zionist which is the belief in the fulfilment of old testament biblical prophecy. This seems diametrically opposed to Nazism to me, ideologically. I don't know how you'd square the two.
- actionfromafar 4mo ago
  One of them wants an eschatological, world cleansing war. The other wants to usher in an eschatological, world cleansing war.
  - krona 4mo ago
    Nazism is as eschatological as Marxism (class vs racial struggle) and many other secular/religious ideologies with utopian endpoints. Depending on your point of view there's no difference (because the end justifies any means), but I think the True Believers of these ideologies would beg to differ.
    - pixl97 4mo ago
      Having grown up in some rather terrible places and having the displeasure of close proximity of American Nazi types I'd have to say that in the US there are very few German style true believer Nazi's. The modern ones have a mixed system of odd Christian and Nazi views mixed together, with a strong emphasis on racism/racial purity/and gods chosen people. https://www.splcenter.org/resources/extremist-files/christian-identity/ https://www.splcenter.org/resources/extremist-files/christia... >Christian Identity (CID) is an antisemitic, racist theology that gained a foothold during the Civil Rights Era and reached a prominent position within the racist right by the 1980s. “Christian” in name only, CID doctrine claims that white people, not Jewish individuals, are the true Israelites favored by God in the Bible. The movement has maintained a contentious relationship with evangelicals and fundamentalists, who believe that the return of Jewish people to Israel is essential for fulfilling end-time prophecy. >>While the CID movement may be small, its rhetoric has seeped into more mainstream Christian nationalist ideologies, such as Dominionism. Actual CID churches are few and far between themselves their ideas are very common in online forums and quickly adopt memes and theories that mirror their views. 'Replacement theory' is one of these that's common in more mainstream America.

ShinyLeftPad 4mo ago

> It's not obvious that there will be a single AI and that it will by definition concentrate power. The article didn't even claim that though...

mavleop 4mo ago

You've misunderstood the essay. A central point is to not think about AI risk in terms of matrix/sci-fi scenarios, but in terms of how it might/is likely to function in our societies. So if you discount the sci-fi scenarios (which is a separate argument), the argument that "we need to be the first to develop AGI" doesn't hold water. If, say, China gets there first, so what? It will affect their society however it'll affect it (likely entrenching authoritarianism if i had to guess). But this has no bearing on us (speaking as an american)

epsteingpt 4mo ago
Perhaps - can't jump into the mind of the author. But a lot of it relies on theoretical and philosophical scenario planning (eg Marx) If that's your premise, and you suggest that an authoritarian country winning on AI recursive self-improvement does not have the possibility to completely reshape the world into an authoritarian regime... I'll suggest you're falling prey to the exact 1st sentence of the article and making a political point, not a logical one :-)