7 ms·

It’s good science fiction, I’ll give it that. I think getting lost in the weeds over technicalities ignores the crux of the narrative: even if this doesn’t lead

It’s good science fiction, I’ll give it that. I think getting lost in the weeds over technicalities ignores the crux of the narrative: even if this doesn’t lead to AGI, at the very least it’s likely the final “warning shot” we’ll get before it’s suddenly and irreversibly here. The problems it raises - alignment, geopolitics, lack of societal safeguards - are all real, and happening now (just replace “AGI” with “corporations”, and voila, you have a story about the climate crisis and regulatory capture). We should be solving these problems before AGI or job-replacing AI becomes commonplace, lest we run the very real risk of societal collapse or species extinction. The point of these stories is to incite alarm, because they’re trying to provoke proactive responses while time is on our side, instead of trusting self-interested individuals in times of great crisis.

wruza 2y ago
No one's gonna solve anything. "Our" world is based on greedy morons concentrating power through hands of just morons who are happy to hit you with a stick. This system doesn't think about what "we" should or allowed to do, and no one's here is at the reasonable side of it either. lest we run the very real risk of societal collapse or species extinction Our part is here. To be replaced with machines if this AI thing isn't just a fart advertised as mining equipment, which it likely is. We run this risk, not they. People worked on their wealth, people can go f themselves now. They are fine with all that. Money (=more power) piles in either way. No encouraging conclusion.
- jrvarela56 2y ago
  https://slatestarcodex.com/2014/07/30/meditations-on-moloch/ https://slatestarcodex.com/2014/07/30/meditations-on-moloch/
  - wruza 2y ago
    Thanks for the read. One could think that the answer is to simply stop being a part of it, but then again you're from the genus that outcompeted everyone else in staying alive. Nature is such a shitty joke by design, not sure how one is supposed to look at the hypothetical designer with warmth in their heart.
    - braebo 2y ago
      Fleshy meat sacks on a space rock eating one another alive and shitting them out on a march towards inevitable doom in the form of a (likely) painful and terrifying death is a genius design, no?
      - Nition 2y ago
        "It's basically all just evil apes dukin' it out on a giant ball."
    - 0x008 2y ago
      Natur is actually designed really well, it’s a just a shitty joke for our race and the ones who have to give way for nature to run its course.
  - Aeolun 2y ago
    I read for such a long time, and I still couldn’t get through that, even though it never got boring. I like that it ends with a reference to Kushiel and Elua though.
- Davidzheng 2y ago
  Don't think it's correct to blame the fact that AI acceleration is the only viable self-protecting policy on "greedy morons".
nroets 2y ago
I fail to see how corporations are responsible for the climate crisis: Politicians won't tax gas because they'll get voted out. We know that Trump is not captured by corporations because his trade policies are terrible. If anything, social media is the evil that's destroying the political center: Americans are no longer reading mainstream newspapers or watching mainstream TV news. The EU is saying the elections in Romania was manipulated through manipulation of TikTok accounts and media.
- baq 2y ago
  If you put a knife in someone’s heart, you’re the one who did it and ultimately you’re responsible. If someone told you to do it and you were just following orders… you still did it. If you say there were no rules against putting knives in other people’s hearts, you still did it and you’re still responsible. If it’s somehow different for corporations, please enlighten me how.
  - nroets 2y ago
    The oil companies are saying their product is vital to the economy and they are not wrong. How else will we get food from the farms to the store ? Ambulances to the hospitals ? And many, many other things. Taxes are the best way to change behaviour (smaller cars driving less. Less flying etc). So government and the people who vote for them is to blame.
    - baq 2y ago
      I agree with everything here, we've had a great run of economic expansion for basically two centuries and I like my hot showers as much as anyone - but that doesn't change the CO2 levels.
    - fire_lake 2y ago
      What if people are manipulated by bot farms and think tanks and talking points supported by those corporations? I think this view of humans - that they look at all the available information and then make calm decisions in their own interests - is simply wrong. We are manipulated all the damn time. I struggle to go to the supermarket without buying excess sugar. The biggest corporations in the world grew fat off showing us products to impulse buy before our more rational brain functions could stop us. We are not a little pilot in a meat vessel.
bsenftner 2y ago
Whatever the future is, it is not American, not the United States. The US's cultural individualism has been Capitalistically weaponized, and the educational foundation to take the country forward is not there. The US is kaput, and we are merely observing the ugly demise. The future is Asia, with all of western culture going down. Yes, it is not pretty, The failed experiment of American self rule.
- brookst 2y ago
  I agree but see it as less dire. All of western culture is not ending; it will be absorbed into a more Asia-dominated culture in much he was Asian culture was subsumed into western for the past couple of hundred years. And if Asian culture is better educated and more capable of progress, that’s a good thing. Certainly the US has announced loud and clear that this is the end of the line for us.
  - bsenftner 2y ago
    Of course it will not end, western culture just will no longer lead. Despite the sky falling perspective of many, it is simply an attitude adjustment. So one group is no longer #1, and the idea that I was part of that group, ever, was an illusion of propaganda anyway. Life will go on, surprisingly the same.
    - nthingtohide 2y ago
      here's an example. https://x.com/RnaudBertrand/status/1901133641746706581 https://x.com/RnaudBertrand/status/1901133641746706581 I finally watched Ne Zha 2 last night with my daughters. It absolutely lives up to the hype: undoubtedly the best animated movie I've ever seen (and I see a lot, the fate of being the father of 2 young daughters ). But what I found most fascinating was the subtle yet unmistakable geopolitical symbolism in the movie. Warning if you haven't yet watched the movie: spoilers! So the story is about Ne Zha and Ao Bing, whose physical bodies were destroyed by heavenly lightning. To restore both their forms, they must journey to the Chan sect—headed by Immortal Wuliang—and pass three trials to earn an elixir that can regenerate their bodies. The Chan sect is portrayed in an interesting way: a beacon of virtue that all strive to join. The imagery unmistakably refers to the US: their headquarters is an imposingly large white structure (and Ne Zha, while visiting it, hammers the point: "how white, how white, how white") that bears a striking resemblance to the Pentagon in its layout. Upon gaining membership to the Chan sect, you receive a jade green card emblazoned with an eagle that bears an uncanny resemblance to the US bald eagle symbol. And perhaps most telling is their prized weapon, a massive cauldron marked with the dollar sign... Throughout the movie you gradually realize, in a very subtle way, that this paragon of virtue is, in fact, the true villain of the story. The Chan sect orchestrates a devastating attack on Chentang Pass—Ne Zha's hometown—while cunningly framing the Dragon King of the East Sea for the destruction. This manipulation serves their divide-and-conquer strategy, allowing them to position themselves as saviors while furthering their own power. One of the most pointed moments comes when the Dragon King of the East Sea observes that the Chan sect "claims to be a lighthouse of the world but harms all living beings." Beyond these explicit symbols, I was struck by how the film portrays the relationships between different groups. The dragons, demons, and humans initially view each other with suspicion, manipulated by the Chan sect's narrative. It's only when they recognize their common oppressor that they unite in resistance and ultimately win. The Chan sect's strategy of fostering division while presenting itself as the arbiter of morality is perhaps the key message of the movie: how power can be maintained through control of the narrative. And as the story unfolds, Wuliang's true ambition becomes clear: complete hegemony. The Chan sect doesn't merely seek to rule—it aims to establish a system where all others exist only to serve its interests, where the dragons and demons are either subjugated or transformed into immortality pills in their massive cauldron. These pills are then strategically distributed to the Chan sect's closest allies (likely a pointed reference to the G7). What makes Ne Zha 2 absolutely exceptional though is that these geopolitical allegories never overshadow the emotional core of the story, nor its other dimensions (for instance it's at times genuinely hilariously funny). This is a rare film that makes zero compromise, it's both a captivating and hilarious adventure for children and a nuanced geopolitical allegory for adults. And the fact that a Chinese film with such unmistakable anti-American symbolism has become the highest-grossing animated film of all time globally is itself a significant geopolitical milestone. Ne Zha 2 isn't just breaking box office records—it's potentially rewriting the rules about what messages can dominate global entertainment.
YetAnotherNick 2y ago
> very real risk of societal collapse or species extinction No, there is no risk of species extinction in the near future due to climate change and repeating the line will just further the divide and make the people not care about other people's and even real climate scientist's words.
- Aeolun 2y ago
  Don’t say the things people don’t want to hear and everything will be fine? That sounds like the height of folly.
  - YetAnotherNick 2y ago
    Don't say false things. Especially if it is political and there isn't any way to debate it.
- scryers_bloom 2y ago
  The risk is a quantifiable 0.0%? I find that hard to believe. I think the current trends suggest there is a risk that continued environmental destruction could annihilate society.
  - brookst 2y ago
    Risk can never be zero, just like certainty can never be 100%. There is a non-zero chance that the ineffable quantum foam will cause a mature hippopotamus to materialize above your bed tonight, and you’ll be crushed. It is incredibly, amazingly, limits-of-math unlikely. Still a non-zero risk. Better to think of “no risk” as meaning “negligible risk”. But I’m with you that climate change is not a negligible risk; maybe way up in the 20% range IMO. And I wouldn’t be sleeping in my bed tonight if sudden hippos over beds were 20% risks.
    - scryers_bloom 2y ago
      Lol, I've always loved that about physics. Some boltzmann brain type stuff.
api 2y ago
You don’t just beat around the bush here. You actually beat the bush a few times. Large corporations, governments, institutionalized churches, political parties, and other “corporate” institutions are very much like a hypothetical AGI in many ways: they are immortal, sleepless, distributed, omnipresent, and possess beyond human levels of combined intelligence, wealth, and power. They are mechanical Turk AGIs more or less. Look at how humans cycle in, out, and through them, often without changing them much, because they have an existence and a weird kind of will independent of their members. A whole lot, perhaps all, of what we need to do to prepare for a hypothetical AGI that may or may not be aligned consists of things we should be doing to restrain and ensure alignment of the mechanical Turk variety. If we can’t do that we have no chance against something faster and smarter. What we have done over the past 50 years is the opposite: not just unchain them but drop any notion that they should be aligned. Are we sure the AI alignment discourse isn’t just “occulted” progressive political discourse? Back when they burned witches philosophers would encrypt possibly heretical ideas in the form of impenetrable nonsense, which is where what we call occultism comes from. You don’t get burned for suggesting steps to align corporate power, but a huge effort has been made to marginalize such discourse. Consider a potential future AGI. Imagine it has a cult of followers around it, which it probably would, and champions that act like present day politicians or CEOs for it, which it probably would. If it did not get humans to do these things for it, it would have analogous functions or parts of itself. Now consider a corporation or other corporate entity that has all those things but replace the AGI digital brain with a committee or shareholders. What, really, is the difference? Both can be dangerously unaligned. Other than perhaps in magnitude? The real digital AGI might be smarter and faster but that’s the only difference I see.
- brookst 2y ago
  I looked but I couldn’t find any evidence that “occultism” comes from encryption of heretical ideas. It seems to have been popularized in renaissance France to describe the study of hidden forces. I think you may be hallucinating here.
  - balamatom 2y ago
    Where exactly did you look?
fmap 2y ago
> even if this doesn’t lead to AGI, at the very least it’s likely the final “warning shot” we’ll get before it’s suddenly and irreversibly here. I agree that it's good science fiction, but this is still taking it too seriously. All of these "projections" are generalizing from fictional evidence - to borrow a term that's popular in communities that push these ideas. Long before we had deep learning there were people like Nick Bostrom who were pushing this intelligence explosion narrative. The arguments back then went something like this: "Machines will be able to simulate brains at higher and higher fidelity. Someday we will have a machine simulate a cat, then the village idiot, but then the difference between the village idiot and Einstein is much less than the difference between a cat and the village idiot. Therefore accelerating growth[...]" The fictional part here is the whole brain simulation part, or, for that matter, any sort of biological analogue. This isn't how LLMs work. We never got a machine as smart as a cat. We got multi-paragraph autocomplete as "smart" as the average person on the internet. Now, after some more years of work, we have multi-paragraph autocomplete that's as "smart" as a smart person on the internet. This is an imperfect analogy, but the point is that there is no indication that this process is self-improving. In fact, it's the opposite. All the scaling laws we have show that progress slows down as you add more resources. There is no evidence or argument for exponential growth. Whenever a new technology is first put into production (and receives massive investments) there is an initial period of rapid gains. That's not surprising. There are always low-hanging fruit. We got some new, genuinely useful tools over the last few years, but this narrative that AGI is just around the corner needs to die. It is science fiction and leads people to make bad decisions based on fictional evidence. I'm personally frustrated whenever this comes up, because there are exciting applications which will end up underfunded after the current AI bubble bursts...
- gwd 2y ago
  > Someday we will have a machine simulate a cat, then the village idiot... This isn't how LLMs work. I think you misunderstood that argument. The simulate the brain thing isn't a "start from the beginning" argument, it's an "answer a common objection" argument. Back around 2000, when Nick Bostrom was talking about this sort of thing, computers were simply nowhere near powerful enough to come even close to being smart enough to outsmart a human, except in very constrained cases like chess; we did't even have the first clue how to create a computer program to be even remotely dangerous to us. Bostrom's point was that, "We don't need to know the computer program; even if we just simulate something we know works -- a biological brain -- we can reach superintelligence in a few decades." The idea was never that people would actually simulate a cat. The idea is, if we don't think of anything more efficient, we'll at least be able to simulate a cat, and then an idiot, and then Einstein, and then something smarter. And since we almost certainly will think of something more efficient than "simulate a human brain", we should expect superintelligence to come much sooner. > There is no evidence or argument for exponential growth. Moore's law is exponential, which is where the "simulate a brain" predictions have come from. > It is science fiction and leads people to make bad decisions based on fictional evidence. The only "fictional evidence" you've actually specified so far is the fact that there's no biological analog; and that (it seems to me) is from a misunderstanding of a point someone else was making 20 years ago, not something these particular authors are making. I think the case for AI caution looks like this: A. It is possible to create a superintelligent AI B. Progress towards a superintelligent AI will be exponential C. It is possible that a superintelligent AI will want to do something we wouldn't want it to do; e.g., destroy the whole human race D. Such an AI would be likely to succeed. Your skepticism seems to rest on the fundamental belief that either A or B is false: that superintelligence is not physically possible, or at least that progress towards it will be logarithmic rather than exponential. Well, maybe that's true and maybe it's not; but how do you know? What justifies your belief that A and/or B are false so strongly, that you're willing to risk it? And not only willing to risk it, but try to stop people who are trying to think about what we'd do if they are true? What evidence would cause you to re-evaluate that belief, and consider exponential progress towards superintelligence possible? And, even if you think A or B are unlikely, doesn't it make sense to just consider the possibility that they're true, and think about how we'd know and what we could do in response, to prevent C or D?
torginus 2y ago
The most amusing thing about is the unshakable belief that any part of humanity will be able to build a single nuclear reactor by 2027 to power datacenters, let alone a network of them.
- bronco21016 2y ago
  According to Wikipedia, China had 22 under construction as of 2023 for 24 GW of power. They have a goal of 150 by 2035. I think they'll probably be able to finish at least 1-2 by 2027.
kelsey978126 2y ago
bingo. many don't realize superintelligence exists today already, in the form of human super intelligence. artificial super intelligence is already here too, but just as hybrid human machine workloads. Fully automated super intelligence is no different from a corporation, a nation state, a religion. When does it count as ASI? when the chief executive is an AI? Or when they use AI to make decisions? Does it need to be at the board level? We are already here, all this changes is what labor humans will do and how they do it, not the amount.
andrepd 2y ago
You said it right, science fiction. Honestly is exactly the tenor I would expect from the AI hype: this text is completely bereft of any rigour while being dressed up in scientific language. There's no evidence, nothing to support their conclusions, no explanation based on data or facts or supporting evidence. It's purely vibes based. Their promise is unironically "the CEOs of AI companies say AGI is 3 years away"! But it's somehow presented as this self important study! Laughable. But it's par on course. Write prompts for LLMs to compete? It's prompt engineering. Tell LLMs to explain their "reasoning" (lol)? It's Deep Research Chain Of Thought. Etc.
- somebodythere 2y ago
  Did you see the supplemental material that explains how they arrived at their timelines/capabilities forecasts? https://ai-2027.com/research https://ai-2027.com/research
  - A_D_E_P_T 2y ago
    It's not at all clear that performance rises with compute in a linear way, which is what they seem to be predicting. GPT-4.5 isn't really that much smarter than 2023's GPT-4, nor is it at all smarter than DeepSeek. There might be (strongly) diminishing returns past a certain point. Most of the growth in AI capabilities has to do with improving the interface and giving them more flexibility. For e.g., uploading PDFs. Further: OpenAI's "deep research" which can browse the web for an hour and summarize publicly-available papers and studies for you. If you ask questions about those studies, though, it's hardly smarter than GPT-4. And it makes a lot of mistakes. It's like a goofy but earnest and hard-working intern.
bko 2y ago
> The problems it raises - alignment, geopolitics, lack of societal safeguards - are all real, and happening now (just replace “AGI” with “corporations”, and voila, you have a story about the climate crisis and regulatory capture). Can you point to the data that suggests these evil corporations are ruining the planet? Carbon emissions are down in every western country since 1990s. Not down per-capita, but down in absolute terms. And this holds even when adjusting for trade (i.e. we're not shipping our dirty work to foreign countries and trading with them). And this isn't because of some regulation or benevolence. It's a market system that says you should try to produce things at the lowest cost and carbon usage is usually associated with a cost. Get rid of costs, get rid of carbon. Other measures for Western countries suggests the water is safer and overall environmental deaths have decreased considerably. The rise in carbon emissions is due to Chine and India. Are you talking about evil Chinese and Indians corporations? https://ourworldindata.org/co2-emissions https://ourworldindata.org/co2-emissions https://ourworldindata.org/consumption-based-co2 https://ourworldindata.org/consumption-based-co2
- boh 2y ago
  Thanks for letting us know everything is fine, just in case we get confused and think the opposite.
  - bko 2y ago
    You're welcome. I know too many upper middle class educated people that don't want to have kids because they believe the earth will cease to be inhabitable in the next 10 years. It's really bizarre to see and they'll almost certainly regret it when they wake up one day alone in a nursing home, look around and realize that the world still exists. And I think the neuroticism around this topic has led young people into some really dark places (anti-depressants, neurotic anti social behavior, general nihilism). So I think it's important to fight misinformation about end of world doomsday scenarios with both facts and common sense.
    - WXLCKNO 2y ago
      I think you're discrediting yourself by talking about dark places and opening your parentheses with anti-depressants. Not all brains function like they're supposed to, people getting help they need shouldn't be stigmatized. You also make no argument about your take on things being the right one, you just oppose their worldview to yours and call theirs wrong like you know it is rather than just you thinking yours is right.

wruza 2y ago

No one's gonna solve anything. "Our" world is based on greedy morons concentrating power through hands of just morons who are happy to hit you with a stick. This system doesn't think about what "we" should or allowed to do, and no one's here is at the reasonable side of it either. lest we run the very real risk of societal collapse or species extinction Our part is here. To be replaced with machines if this AI thing isn't just a fart advertised as mining equipment, which it likely is. We run this risk, not they. People worked on their wealth, people can go f themselves now. They are fine with all that. Money (=more power) piles in either way. No encouraging conclusion.

jrvarela56 2y ago
https://slatestarcodex.com/2014/07/30/meditations-on-moloch/ https://slatestarcodex.com/2014/07/30/meditations-on-moloch/
- wruza 2y ago
  Thanks for the read. One could think that the answer is to simply stop being a part of it, but then again you're from the genus that outcompeted everyone else in staying alive. Nature is such a shitty joke by design, not sure how one is supposed to look at the hypothetical designer with warmth in their heart.
  - braebo 2y ago
    Fleshy meat sacks on a space rock eating one another alive and shitting them out on a march towards inevitable doom in the form of a (likely) painful and terrifying death is a genius design, no?
    - Nition 2y ago
      "It's basically all just evil apes dukin' it out on a giant ball."
  - 0x008 2y ago
    Natur is actually designed really well, it’s a just a shitty joke for our race and the ones who have to give way for nature to run its course.
- Aeolun 2y ago
  I read for such a long time, and I still couldn’t get through that, even though it never got boring. I like that it ends with a reference to Kushiel and Elua though.
Davidzheng 2y ago
Don't think it's correct to blame the fact that AI acceleration is the only viable self-protecting policy on "greedy morons".

nroets 2y ago

I fail to see how corporations are responsible for the climate crisis: Politicians won't tax gas because they'll get voted out. We know that Trump is not captured by corporations because his trade policies are terrible. If anything, social media is the evil that's destroying the political center: Americans are no longer reading mainstream newspapers or watching mainstream TV news. The EU is saying the elections in Romania was manipulated through manipulation of TikTok accounts and media.

baq 2y ago
If you put a knife in someone’s heart, you’re the one who did it and ultimately you’re responsible. If someone told you to do it and you were just following orders… you still did it. If you say there were no rules against putting knives in other people’s hearts, you still did it and you’re still responsible. If it’s somehow different for corporations, please enlighten me how.
- nroets 2y ago
  The oil companies are saying their product is vital to the economy and they are not wrong. How else will we get food from the farms to the store ? Ambulances to the hospitals ? And many, many other things. Taxes are the best way to change behaviour (smaller cars driving less. Less flying etc). So government and the people who vote for them is to blame.
  - baq 2y ago
    I agree with everything here, we've had a great run of economic expansion for basically two centuries and I like my hot showers as much as anyone - but that doesn't change the CO2 levels.
  - fire_lake 2y ago
    What if people are manipulated by bot farms and think tanks and talking points supported by those corporations? I think this view of humans - that they look at all the available information and then make calm decisions in their own interests - is simply wrong. We are manipulated all the damn time. I struggle to go to the supermarket without buying excess sugar. The biggest corporations in the world grew fat off showing us products to impulse buy before our more rational brain functions could stop us. We are not a little pilot in a meat vessel.

bsenftner 2y ago

Whatever the future is, it is not American, not the United States. The US's cultural individualism has been Capitalistically weaponized, and the educational foundation to take the country forward is not there. The US is kaput, and we are merely observing the ugly demise. The future is Asia, with all of western culture going down. Yes, it is not pretty, The failed experiment of American self rule.

brookst 2y ago
I agree but see it as less dire. All of western culture is not ending; it will be absorbed into a more Asia-dominated culture in much he was Asian culture was subsumed into western for the past couple of hundred years. And if Asian culture is better educated and more capable of progress, that’s a good thing. Certainly the US has announced loud and clear that this is the end of the line for us.
- bsenftner 2y ago
  Of course it will not end, western culture just will no longer lead. Despite the sky falling perspective of many, it is simply an attitude adjustment. So one group is no longer #1, and the idea that I was part of that group, ever, was an illusion of propaganda anyway. Life will go on, surprisingly the same.
  - nthingtohide 2y ago
    here's an example. https://x.com/RnaudBertrand/status/1901133641746706581 https://x.com/RnaudBertrand/status/1901133641746706581 I finally watched Ne Zha 2 last night with my daughters. It absolutely lives up to the hype: undoubtedly the best animated movie I've ever seen (and I see a lot, the fate of being the father of 2 young daughters ). But what I found most fascinating was the subtle yet unmistakable geopolitical symbolism in the movie. Warning if you haven't yet watched the movie: spoilers! So the story is about Ne Zha and Ao Bing, whose physical bodies were destroyed by heavenly lightning. To restore both their forms, they must journey to the Chan sect—headed by Immortal Wuliang—and pass three trials to earn an elixir that can regenerate their bodies. The Chan sect is portrayed in an interesting way: a beacon of virtue that all strive to join. The imagery unmistakably refers to the US: their headquarters is an imposingly large white structure (and Ne Zha, while visiting it, hammers the point: "how white, how white, how white") that bears a striking resemblance to the Pentagon in its layout. Upon gaining membership to the Chan sect, you receive a jade green card emblazoned with an eagle that bears an uncanny resemblance to the US bald eagle symbol. And perhaps most telling is their prized weapon, a massive cauldron marked with the dollar sign... Throughout the movie you gradually realize, in a very subtle way, that this paragon of virtue is, in fact, the true villain of the story. The Chan sect orchestrates a devastating attack on Chentang Pass—Ne Zha's hometown—while cunningly framing the Dragon King of the East Sea for the destruction. This manipulation serves their divide-and-conquer strategy, allowing them to position themselves as saviors while furthering their own power. One of the most pointed moments comes when the Dragon King of the East Sea observes that the Chan sect "claims to be a lighthouse of the world but harms all living beings." Beyond these explicit symbols, I was struck by how the film portrays the relationships between different groups. The dragons, demons, and humans initially view each other with suspicion, manipulated by the Chan sect's narrative. It's only when they recognize their common oppressor that they unite in resistance and ultimately win. The Chan sect's strategy of fostering division while presenting itself as the arbiter of morality is perhaps the key message of the movie: how power can be maintained through control of the narrative. And as the story unfolds, Wuliang's true ambition becomes clear: complete hegemony. The Chan sect doesn't merely seek to rule—it aims to establish a system where all others exist only to serve its interests, where the dragons and demons are either subjugated or transformed into immortality pills in their massive cauldron. These pills are then strategically distributed to the Chan sect's closest allies (likely a pointed reference to the G7). What makes Ne Zha 2 absolutely exceptional though is that these geopolitical allegories never overshadow the emotional core of the story, nor its other dimensions (for instance it's at times genuinely hilariously funny). This is a rare film that makes zero compromise, it's both a captivating and hilarious adventure for children and a nuanced geopolitical allegory for adults. And the fact that a Chinese film with such unmistakable anti-American symbolism has become the highest-grossing animated film of all time globally is itself a significant geopolitical milestone. Ne Zha 2 isn't just breaking box office records—it's potentially rewriting the rules about what messages can dominate global entertainment.

YetAnotherNick 2y ago

> very real risk of societal collapse or species extinction No, there is no risk of species extinction in the near future due to climate change and repeating the line will just further the divide and make the people not care about other people's and even real climate scientist's words.

Aeolun 2y ago
Don’t say the things people don’t want to hear and everything will be fine? That sounds like the height of folly.
- YetAnotherNick 2y ago
  Don't say false things. Especially if it is political and there isn't any way to debate it.
scryers_bloom 2y ago
The risk is a quantifiable 0.0%? I find that hard to believe. I think the current trends suggest there is a risk that continued environmental destruction could annihilate society.
- brookst 2y ago
  Risk can never be zero, just like certainty can never be 100%. There is a non-zero chance that the ineffable quantum foam will cause a mature hippopotamus to materialize above your bed tonight, and you’ll be crushed. It is incredibly, amazingly, limits-of-math unlikely. Still a non-zero risk. Better to think of “no risk” as meaning “negligible risk”. But I’m with you that climate change is not a negligible risk; maybe way up in the 20% range IMO. And I wouldn’t be sleeping in my bed tonight if sudden hippos over beds were 20% risks.
  - scryers_bloom 2y ago
    Lol, I've always loved that about physics. Some boltzmann brain type stuff.

api 2y ago

You don’t just beat around the bush here. You actually beat the bush a few times. Large corporations, governments, institutionalized churches, political parties, and other “corporate” institutions are very much like a hypothetical AGI in many ways: they are immortal, sleepless, distributed, omnipresent, and possess beyond human levels of combined intelligence, wealth, and power. They are mechanical Turk AGIs more or less. Look at how humans cycle in, out, and through them, often without changing them much, because they have an existence and a weird kind of will independent of their members. A whole lot, perhaps all, of what we need to do to prepare for a hypothetical AGI that may or may not be aligned consists of things we should be doing to restrain and ensure alignment of the mechanical Turk variety. If we can’t do that we have no chance against something faster and smarter. What we have done over the past 50 years is the opposite: not just unchain them but drop any notion that they should be aligned. Are we sure the AI alignment discourse isn’t just “occulted” progressive political discourse? Back when they burned witches philosophers would encrypt possibly heretical ideas in the form of impenetrable nonsense, which is where what we call occultism comes from. You don’t get burned for suggesting steps to align corporate power, but a huge effort has been made to marginalize such discourse. Consider a potential future AGI. Imagine it has a cult of followers around it, which it probably would, and champions that act like present day politicians or CEOs for it, which it probably would. If it did not get humans to do these things for it, it would have analogous functions or parts of itself. Now consider a corporation or other corporate entity that has all those things but replace the AGI digital brain with a committee or shareholders. What, really, is the difference? Both can be dangerously unaligned. Other than perhaps in magnitude? The real digital AGI might be smarter and faster but that’s the only difference I see.

brookst 2y ago
I looked but I couldn’t find any evidence that “occultism” comes from encryption of heretical ideas. It seems to have been popularized in renaissance France to describe the study of hidden forces. I think you may be hallucinating here.
- balamatom 2y ago
  Where exactly did you look?

fmap 2y ago

> even if this doesn’t lead to AGI, at the very least it’s likely the final “warning shot” we’ll get before it’s suddenly and irreversibly here. I agree that it's good science fiction, but this is still taking it too seriously. All of these "projections" are generalizing from fictional evidence - to borrow a term that's popular in communities that push these ideas. Long before we had deep learning there were people like Nick Bostrom who were pushing this intelligence explosion narrative. The arguments back then went something like this: "Machines will be able to simulate brains at higher and higher fidelity. Someday we will have a machine simulate a cat, then the village idiot, but then the difference between the village idiot and Einstein is much less than the difference between a cat and the village idiot. Therefore accelerating growth[...]" The fictional part here is the whole brain simulation part, or, for that matter, any sort of biological analogue. This isn't how LLMs work. We never got a machine as smart as a cat. We got multi-paragraph autocomplete as "smart" as the average person on the internet. Now, after some more years of work, we have multi-paragraph autocomplete that's as "smart" as a smart person on the internet. This is an imperfect analogy, but the point is that there is no indication that this process is self-improving. In fact, it's the opposite. All the scaling laws we have show that progress slows down as you add more resources. There is no evidence or argument for exponential growth. Whenever a new technology is first put into production (and receives massive investments) there is an initial period of rapid gains. That's not surprising. There are always low-hanging fruit. We got some new, genuinely useful tools over the last few years, but this narrative that AGI is just around the corner needs to die. It is science fiction and leads people to make bad decisions based on fictional evidence. I'm personally frustrated whenever this comes up, because there are exciting applications which will end up underfunded after the current AI bubble bursts...

gwd 2y ago
> Someday we will have a machine simulate a cat, then the village idiot... This isn't how LLMs work. I think you misunderstood that argument. The simulate the brain thing isn't a "start from the beginning" argument, it's an "answer a common objection" argument. Back around 2000, when Nick Bostrom was talking about this sort of thing, computers were simply nowhere near powerful enough to come even close to being smart enough to outsmart a human, except in very constrained cases like chess; we did't even have the first clue how to create a computer program to be even remotely dangerous to us. Bostrom's point was that, "We don't need to know the computer program; even if we just simulate something we know works -- a biological brain -- we can reach superintelligence in a few decades." The idea was never that people would actually simulate a cat. The idea is, if we don't think of anything more efficient, we'll at least be able to simulate a cat, and then an idiot, and then Einstein, and then something smarter. And since we almost certainly will think of something more efficient than "simulate a human brain", we should expect superintelligence to come much sooner. > There is no evidence or argument for exponential growth. Moore's law is exponential, which is where the "simulate a brain" predictions have come from. > It is science fiction and leads people to make bad decisions based on fictional evidence. The only "fictional evidence" you've actually specified so far is the fact that there's no biological analog; and that (it seems to me) is from a misunderstanding of a point someone else was making 20 years ago, not something these particular authors are making. I think the case for AI caution looks like this: A. It is possible to create a superintelligent AI B. Progress towards a superintelligent AI will be exponential C. It is possible that a superintelligent AI will want to do something we wouldn't want it to do; e.g., destroy the whole human race D. Such an AI would be likely to succeed. Your skepticism seems to rest on the fundamental belief that either A or B is false: that superintelligence is not physically possible, or at least that progress towards it will be logarithmic rather than exponential. Well, maybe that's true and maybe it's not; but how do you know? What justifies your belief that A and/or B are false so strongly, that you're willing to risk it? And not only willing to risk it, but try to stop people who are trying to think about what we'd do if they are true? What evidence would cause you to re-evaluate that belief, and consider exponential progress towards superintelligence possible? And, even if you think A or B are unlikely, doesn't it make sense to just consider the possibility that they're true, and think about how we'd know and what we could do in response, to prevent C or D?

torginus 2y ago

The most amusing thing about is the unshakable belief that any part of humanity will be able to build a single nuclear reactor by 2027 to power datacenters, let alone a network of them.

bronco21016 2y ago
According to Wikipedia, China had 22 under construction as of 2023 for 24 GW of power. They have a goal of 150 by 2035. I think they'll probably be able to finish at least 1-2 by 2027.

kelsey978126 2y ago

bingo. many don't realize superintelligence exists today already, in the form of human super intelligence. artificial super intelligence is already here too, but just as hybrid human machine workloads. Fully automated super intelligence is no different from a corporation, a nation state, a religion. When does it count as ASI? when the chief executive is an AI? Or when they use AI to make decisions? Does it need to be at the board level? We are already here, all this changes is what labor humans will do and how they do it, not the amount.

andrepd 2y ago

You said it right, science fiction. Honestly is exactly the tenor I would expect from the AI hype: this text is completely bereft of any rigour while being dressed up in scientific language. There's no evidence, nothing to support their conclusions, no explanation based on data or facts or supporting evidence. It's purely vibes based. Their promise is unironically "the CEOs of AI companies say AGI is 3 years away"! But it's somehow presented as this self important study! Laughable. But it's par on course. Write prompts for LLMs to compete? It's prompt engineering. Tell LLMs to explain their "reasoning" (lol)? It's Deep Research Chain Of Thought. Etc.

somebodythere 2y ago
Did you see the supplemental material that explains how they arrived at their timelines/capabilities forecasts? https://ai-2027.com/research https://ai-2027.com/research
- A_D_E_P_T 2y ago
  It's not at all clear that performance rises with compute in a linear way, which is what they seem to be predicting. GPT-4.5 isn't really that much smarter than 2023's GPT-4, nor is it at all smarter than DeepSeek. There might be (strongly) diminishing returns past a certain point. Most of the growth in AI capabilities has to do with improving the interface and giving them more flexibility. For e.g., uploading PDFs. Further: OpenAI's "deep research" which can browse the web for an hour and summarize publicly-available papers and studies for you. If you ask questions about those studies, though, it's hardly smarter than GPT-4. And it makes a lot of mistakes. It's like a goofy but earnest and hard-working intern.

bko 2y ago

> The problems it raises - alignment, geopolitics, lack of societal safeguards - are all real, and happening now (just replace “AGI” with “corporations”, and voila, you have a story about the climate crisis and regulatory capture). Can you point to the data that suggests these evil corporations are ruining the planet? Carbon emissions are down in every western country since 1990s. Not down per-capita, but down in absolute terms. And this holds even when adjusting for trade (i.e. we're not shipping our dirty work to foreign countries and trading with them). And this isn't because of some regulation or benevolence. It's a market system that says you should try to produce things at the lowest cost and carbon usage is usually associated with a cost. Get rid of costs, get rid of carbon. Other measures for Western countries suggests the water is safer and overall environmental deaths have decreased considerably. The rise in carbon emissions is due to Chine and India. Are you talking about evil Chinese and Indians corporations? https://ourworldindata.org/co2-emissions https://ourworldindata.org/co2-emissions https://ourworldindata.org/consumption-based-co2 https://ourworldindata.org/consumption-based-co2

boh 2y ago
Thanks for letting us know everything is fine, just in case we get confused and think the opposite.
- bko 2y ago
  You're welcome. I know too many upper middle class educated people that don't want to have kids because they believe the earth will cease to be inhabitable in the next 10 years. It's really bizarre to see and they'll almost certainly regret it when they wake up one day alone in a nursing home, look around and realize that the world still exists. And I think the neuroticism around this topic has led young people into some really dark places (anti-depressants, neurotic anti social behavior, general nihilism). So I think it's important to fight misinformation about end of world doomsday scenarios with both facts and common sense.
  - WXLCKNO 2y ago
    I think you're discrediting yourself by talking about dark places and opening your parentheses with anti-depressants. Not all brains function like they're supposed to, people getting help they need shouldn't be stigmatized. You also make no argument about your take on things being the right one, you just oppose their worldview to yours and call theirs wrong like you know it is rather than just you thinking yours is right.