5 ms·

The 'dangerous' promise of a techno-utopian future

49 points by pseudolus 2y ago

roenxi 2y ago
A fun thought experiment is describing modern living to someone in the early Georgian era in an uncharitable way. It would be relatively easy to sell it is a godless industrial landscape. A city dweller could feasibly go days without seeing a tree or even leaving their apartment. Threat of annihilation looming over everyone's heads and the various major armies that can reach out and do what they like in most parts of the world. Constant risk from authoritarian ideologies that spread like wildfire. The future is always scary. The present is pretty scary. The past was much less comfortable.
- hsuduebc2 2y ago
  You are right. But I thinking it's fair to say that industrial progress did have some consequences that could be easily prevented. Mostly at the beginning. But I do not like this alarmist view either. It's usually useless and just make people feeling more endangered which is not a good prerequisite for racional debate. I would be worried more not about the technologies it self but by their human overlords.
- deleted 2y ago
  [deleted]
refurb 2y ago
[flagged]
- deleted 2y ago
  [deleted]
AdieuToLogic 2y ago
Back in 1995, Carl Sagan had a relevant thought: I have a foreboding of an America in my children's or grandchildren's time - [...] when awesome technological powers are in the hands of a very few, and no one representing the public interest can even grasp the issues; when the people have lost the ability to set their own agendas or knowledgeably question those in authority; when, clutching our crystals and nervously consulting our horoscopes, our critical faculties in decline, unable to distinguish between what feels good and what’s true, we slide, almost without noticing, back into superstition and darkness.[0] 0 - https://www.azquotes.com/quote/1069554 https://www.azquotes.com/quote/1069554
- zfg 2y ago
  More here: https://www.openculture.com/2025/02/carl-sagan-predicts-the-decline-of-america-unable-to-know-whats-true.html https://www.openculture.com/2025/02/carl-sagan-predicts-the-...
- kcplate 2y ago
  I have seen that quote dragged out lately as some sort of prophecy of 2025…but it really speaks to the exact opposite in my opinion. Smartphones and the internet have given the layperson awesome technology literally in their hands. The entire election of 2024 was driven by the people specifically questioning those in authority…and just what exactly is the “public interest”? Is it what the public demands? Or is it what some elite few say it should be? Seems to me whatever time that Sagan envisioned, it certainly isn’t this time we are in right now.
  - bdangubic 2y ago
    The entire election of 2024 was driven by the people specifically questioning those in authority you really think this???! so we should expect the same in 2028 but at much larger scale now though, as now we have actual authority who is not caring for either constitution or the rule of law, left side winning all 50 states? I think not, the questioning of authority is what people in power make people without think what actually happen… :)
  - xvector 2y ago
    I 100% agree. Anyone can use the Sagan quote to justify why they lost. The quote is just used as a fancy way of calling people with different views stupid. "Society is failing because people are dumber, and thats why they voted for X, when a smart person like me would have voted for Y!"
    - __loam 2y ago
      This is such a reductive view of what happened in 2024. Oligarchs openly supported a political candidate that has attacked experts time and again on things that are provably true because it's inconvenient to their narrative. We know Musk spent hundreds of millions of dollars supporting one side because it was in his interest. Mark Zuckerberg loves that side because they're not going to regulate his objectively harmful platform. Since this administration entered office, it has attacked agencies and organizations that do things like prepare for climate change, educate our children, provide essential Healthcare to 10s of millions of Americans, do basic scientific research, and protect consumers from financial fraud. Even if you get rid of these organizations, climate disasters will still happen (and insurance companies will still pull out of states like Florida and California), kids still need basic education to function in a high technology society, people are still getting sick, China is pulling ahead on public research, and fraudsters are still stealing from the American public. This administration is bad because it's attacking the career public servants and agencies that are a net positive for our society. We get back more than the investment here in innumerable ways. It is stupid to try and dismantle the parts of government that work and work well. So shut the hell up with this if they were smart they would have voted for Y bullshit as if this was ever a good faith alternative to the liberal program the dems ran on.
deleted 2y ago
[deleted]
AndrewKemendo 2y ago
Can someone give an alternative vision of the next 100-200 years that doesn't include AGI/ASI? I don't see it. At least one that isn't a nuclear wasteland. I have thought about this every day for 32 years and don't know of any possible alternative future than one run alongside or by Artificial Superhuman Systems. I just can't possibly look into the future and expect the current state of machines to plateau here and stop. In fact that's my life's work and has been so since I was 13.[1] I even discussed this at the AGI conference in Laval in 2014 with Turing winner Yoshua Bengio [2] The Overcoming Bias blog, which turned into Less wrong, was the best place for these discussions until it became a cult of personality around 2013. The SIAI and all the mess that it became at least was a chance to discuss it. There's no serious place to discuss it. AGI discussion went from nerds that everyone pointed and laughed at, to now being an imminent existential threat "To the average person, this all sounds like the plot of a sci-fi novel. There might be a tendency to dismiss a techno-utopia as "billionaire boys and their toys" but Torres says the danger comes from the amount of power and influence these billionaires have to make policy and spend money." There have been, at least a dozen core books from Bostrom, Kelly, Kurzweil, Shannon etc... that have discussed this concept in precise and exacting detail. Arguably the whole reason to make computers has been to make AGI/ASI since the first mechanical turk etc... So I'm not really seeing a future without ASI and despite a lot of talk inside computing - for the entire history of computing - it's happening now, and everyone is surprised for some reason. [1]Here's my position in more detail: https://kemendo.com/Beyond-Human-Limits.html https://kemendo.com/Beyond-Human-Limits.html [2] https://www.youtube.com/watch?v=exhdfIPzj24 https://www.youtube.com/watch?v=exhdfIPzj24
- zfg 2y ago
  > Can someone give an alternative vision of the next 100-200 years that doesn't include AGI/ASI? I don't see it. At least one that isn't a nuclear wasteland. Eh? Without artificial super intelligence the future is a nuclear wasteland? How does that follow? Why wouldn't an artificial super intelligence conclude that humans are the problem and nuke the lot?
  - AndrewKemendo 2y ago
    My claim is that the only world that doesn’t make AGI/ASI is one where humanity is wiped out first
m3kw9 2y ago
To the people in the BC era this is already techno-utopian, we just take it for granted
throw0101d 2y ago
One has to ask whose utopia? Whose vision is being implemented? See stories from a few days ago, "We need network societies, not network states": * https://news.ycombinator.com/item?id=42941924 https://news.ycombinator.com/item?id=42941924 Or perhaps "The bro-ligarchs have a vision for the new Trump term": > All of these men see themselves as the heroes or protagonists in their own sci-fi saga. And a key part of being a “technological superman” — or ubermensch, as the German philosopher Friedrich Nietzsche would say — is that you’re above the law. Common-sense morality doesn’t apply to you because you’re a superior being on a superior mission. Thiel, it should be noted, is a big Nietzsche fan, though his is an extremely selective reading of the philosopher’s work. > The ubermensch ideology helps explain the broligarchs’ disturbing gender politics. “The ‘bro’ part of broligarch is not incidental to this — it’s built on this idea that not only are these guys superior, they are superior because they’re guys,” Harrington said. […] > The so-called network state is “a fancy name for tech authoritarianism,” journalist Gil Duran, who has spent the past year reporting on these building projects, told me. “The idea is to build power over the long term by controlling money, politics, technology, and land.” * https://www.vox.com/future-perfect/395646/trump-inauguration-broligarchs-musk-zuckerberg-bezos-thiel https://www.vox.com/future-perfect/395646/trump-inauguration...
tim333 2y ago
The article is surprisingly positive given the headline. On the plus side we may have: >trillions of super happy people who basically live forever and on the minus: >maybe some kind of "own goal" or "disaster" brought about by tech companies — like mass unemployment in a short time It sounds like the upside outweighs the downside there. By the way I'm not sure about "moral obligation to colonize as much of the universe as possible" but it's quite likely some will want to colonize like Musk with Mars. There are some logistical issues there though.
- maxerickson 2y ago
  Ah yes, Pascal's wager. Better believe in god just to be sure.

49 points by pseudolus 2y ago

A fun thought experiment is describing modern living to someone in the early Georgian era in an uncharitable way. It would be relatively easy to sell it is a godless industrial landscape. A city dweller could feasibly go days without seeing a tree or even leaving their apartment. Threat of annihilation looming over everyone's heads and the various major armies that can reach out and do what they like in most parts of the world. Constant risk from authoritarian ideologies that spread like wildfire. The future is always scary. The present is pretty scary. The past was much less comfortable.

hsuduebc2 2y ago
You are right. But I thinking it's fair to say that industrial progress did have some consequences that could be easily prevented. Mostly at the beginning. But I do not like this alarmist view either. It's usually useless and just make people feeling more endangered which is not a good prerequisite for racional debate. I would be worried more not about the technologies it self but by their human overlords.
deleted 2y ago
[deleted]

refurb 2y ago

[flagged]

deleted 2y ago
[deleted]

AdieuToLogic 2y ago

Back in 1995, Carl Sagan had a relevant thought: I have a foreboding of an America in my children's or grandchildren's time - [...] when awesome technological powers are in the hands of a very few, and no one representing the public interest can even grasp the issues; when the people have lost the ability to set their own agendas or knowledgeably question those in authority; when, clutching our crystals and nervously consulting our horoscopes, our critical faculties in decline, unable to distinguish between what feels good and what’s true, we slide, almost without noticing, back into superstition and darkness.[0] 0 - https://www.azquotes.com/quote/1069554 https://www.azquotes.com/quote/1069554

zfg 2y ago
More here: https://www.openculture.com/2025/02/carl-sagan-predicts-the-decline-of-america-unable-to-know-whats-true.html https://www.openculture.com/2025/02/carl-sagan-predicts-the-...
kcplate 2y ago
I have seen that quote dragged out lately as some sort of prophecy of 2025…but it really speaks to the exact opposite in my opinion. Smartphones and the internet have given the layperson awesome technology literally in their hands. The entire election of 2024 was driven by the people specifically questioning those in authority…and just what exactly is the “public interest”? Is it what the public demands? Or is it what some elite few say it should be? Seems to me whatever time that Sagan envisioned, it certainly isn’t this time we are in right now.
- bdangubic 2y ago
  The entire election of 2024 was driven by the people specifically questioning those in authority you really think this???! so we should expect the same in 2028 but at much larger scale now though, as now we have actual authority who is not caring for either constitution or the rule of law, left side winning all 50 states? I think not, the questioning of authority is what people in power make people without think what actually happen… :)
- xvector 2y ago
  I 100% agree. Anyone can use the Sagan quote to justify why they lost. The quote is just used as a fancy way of calling people with different views stupid. "Society is failing because people are dumber, and thats why they voted for X, when a smart person like me would have voted for Y!"
  - __loam 2y ago
    This is such a reductive view of what happened in 2024. Oligarchs openly supported a political candidate that has attacked experts time and again on things that are provably true because it's inconvenient to their narrative. We know Musk spent hundreds of millions of dollars supporting one side because it was in his interest. Mark Zuckerberg loves that side because they're not going to regulate his objectively harmful platform. Since this administration entered office, it has attacked agencies and organizations that do things like prepare for climate change, educate our children, provide essential Healthcare to 10s of millions of Americans, do basic scientific research, and protect consumers from financial fraud. Even if you get rid of these organizations, climate disasters will still happen (and insurance companies will still pull out of states like Florida and California), kids still need basic education to function in a high technology society, people are still getting sick, China is pulling ahead on public research, and fraudsters are still stealing from the American public. This administration is bad because it's attacking the career public servants and agencies that are a net positive for our society. We get back more than the investment here in innumerable ways. It is stupid to try and dismantle the parts of government that work and work well. So shut the hell up with this if they were smart they would have voted for Y bullshit as if this was ever a good faith alternative to the liberal program the dems ran on.

deleted 2y ago

[deleted]

AndrewKemendo 2y ago

Can someone give an alternative vision of the next 100-200 years that doesn't include AGI/ASI? I don't see it. At least one that isn't a nuclear wasteland. I have thought about this every day for 32 years and don't know of any possible alternative future than one run alongside or by Artificial Superhuman Systems. I just can't possibly look into the future and expect the current state of machines to plateau here and stop. In fact that's my life's work and has been so since I was 13.[1] I even discussed this at the AGI conference in Laval in 2014 with Turing winner Yoshua Bengio [2] The Overcoming Bias blog, which turned into Less wrong, was the best place for these discussions until it became a cult of personality around 2013. The SIAI and all the mess that it became at least was a chance to discuss it. There's no serious place to discuss it. AGI discussion went from nerds that everyone pointed and laughed at, to now being an imminent existential threat "To the average person, this all sounds like the plot of a sci-fi novel. There might be a tendency to dismiss a techno-utopia as "billionaire boys and their toys" but Torres says the danger comes from the amount of power and influence these billionaires have to make policy and spend money." There have been, at least a dozen core books from Bostrom, Kelly, Kurzweil, Shannon etc... that have discussed this concept in precise and exacting detail. Arguably the whole reason to make computers has been to make AGI/ASI since the first mechanical turk etc... So I'm not really seeing a future without ASI and despite a lot of talk inside computing - for the entire history of computing - it's happening now, and everyone is surprised for some reason. [1]Here's my position in more detail: https://kemendo.com/Beyond-Human-Limits.html https://kemendo.com/Beyond-Human-Limits.html [2] https://www.youtube.com/watch?v=exhdfIPzj24 https://www.youtube.com/watch?v=exhdfIPzj24

zfg 2y ago
> Can someone give an alternative vision of the next 100-200 years that doesn't include AGI/ASI? I don't see it. At least one that isn't a nuclear wasteland. Eh? Without artificial super intelligence the future is a nuclear wasteland? How does that follow? Why wouldn't an artificial super intelligence conclude that humans are the problem and nuke the lot?
- AndrewKemendo 2y ago
  My claim is that the only world that doesn’t make AGI/ASI is one where humanity is wiped out first

m3kw9 2y ago

To the people in the BC era this is already techno-utopian, we just take it for granted

throw0101d 2y ago

One has to ask whose utopia? Whose vision is being implemented? See stories from a few days ago, "We need network societies, not network states": * https://news.ycombinator.com/item?id=42941924 https://news.ycombinator.com/item?id=42941924 Or perhaps "The bro-ligarchs have a vision for the new Trump term": > All of these men see themselves as the heroes or protagonists in their own sci-fi saga. And a key part of being a “technological superman” — or ubermensch, as the German philosopher Friedrich Nietzsche would say — is that you’re above the law. Common-sense morality doesn’t apply to you because you’re a superior being on a superior mission. Thiel, it should be noted, is a big Nietzsche fan, though his is an extremely selective reading of the philosopher’s work. > The ubermensch ideology helps explain the broligarchs’ disturbing gender politics. “The ‘bro’ part of broligarch is not incidental to this — it’s built on this idea that not only are these guys superior, they are superior because they’re guys,” Harrington said. […] > The so-called network state is “a fancy name for tech authoritarianism,” journalist Gil Duran, who has spent the past year reporting on these building projects, told me. “The idea is to build power over the long term by controlling money, politics, technology, and land.” * https://www.vox.com/future-perfect/395646/trump-inauguration-broligarchs-musk-zuckerberg-bezos-thiel https://www.vox.com/future-perfect/395646/trump-inauguration...

tim333 2y ago

The article is surprisingly positive given the headline. On the plus side we may have: >trillions of super happy people who basically live forever and on the minus: >maybe some kind of "own goal" or "disaster" brought about by tech companies — like mass unemployment in a short time It sounds like the upside outweighs the downside there. By the way I'm not sure about "moral obligation to colonize as much of the universe as possible" but it's quite likely some will want to colonize like Musk with Mars. There are some logistical issues there though.

maxerickson 2y ago
Ah yes, Pascal's wager. Better believe in god just to be sure.