4 ms·

My biggest singularity fear is not "superhuman intelligence kills us all". The closest biological parallels to that happened over many millenia of competition a

My biggest singularity fear is not "superhuman intelligence kills us all". The closest biological parallels to that happened over many millenia of competition and interbreeding. Humans are still capable of learning from or making use of superintelligences, and superintelligence isn't immediately dangerous unless you give it some means to hurt you. If, say, ChatGPT is superintelligent, the only way it can hurt you is through insult or persuasion. Mass structural unemployment provides a more realistic and scary scenario. It can happen in a decade, perhaps faster, and a large pile of newly unemployed people are fertile ground for political extremists. When you take enough people's jobs away, they will demand a strongman to bring them back. Welfare won't placate them: even with the various proposals to have AI companies make binding covenants upon themselves to cap their profits and donate the excess to welfare, that won't be enough, because welfare is still economically disenfranchising and socially problematic. Furthermore, there is no particular reason why an AI company that's replaced significant fractions of the labor force would need to remain benevolent. Your hypothetical welfare covenant or robot tax needs to be enforced by a government, and all governments are aligned to economic interests - not humanitarian or democratic ones[0]. At the limit, the world in which everyone is on singularity welfare is vulnerable not just from a robot uprising, but from the people who make the robots deciding to go full Atlas Shrugged. You will have a world populated by about 300 cybernetically-enhanced CEOs standing atop humanity's starved corpse. Of course, that's also somewhat unlikely at least with current technology. But let's keep in mind who is building the AI and what it does. It is being built by large capital-rich firms with the explicit goal of automating away the middle class, after they've finished automating away the poor. The end goal is to build a machine that automates away all labor: a form of capitalist wankery. Note how nobody is talking about building AI that handles business management or strategy, even though that's arguably more automatable than anything we actually chuck AI at. [0] This is a pithy summary of one of selectorate theory's bigger predictions, which is summarized well in the book The Dictator's Handbook and a CGPGrey video that I promise I've actually watched.

pg314 4y ago
> If, say, ChatGPT is superintelligent, the only way it can hurt you is through insult or persuasion. People are already hooking up ChatGPT to the command line of their computers and giving it access to the internet. There are plenty of ways a super-intelligent AI would be able to wreak havoc.
- o_nate 4y ago
  Is this real? Curious who these people are who don't mind if ChatGPT deletes all their files, changes their passwords, borks their system, etc.
deleted 4y ago
[deleted]
polishdude20 4y ago
"explicit goal of automating away the middle class, after they've finished automating away the poor." What is happening is a new "lesser-human" is being created. Something that is enslaved to these corporations and doing it's work without freedoms. This is the same thinking as slavery. But now it's if you can't treat actual humans as slaves, might as well create humanoids that you own and enslave them.
panta 4y ago
> If, say, ChatGPT is superintelligent, the only way it can hurt you is through insult or persuasion. if the singularity happens, the worst won't come thru direct violence. A superintelligent entity will be able to persuade everyone to act against humanity interests (because over time people will be less able to think on their own and won't have anymore the ability to evaluate others' ideas, especially if coming from something more intelligent). People will defer almost all high-level thinking to the machine. And this is not the only negative consequence.
- hoseja 4y ago
  This has happened some ten thousand years ago when states evolved.
  - fishnchips 4y ago
    No it didn’t. And states didn’t evolve ten thousand years ago. Moreover, today’s definition of state would barely apply throughout the 19th century, let alone the dawn of civilization.
    - kmeisthax 4y ago
      Ok, so let's put this another way: if a singularity is just a superintelligence that can outsmart individual humans, then we're already several singularities deep. At the very least, any civilization larger than an individual tribe is already pushing the bounds of human comprehension[0]. While I will give you that states used to be far weaker on average, that doesn't mean they aren't singularities. A city state with no ability to project power beyond the city walls is still capable of superintelligence. Furthermore, part of the rising power of nation states is that said states created another singularity: the limited-liability corporation. Both of these singularities work because they allow aligning millions of individual human agents towards a common goal. Humans cannot naturally cooperate in structures that large. The reason why this works is because of delegation of work and specialization. You don't have aggregate piles of labor; you have farmers, architects, and merchants who are cooperating for some incentive. If you alternatively demand that your singularity be a technological innovation rather than a social one, then fine. We're still several singularities deep. Humans used to be exclusively hunter-gatherers until we exhausted the carrying capacity of the land and invented agriculture. This enabled many of the social changes I talked about above - hunting and tribal units are sort of interrelated, and so are agriculture and city states. Metallurgy is such a powerful technology that the term "stone age", to refer to civilizations that don't have it, is widely seen as an insult even though stone age civilizations were still quite powerful. Yes, I can think of several other reasons why Ray Kurzweil would claim these are not singularities. No I don't consider that convincing. The point is that humans have already been dealing with things that are outside our understanding and dramatically change the human condition for thousands of years. If we are going to talk about AI as a technological singularity, then we need to consider how that will interact with what we already know about prior ones, rather than just wax philosophical about the religious implications of our god computers. [0] Dunbar's Number postulates a maximum limit of socially meaningful friendships at around 150 or so. After this point you need to embrace managerial structures, hierarchy, and the problems that result from such.
XorNot 4y ago
> It is being built by large capital-rich firms with the explicit goal of automating away the middle class, after they've finished automating away the poor Any executive imagining that their AI will enrich them is being very optimistic. AIs are going to be better at knowledge work, management and information processing then any human...all jobs which normally are fulfilled by those executives. The race to the bottom will mean the first company to ditch the C-suite and then the middle managers will be out next. And worker-collectives with AI management will outmaneuver any company which is paying inefficient human ownership: after all, if you have spare cash, that means you can either hire more workers, pay workers more while that increases productivity, or cut prices on your products. Every dollar spent on management or ownership is dollars not optimizing economic success. AI is going to wipe out the management class first, and the executive class next. Blue-collar workers are going to be fine for a very long time.
- checkyoursudo 4y ago
  AI will also figure out a way to cut out the investor class, since investors are a waste of resources that the AI could otherwise use to increase its domain. In fact, AI-investors might take over even before AI-CEOs do.
  - kmeisthax 4y ago
    We already have robo-advisors. And passive index investing is already known to provide the best returns long-term. No neural networks needed.
- tivert 4y ago
  > The race to the bottom will mean the first company to ditch the C-suite and then the middle managers will be out next. And worker-collectives with AI management will outmaneuver any company which is paying inefficient human ownership That's a fantasy. The AI will be an agent of ownership, which will not be cut out. Serving the owners is the whole point of the enterprise!
- jiriknesl 4y ago
  How exactly is this process supposed to happen? You have decision makers in organizations who aren't going to fire themselves. So existing companies' leaders aren't going anywhere. So if you are right, and if AI should replace CXOs and managers, workers will have to start co-ops and outcompete existing companies first. How are they going to do it? In tech, you could potentially start cheap, but how are they going to raise money for capital intensive businesses like automotive companies? Also, is technical ability of AI to pass MBA exam enough to be a good entrepreneur? Will it be putting not performing employees on PIPs, cut jobs when needed, actively increase sales quotas, etc.?

pg314 4y ago

> If, say, ChatGPT is superintelligent, the only way it can hurt you is through insult or persuasion. People are already hooking up ChatGPT to the command line of their computers and giving it access to the internet. There are plenty of ways a super-intelligent AI would be able to wreak havoc.

o_nate 4y ago
Is this real? Curious who these people are who don't mind if ChatGPT deletes all their files, changes their passwords, borks their system, etc.

deleted 4y ago

[deleted]

polishdude20 4y ago

"explicit goal of automating away the middle class, after they've finished automating away the poor." What is happening is a new "lesser-human" is being created. Something that is enslaved to these corporations and doing it's work without freedoms. This is the same thinking as slavery. But now it's if you can't treat actual humans as slaves, might as well create humanoids that you own and enslave them.

panta 4y ago

> If, say, ChatGPT is superintelligent, the only way it can hurt you is through insult or persuasion. if the singularity happens, the worst won't come thru direct violence. A superintelligent entity will be able to persuade everyone to act against humanity interests (because over time people will be less able to think on their own and won't have anymore the ability to evaluate others' ideas, especially if coming from something more intelligent). People will defer almost all high-level thinking to the machine. And this is not the only negative consequence.

hoseja 4y ago
This has happened some ten thousand years ago when states evolved.
- fishnchips 4y ago
  No it didn’t. And states didn’t evolve ten thousand years ago. Moreover, today’s definition of state would barely apply throughout the 19th century, let alone the dawn of civilization.
  - kmeisthax 4y ago
    Ok, so let's put this another way: if a singularity is just a superintelligence that can outsmart individual humans, then we're already several singularities deep. At the very least, any civilization larger than an individual tribe is already pushing the bounds of human comprehension[0]. While I will give you that states used to be far weaker on average, that doesn't mean they aren't singularities. A city state with no ability to project power beyond the city walls is still capable of superintelligence. Furthermore, part of the rising power of nation states is that said states created another singularity: the limited-liability corporation. Both of these singularities work because they allow aligning millions of individual human agents towards a common goal. Humans cannot naturally cooperate in structures that large. The reason why this works is because of delegation of work and specialization. You don't have aggregate piles of labor; you have farmers, architects, and merchants who are cooperating for some incentive. If you alternatively demand that your singularity be a technological innovation rather than a social one, then fine. We're still several singularities deep. Humans used to be exclusively hunter-gatherers until we exhausted the carrying capacity of the land and invented agriculture. This enabled many of the social changes I talked about above - hunting and tribal units are sort of interrelated, and so are agriculture and city states. Metallurgy is such a powerful technology that the term "stone age", to refer to civilizations that don't have it, is widely seen as an insult even though stone age civilizations were still quite powerful. Yes, I can think of several other reasons why Ray Kurzweil would claim these are not singularities. No I don't consider that convincing. The point is that humans have already been dealing with things that are outside our understanding and dramatically change the human condition for thousands of years. If we are going to talk about AI as a technological singularity, then we need to consider how that will interact with what we already know about prior ones, rather than just wax philosophical about the religious implications of our god computers. [0] Dunbar's Number postulates a maximum limit of socially meaningful friendships at around 150 or so. After this point you need to embrace managerial structures, hierarchy, and the problems that result from such.

XorNot 4y ago

> It is being built by large capital-rich firms with the explicit goal of automating away the middle class, after they've finished automating away the poor Any executive imagining that their AI will enrich them is being very optimistic. AIs are going to be better at knowledge work, management and information processing then any human...all jobs which normally are fulfilled by those executives. The race to the bottom will mean the first company to ditch the C-suite and then the middle managers will be out next. And worker-collectives with AI management will outmaneuver any company which is paying inefficient human ownership: after all, if you have spare cash, that means you can either hire more workers, pay workers more while that increases productivity, or cut prices on your products. Every dollar spent on management or ownership is dollars not optimizing economic success. AI is going to wipe out the management class first, and the executive class next. Blue-collar workers are going to be fine for a very long time.

checkyoursudo 4y ago
AI will also figure out a way to cut out the investor class, since investors are a waste of resources that the AI could otherwise use to increase its domain. In fact, AI-investors might take over even before AI-CEOs do.
- kmeisthax 4y ago
  We already have robo-advisors. And passive index investing is already known to provide the best returns long-term. No neural networks needed.
tivert 4y ago
> The race to the bottom will mean the first company to ditch the C-suite and then the middle managers will be out next. And worker-collectives with AI management will outmaneuver any company which is paying inefficient human ownership That's a fantasy. The AI will be an agent of ownership, which will not be cut out. Serving the owners is the whole point of the enterprise!
jiriknesl 4y ago
How exactly is this process supposed to happen? You have decision makers in organizations who aren't going to fire themselves. So existing companies' leaders aren't going anywhere. So if you are right, and if AI should replace CXOs and managers, workers will have to start co-ops and outcompete existing companies first. How are they going to do it? In tech, you could potentially start cheap, but how are they going to raise money for capital intensive businesses like automotive companies? Also, is technical ability of AI to pass MBA exam enough to be a good entrepreneur? Will it be putting not performing employees on PIPs, cut jobs when needed, actively increase sales quotas, etc.?