3 ms·

> If, say, ChatGPT is superintelligent, the only way it can hurt you is through insult or persuasion. if the singularity happens, the worst won't come thru dir

by panta 3y ago

> If, say, ChatGPT is superintelligent, the only way it can hurt you is through insult or persuasion. if the singularity happens, the worst won't come thru direct violence. A superintelligent entity will be able to persuade everyone to act against humanity interests (because over time people will be less able to think on their own and won't have anymore the ability to evaluate others' ideas, especially if coming from something more intelligent). People will defer almost all high-level thinking to the machine. And this is not the only negative consequence.

hoseja 3y ago
This has happened some ten thousand years ago when states evolved.
- fishnchips 3y ago
  No it didn’t. And states didn’t evolve ten thousand years ago. Moreover, today’s definition of state would barely apply throughout the 19th century, let alone the dawn of civilization.
  - kmeisthax 3y ago
    Ok, so let's put this another way: if a singularity is just a superintelligence that can outsmart individual humans, then we're already several singularities deep. At the very least, any civilization larger than an individual tribe is already pushing the bounds of human comprehension[0]. While I will give you that states used to be far weaker on average, that doesn't mean they aren't singularities. A city state with no ability to project power beyond the city walls is still capable of superintelligence. Furthermore, part of the rising power of nation states is that said states created another singularity: the limited-liability corporation. Both of these singularities work because they allow aligning millions of individual human agents towards a common goal. Humans cannot naturally cooperate in structures that large. The reason why this works is because of delegation of work and specialization. You don't have aggregate piles of labor; you have farmers, architects, and merchants who are cooperating for some incentive. If you alternatively demand that your singularity be a technological innovation rather than a social one, then fine. We're still several singularities deep. Humans used to be exclusively hunter-gatherers until we exhausted the carrying capacity of the land and invented agriculture. This enabled many of the social changes I talked about above - hunting and tribal units are sort of interrelated, and so are agriculture and city states. Metallurgy is such a powerful technology that the term "stone age", to refer to civilizations that don't have it, is widely seen as an insult even though stone age civilizations were still quite powerful. Yes, I can think of several other reasons why Ray Kurzweil would claim these are not singularities. No I don't consider that convincing. The point is that humans have already been dealing with things that are outside our understanding and dramatically change the human condition for thousands of years. If we are going to talk about AI as a technological singularity, then we need to consider how that will interact with what we already know about prior ones, rather than just wax philosophical about the religious implications of our god computers. [0] Dunbar's Number postulates a maximum limit of socially meaningful friendships at around 150 or so. After this point you need to embrace managerial structures, hierarchy, and the problems that result from such.
    - fishnchips 3y ago
      As a sociologist by education I find such sweeping theories useless. Human and social history is incredibly complex with hundreds of factors and a good deal of pure randomness involved. Grand theories aiming to reduce this complexity to a single factor are either plain wrong or so vague that they can’t be proven wrong, which limits their usefulness as tools of thought. You’ll have the same issue with historical Marxism as well as with pop science like your man Harari.
      - kmeisthax 3y ago
        I didn't even know who the hell Harari was until you mentioned him.
        fishnchips 3y ago
        OK, the phrase about the exhausting the capacity of the land had that vibe. Because there’s literally no evidence that this is what happened. Agrarian and hunter-gatherer societies coexisted side by side for millennia and becoming a settled community was hardly an upgrade based on the data we have (diet, health, leisure time and longevity).

by panta 3y ago

hoseja 3y ago

This has happened some ten thousand years ago when states evolved.

fishnchips 3y ago
No it didn’t. And states didn’t evolve ten thousand years ago. Moreover, today’s definition of state would barely apply throughout the 19th century, let alone the dawn of civilization.
- kmeisthax 3y ago
  Ok, so let's put this another way: if a singularity is just a superintelligence that can outsmart individual humans, then we're already several singularities deep. At the very least, any civilization larger than an individual tribe is already pushing the bounds of human comprehension[0]. While I will give you that states used to be far weaker on average, that doesn't mean they aren't singularities. A city state with no ability to project power beyond the city walls is still capable of superintelligence. Furthermore, part of the rising power of nation states is that said states created another singularity: the limited-liability corporation. Both of these singularities work because they allow aligning millions of individual human agents towards a common goal. Humans cannot naturally cooperate in structures that large. The reason why this works is because of delegation of work and specialization. You don't have aggregate piles of labor; you have farmers, architects, and merchants who are cooperating for some incentive. If you alternatively demand that your singularity be a technological innovation rather than a social one, then fine. We're still several singularities deep. Humans used to be exclusively hunter-gatherers until we exhausted the carrying capacity of the land and invented agriculture. This enabled many of the social changes I talked about above - hunting and tribal units are sort of interrelated, and so are agriculture and city states. Metallurgy is such a powerful technology that the term "stone age", to refer to civilizations that don't have it, is widely seen as an insult even though stone age civilizations were still quite powerful. Yes, I can think of several other reasons why Ray Kurzweil would claim these are not singularities. No I don't consider that convincing. The point is that humans have already been dealing with things that are outside our understanding and dramatically change the human condition for thousands of years. If we are going to talk about AI as a technological singularity, then we need to consider how that will interact with what we already know about prior ones, rather than just wax philosophical about the religious implications of our god computers. [0] Dunbar's Number postulates a maximum limit of socially meaningful friendships at around 150 or so. After this point you need to embrace managerial structures, hierarchy, and the problems that result from such.
  - fishnchips 3y ago
    As a sociologist by education I find such sweeping theories useless. Human and social history is incredibly complex with hundreds of factors and a good deal of pure randomness involved. Grand theories aiming to reduce this complexity to a single factor are either plain wrong or so vague that they can’t be proven wrong, which limits their usefulness as tools of thought. You’ll have the same issue with historical Marxism as well as with pop science like your man Harari.
    - kmeisthax 3y ago
      I didn't even know who the hell Harari was until you mentioned him.
      - fishnchips 3y ago
        OK, the phrase about the exhausting the capacity of the land had that vibe. Because there’s literally no evidence that this is what happened. Agrarian and hunter-gatherer societies coexisted side by side for millennia and becoming a settled community was hardly an upgrade based on the data we have (diet, health, leisure time and longevity).