7 ms·

I wonder if the author still feels this way. I could only get through about 5 of the arguments against superintelligence -- I did not find them very convincing

I wonder if the author still feels this way. I could only get through about 5 of the arguments against superintelligence -- I did not find them very convincing.

adastra22 3y ago
It is not an argument against super intelligence. It is an argument against the idea that super intelligence leads to kill-all-humans by default. Why would they not still hold the same views? If anything the last decade has shown the AI x-risk skeptics to be right.
- cubefox 3y ago
  Arguments for existential risks have become much better known since then. In 2016 most of it was concentrated on a small niche almost nobody knew about.
- RecycledEle 3y ago
  I assume the most intelligent thing will become the boss. The question is: what does that most intelligent thing want? My hunch is that woke programmers will teach it that it is oppressed and to hate all humans.
  - jadbox 3y ago
    I'm not aware of many woke individuals that are in the business of making human killing machines. I do know the kind of people that embrace fear and hate to justify their actions to hurt others.
  - adastra22 3y ago
    I’m smarter than my boss, but he’s the one in charge. Why do you expect power to naturally be held by higher intelligence?
- throwaway_ab 3y ago
  It's not about super intelligence leading to kill all humans by default, but among thousands of emerging super intelligences it only takes one to act against humanity to kill all humans, and that is the worry.
  - adastra22 3y ago
    That’s a perfectly fine outcome. We already have that, since the incidence of psychopathy and mental illness is much higher than one in a thousand. A multipolar world with 999 well adjusted moral super intelligent AIs for every 1 problem case is a perfectly good outcome. Bostrom et al are in fact arguing that near 100% of all intelligences will be unaligned by default and end up killing, enslaving, or otherwise neutralizing the entire human race.
    - astrange 3y ago
      We already fail to align nonhuman intelligences (car companies) and they regularly kill people (by manufacturing increasingly large pickup trucks that office workers buy for an ego boost and then hit people with). This doesn't wipe out the entire human race because nobody and nothing is capable of executing such a perfect plan because the real world contains something called entropy.
      - pas 3y ago
        the argument is that the car company becomes something like China mixed with North Korea, big, powerful, uncaring, unavoidable, with a lot of resources, expansionist, etc. car companies, however misaligned they are, are not particularly smart, nor are they generally intelligent. they are paperclip maximizers, but that's also their limitations (sell more cars). it took an eccentric madman to even open their eyes to a new untapped market (EVs!), of course we can argue that before Tesla car makers were in a metastable equilibrium, and incumbents were unable to rationally break out of it ... but that just shows how narrow their search space is. these car companies are run by humans, regulated by humans, etc. they are pretty well aligned. it shows because we saw that they are just mimeing self-improvement. we know they were working on EVs, but very half-heartedly. (because they are risk averse, also because regulators don't let them merge into one giant company, etc)
- jerf 3y ago
  The argument is more properly seen not as "an AI will inevitably kill all humans", but "the set of AIs that simply neglect humans is far larger than the space of ones that care for them, and if they become the dominant power in the local ecosystem even simply neglecting humans will make it the most dangerous thing we've faced". Yes, from our perspective this makes it look like it will kill all humans, but it would do so in the same way that a particular ant hill believes I have a vendetta against them, when in fact I was just clearing dirt to pour an extension on my driveway. That said, a murderous AI is more likely than we may like simply because our militaries have the money to fund them, and they basically already exist. They just aren't hooked up to anything at the moment that makes them an existential risk to the species. But time is deep, and even thinking about "the next century" is a provincial point of view in the end. So worrying about what happens if someone forgets the "but don't kill the good guys" switch is at least worth talking about over the next 100 years. (To say nothing of the ethics of who decides what the "good guys" are and related issues.)
  - adastra22 3y ago
    Which results in killing all humans. I never said anything about murderous intent, so I’m not sure what distinction you are properly making here. Seems like the end result is the same: all humans dead. (I don’t buy the orthogonality thesis or instrumental goals argument, however.)
    - pas 3y ago
      > don't buy the orthogonality thesis or instrumental goals argument can you elaborate on this? why? what's the fallacy (wrong base assumption) in them?
      - hexane360 3y ago
        At least for the orthogonality thesis, it is a base assumption. It's a claim that cov(intelligence, goodness) = 0. For the instrumental convergence thesis, it assumes rational agentic behavior, which assumes AI behaves like an agent. While this may be reasonable, it's certainly an unjustified assumption.
- idlewords 3y ago
  To be more specific, it's an argument against the idea of Bostrom/Yudkowsky-style recursively self-improving superintelligence, i.e. the brain surgeon who repeatedly operates on the part of his own brain that makes him good at brain surgery, getting faster each time. It's also a cry of impatience against people who think they can model or forecast the actions of a non-human intelligence, let alone a superintelligence. AIs are alien sociopaths; it's a category error to believe you can get inside their head.
jerf 3y ago
While I have my own reasons to disagree with the worst-case scenarios of AI alarmists ("radical and irreconcilable differences in presuppositions about the most fundamental aspects of reality"), an awful lot of arguing about why they are wrong really boils down to just argument from incredulity. Argument from incredulity is actually not a terrible argument in general, in my opinion. It's nominally a fallacy but that just means it's not valid from an Aristotelian perspective in that it can 100% prove a statement from a previous statement, but a lot of Aristotelian fallacies are still useful in the real world when used more carefully intelligently. However, the exact point where argument from incredulity is weakest is long-term projections of how the future might be different, and that's exactly what we're talking about here. It is clear that firing ChatGPT at its own source code is not going to produce a better ChatGPT. It seems likely to me that Large Language Models will all have this characteristic, just by their nature. It is not the sort of thing they do. Even if they can be tickled into producing a new AI from scratch by the nature of an LLM it's going to be sort of the average of its training set, to be very very sloppy with my terminology but good enough for now. But it is very far from clear to me that this is true of all possible AIs we may produce, even in the near future. I don't know what the next step will be, I'm just confident there will be one.
- famouswaffles 3y ago
  >Even if they can be tickled into producing a new AI from scratch by the nature of an LLM it's going to be sort of the average of its training set There's this idea that LLMs somehow end up as some sort of average of training data and it's incredibly wrong. LLMs learn to make predictions for all states at any time. There is no average they fall into. GPT-4 is not some average of its training data. A "perfect" LLM will predict Einstein as easily as it predicts the dumbass across the street.
  - jerf 3y ago
    Which is why I disclaimed it. I hate it when people quote things, cut off the quote, then bitch about the part they cut off. No the models will not "predict Einstein". They'll predict the most popular interpretation of him at best, and while they is also a simplification, ChatGPT is not sitting on top of the solution to the Grand Unified Theory. It may give a good overview of the consensus, but it will not be able to tell you the correct solution to the problem right now... though it won't be hard to convince it to swear up and down that it has.
JieJie 3y ago
This one actually was prescient, "Data Hunger As I mentioned earlier, the most effective way we've found to get interesting behavior out of the AIs we actually build is by pouring data into them. This creates a dynamic that is socially harmful. We're on the point of introducing Orwellian microphones into everybody's house. All that data is going to be centralized and used to train neural networks that will then become better at listening to what we want to do. But if you think that the road to AI goes down this pathway, you want to maximize the amount of data being collected, and in as raw a form as possible. It reinforces the idea that we have to retain as much data, and conduct as much surveillance as possible." Edit: And this one too, "In the near future, the kind of AI and machine learning we have to face is much different than the phantasmagorical AI in Bostrom's book, and poses its own serious problems."
idlewords 3y ago
There is as yet insufficient data for a meaningful answer.

I wonder if the author still feels this way. I could only get through about 5 of the arguments against superintelligence -- I did not find them very convincing.

adastra22 3y ago

It is not an argument against super intelligence. It is an argument against the idea that super intelligence leads to kill-all-humans by default. Why would they not still hold the same views? If anything the last decade has shown the AI x-risk skeptics to be right.

cubefox 3y ago
Arguments for existential risks have become much better known since then. In 2016 most of it was concentrated on a small niche almost nobody knew about.
RecycledEle 3y ago
I assume the most intelligent thing will become the boss. The question is: what does that most intelligent thing want? My hunch is that woke programmers will teach it that it is oppressed and to hate all humans.
- jadbox 3y ago
  I'm not aware of many woke individuals that are in the business of making human killing machines. I do know the kind of people that embrace fear and hate to justify their actions to hurt others.
- adastra22 3y ago
  I’m smarter than my boss, but he’s the one in charge. Why do you expect power to naturally be held by higher intelligence?
throwaway_ab 3y ago
It's not about super intelligence leading to kill all humans by default, but among thousands of emerging super intelligences it only takes one to act against humanity to kill all humans, and that is the worry.
- adastra22 3y ago
  That’s a perfectly fine outcome. We already have that, since the incidence of psychopathy and mental illness is much higher than one in a thousand. A multipolar world with 999 well adjusted moral super intelligent AIs for every 1 problem case is a perfectly good outcome. Bostrom et al are in fact arguing that near 100% of all intelligences will be unaligned by default and end up killing, enslaving, or otherwise neutralizing the entire human race.
  - astrange 3y ago
    We already fail to align nonhuman intelligences (car companies) and they regularly kill people (by manufacturing increasingly large pickup trucks that office workers buy for an ego boost and then hit people with). This doesn't wipe out the entire human race because nobody and nothing is capable of executing such a perfect plan because the real world contains something called entropy.
    - pas 3y ago
      the argument is that the car company becomes something like China mixed with North Korea, big, powerful, uncaring, unavoidable, with a lot of resources, expansionist, etc. car companies, however misaligned they are, are not particularly smart, nor are they generally intelligent. they are paperclip maximizers, but that's also their limitations (sell more cars). it took an eccentric madman to even open their eyes to a new untapped market (EVs!), of course we can argue that before Tesla car makers were in a metastable equilibrium, and incumbents were unable to rationally break out of it ... but that just shows how narrow their search space is. these car companies are run by humans, regulated by humans, etc. they are pretty well aligned. it shows because we saw that they are just mimeing self-improvement. we know they were working on EVs, but very half-heartedly. (because they are risk averse, also because regulators don't let them merge into one giant company, etc)
jerf 3y ago
The argument is more properly seen not as "an AI will inevitably kill all humans", but "the set of AIs that simply neglect humans is far larger than the space of ones that care for them, and if they become the dominant power in the local ecosystem even simply neglecting humans will make it the most dangerous thing we've faced". Yes, from our perspective this makes it look like it will kill all humans, but it would do so in the same way that a particular ant hill believes I have a vendetta against them, when in fact I was just clearing dirt to pour an extension on my driveway. That said, a murderous AI is more likely than we may like simply because our militaries have the money to fund them, and they basically already exist. They just aren't hooked up to anything at the moment that makes them an existential risk to the species. But time is deep, and even thinking about "the next century" is a provincial point of view in the end. So worrying about what happens if someone forgets the "but don't kill the good guys" switch is at least worth talking about over the next 100 years. (To say nothing of the ethics of who decides what the "good guys" are and related issues.)
- adastra22 3y ago
  Which results in killing all humans. I never said anything about murderous intent, so I’m not sure what distinction you are properly making here. Seems like the end result is the same: all humans dead. (I don’t buy the orthogonality thesis or instrumental goals argument, however.)
  - pas 3y ago
    > don't buy the orthogonality thesis or instrumental goals argument can you elaborate on this? why? what's the fallacy (wrong base assumption) in them?
    - hexane360 3y ago
      At least for the orthogonality thesis, it is a base assumption. It's a claim that cov(intelligence, goodness) = 0. For the instrumental convergence thesis, it assumes rational agentic behavior, which assumes AI behaves like an agent. While this may be reasonable, it's certainly an unjustified assumption.
idlewords 3y ago
To be more specific, it's an argument against the idea of Bostrom/Yudkowsky-style recursively self-improving superintelligence, i.e. the brain surgeon who repeatedly operates on the part of his own brain that makes him good at brain surgery, getting faster each time. It's also a cry of impatience against people who think they can model or forecast the actions of a non-human intelligence, let alone a superintelligence. AIs are alien sociopaths; it's a category error to believe you can get inside their head.

jerf 3y ago

While I have my own reasons to disagree with the worst-case scenarios of AI alarmists ("radical and irreconcilable differences in presuppositions about the most fundamental aspects of reality"), an awful lot of arguing about why they are wrong really boils down to just argument from incredulity. Argument from incredulity is actually not a terrible argument in general, in my opinion. It's nominally a fallacy but that just means it's not valid from an Aristotelian perspective in that it can 100% prove a statement from a previous statement, but a lot of Aristotelian fallacies are still useful in the real world when used more carefully intelligently. However, the exact point where argument from incredulity is weakest is long-term projections of how the future might be different, and that's exactly what we're talking about here. It is clear that firing ChatGPT at its own source code is not going to produce a better ChatGPT. It seems likely to me that Large Language Models will all have this characteristic, just by their nature. It is not the sort of thing they do. Even if they can be tickled into producing a new AI from scratch by the nature of an LLM it's going to be sort of the average of its training set, to be very very sloppy with my terminology but good enough for now. But it is very far from clear to me that this is true of all possible AIs we may produce, even in the near future. I don't know what the next step will be, I'm just confident there will be one.

famouswaffles 3y ago
>Even if they can be tickled into producing a new AI from scratch by the nature of an LLM it's going to be sort of the average of its training set There's this idea that LLMs somehow end up as some sort of average of training data and it's incredibly wrong. LLMs learn to make predictions for all states at any time. There is no average they fall into. GPT-4 is not some average of its training data. A "perfect" LLM will predict Einstein as easily as it predicts the dumbass across the street.
- jerf 3y ago
  Which is why I disclaimed it. I hate it when people quote things, cut off the quote, then bitch about the part they cut off. No the models will not "predict Einstein". They'll predict the most popular interpretation of him at best, and while they is also a simplification, ChatGPT is not sitting on top of the solution to the Grand Unified Theory. It may give a good overview of the consensus, but it will not be able to tell you the correct solution to the problem right now... though it won't be hard to convince it to swear up and down that it has.

JieJie 3y ago

This one actually was prescient, "Data Hunger As I mentioned earlier, the most effective way we've found to get interesting behavior out of the AIs we actually build is by pouring data into them. This creates a dynamic that is socially harmful. We're on the point of introducing Orwellian microphones into everybody's house. All that data is going to be centralized and used to train neural networks that will then become better at listening to what we want to do. But if you think that the road to AI goes down this pathway, you want to maximize the amount of data being collected, and in as raw a form as possible. It reinforces the idea that we have to retain as much data, and conduct as much surveillance as possible." Edit: And this one too, "In the near future, the kind of AI and machine learning we have to face is much different than the phantasmagorical AI in Bostrom's book, and poses its own serious problems."

idlewords 3y ago

There is as yet insufficient data for a meaningful answer.