6 ms·

But what argument is there to refute? It feels like Aquinas “proving” God’s existence by stating that it is self evident. They can’t point to an existing syste

But what argument is there to refute? It feels like Aquinas “proving” God’s existence by stating that it is self evident. They can’t point to an existing system that poses existential risk, because it doesn’t exist. They can’t point to a clear architecture for such a system, because we don’t know how to build it. So again, what can be refuted?

notahacker 3y ago
Seems apt the term "Bulverism" comes from CS Lewis, since he was also positing that an unseen, unfalsifiable entity would grant eternal reward to people that listened to him and eternal damnation to those that didn't...
- alasdair_ 3y ago
  The irony of critiquing Bulverism as a concept, not by attacking the idea itself, but instead by assuming it is wrong and attacking the character of the author, is staggeringly hilarious.
  - notahacker 3y ago
    I'm replying in agreement with someone who already pointed out the obvious flaw in labelling any questioning of the inspirations or motivations of AI researchers as "Bulverism": none of the stuff they're saying is actually a claim that can be falsified in the first place! I'm unconvinced by the position that the only valid means of casting doubt on a claim is through forensic examination of hard data that may be inaccessible to the interlocutor (like most people's bank accounts...), but whether that is or isn't a generally good approach is irrelevant here as we're talking about claims about courses of action to avoid hypothetical threats. I just noted it was a particularly useful rhetorical flourish when advocating acting on beliefs which aren't readily falsifiable, something CS Lewis was extremely proud of doing and certainly wouldn't have considered a character flaw! Ironically, your reply also failed to falsify anything I said and instead critiqued my assumed motivations for making the comment. It's Bulverism all the way down!
    - deltaninenine 3y ago
      Logical Induction has been successful in predicting future events.
      - notahacker 3y ago
        Sometimes it makes good predictions, sometimes bad. But "advances in AI might lead to Armageddon" isn't the only conclusion induction can reach. Induction can also lead to people concluding certain arguments seem to a mashup of traditional millennialist "end times" preoccupations with the sort of sci-fi they grew up with, or that this looks a lot like a movement towards regulatory capture. Ultimately any (possibly even all) these inferences from past trends and recent actions can be correct, but none of them are falsifiable. So I don't think it's a good idea to insist that people should be falsifying the idea that AI is a risk before we start questioning whether the behaviour of some of the entities on the list says more about their motivations than their words.
duvenaud 3y ago
Here's one of my concrete worries: At some point, humans are going to be outcompeted by AI at basically every important job. At that point, how are we going to maintain political power in the long run? Humanity is going to be like an out-of-touch old person on the internet - we'll either have to delegate everything important (which is risky), or eventually get scammed or extorted out of all our resources and influence. I agree we don't necessarily know the details of how to build such a system, but am pretty sure we will be able to eventually.
- Infernal 3y ago
  “Humans are going to be outcompeted by AI” is the concrete bit as best I can tell. Historically humans are not outcompeted by new tools, but humans using old tools are outcompeted by humans using new tools. It’s not “all humans vs the new tool”, as the tool has no agency. If you meant “humans using old tools get outcompeted by humans using AI”, then I agree but I don’t see it any differently than previous efficiency improvements with new tooling. If you meant ”all humans get outcompeted by AI”, then I think you have a lot of work to do to demonstrate how AI is going to replace humans in “every important job”, and not simply replace some of the tools in the humans’ toolbox.
  - duvenaud 3y ago
    I see what you mean - for a while, the best chess was played by humans aided by chess engines. But that era has passed, and now having a human trying to aid the best chess engines just results in worse chess (or the same, if the human does nothing). But whether there a few humans in the loop doesn't change the likely outcomes, if their actions are constrained by competition. What abilities do humans have that AIs will never have?
    - jvanderbot 3y ago
      Well, in this case, we have the ability to invent chess (a game that will be popular for centuries), invent computers, and invent chess tournaments, and invent programs that can solve chess, and invent all the supporting agriculture, power, telco, silicon boards, etc that allow someone to run a program to beat a person at chess. Then we have bodies to accomplish everything on top of it. The "idea" isn't enough. We have to "do" it. If you take a chess playing robot as the peak of the pyramid, there are probably millions of people and trillions of dollars toiling away to support it. Imagine all the power lines, sewage, HVAC systems, etc that humans crawl around in to keep working. And really, are we "beaten" at chess, or are we now "unbeatable" at chess. If an alien warship came and said "we will destroy earth if you lose at chess", wouldn't we throw our algorithms at it? I say we're now unbeatable at chess.
SpaceManNabs 3y ago
> They can’t point to an existing system that poses existential risk, because it doesn’t exist. There are judges using automated decision systems to excuse away decisions that send people back to jail for recidivism purposes. These systems are just enforcing societal biases at scale. It is clear that we are ready to acquiesce control to AI systems without much care to any extra ethical considerations.
- brookst 3y ago
  I think you just said that the problem is systemic in our judicial system, and that AI has nothing to do with it.
  - SpaceManNabs 3y ago
    AI is the tool that provides "objective to truth" that enables such behavior. It is definite unique in its depth, scale, and implications.
- NathanFulton 3y ago
  Absolutely. These are the types of pragmatic, real problems we should be focusing on instead of the "risk of extinction from AI". (The statement at hand reads "mitigating the risk of extinction from AI should be a global priority alongside other societal-scale risks such as pandemics and nuclear war.")
  - holmesworcester 3y ago
    Einstein's letter to Roosevelt was written before the atomic bomb existed. There's a point where people see a path, and they gain confidence in their intuition from the fact that other members of their field also see a path. Einstein's letter said 'almost certain' and 'in the immediate future' but it makes sense to sound the alarm about AI earlier, both given what we know about the rate of progress of general purpose technologies and given that the AI risk, if real, is greater than the risk Einstein envisioned (total extermination as opposed to military defeat to a mass murderer.)
deltaninenine 3y ago
It's arrived at through induction. Induction is logic involving probability. Probabilistic logic and predictions of the future are valid logic that has demonstrably worked in other situations so if such logic has a level of validity then induction is a candidate for refutation. So we know a human of human intelligence can take over a humans job and endanger other humans. AI has been steadily increasing in intelligence. The latest leap with LLMs crossed certain boundaries of creativity and natural language. By induction the trendline points to machines approaching human intelligence. Also by induction if humans of human intelligence can endanger humanity then a machine of human intelligence should do the same. Now. All of this induction is something you and everyone already knows. We know that this level of progress increases the inductive probabilities of this speculation playing out. None of us needs to be explained any of this logic as we are all well aware of it. What's going on is that humans like to speculate on a future that's more convenient for them. Science shows human psychology is more optimistic then realistic. Hence why so many people are in denial.
ericb 3y ago
What? ChatGPT 4 can already pass the bar exam and is fluent in every language. It is super intelligent. Today. No human can do that, the system is here, and so is an architecture. As for the existential risk, assume nothing other than evil humans will use it to do evil human stuff. Most technology iteratively gets better, so there's no big leaps of imagination required to imagine that we're equipping bad humans with super-human, super-intelligent assistants.
- ilaksh 3y ago
  Right. And it would be a complete break from the history of computing if human-level GPT doesn't get 100+ times faster in the next few years. Certainly within five years. All it takes is for someone to give an AI that thinks 100 times faster than humans an overly broad goal. Then the only way to counteract it is with another AI with overly broad goals. And you can't tell it to stop and wait for humans to check it's decisions, because while it is waiting for you to come back from your lunch break to try to figure out what it is asking, the competitor's AI did the equivalent of a week of work. So then even if at some level people are "in control" of the AIs, practically speaking they are spectators. And there is no way you will be able to prevent all people from creating fully autonomous lifelike AI with its own goals and instincts. Combine that with hyperspeed and you are truly at it's mercy.
  - mrtranscendence 3y ago
    Computational power does not grow at the rate of over 100x within the span of “a few years”. If that were the case we’d have vastly more powerful kit by now.
    - ilaksh 3y ago
      I didn't quite say that. The efficiency of this very specific application absolutely can and almost certainly will increase by more than one order of magnitude within four years. It's got a massive new investment and research focus, is a very specific application, and room for improvement in AI model, software, and hardware. Even if we have to "cheat" to get to 100 times performance in less than five years the effect will be the same. For example, there might be a way to accelerate something like the Tree of Thoughts in hardware. So if the hardware can't actually speed up by that much, the effectiveness of the system still has increased greatly.
a257 3y ago
> They can’t point to an existing system that poses existential risk, because it doesn’t exist. They can’t point to a clear architecture for such a system, because we don’t know how to build it. Inductive reasoning is in favor of their argument being possible. From observing nature, we know that a variety of intelligent species can emerge from physical phenomenon alone. Historically, the dominance of one intelligent species has contributed to the extinction of others. Given this, it can be said that AI might cause our extinction.
Symmetry 3y ago
The idea is that if you build a system that poses an existential risk you want to be reasonably sure it's safe before you turn it on, not afterwards. It would have been irresponsible for the scientists at Los Alamost to do the math on whether an atomic explosion would create a sustained fusion reaction in the atmosphere until after their first test, for example. I don't think it's possible for a large language model, operating in a conventional feed forward way, to really pose a significant danger. But I do think it's hard to say exactly what advances could lead to a dangerous intelligence and with the current state of the art it looks to me at least like we might very well be only one breakthrough away from that. Hence the calls for prudence. The scientists creating the atomic bomb knew a lot more about what they were doing than we do. Their computations sometimes gave the wrong result, see Castle Bravo, but had a good framework for understanding everything that was happening. We're more like cavemen who've learned to reliably make fire but still don't understand it. Why can current versions of GPT reliably add large numbers together when previous versions couldn't? We're still a very long way away from being able to answer questions like that.
lxnn 3y ago
You can't take an empirical approach to existential risk as you don't get the opportunity to learn from your mistakes. You have to prospectively reason about it and plan for it.

notahacker 3y ago

Seems apt the term "Bulverism" comes from CS Lewis, since he was also positing that an unseen, unfalsifiable entity would grant eternal reward to people that listened to him and eternal damnation to those that didn't...

alasdair_ 3y ago
The irony of critiquing Bulverism as a concept, not by attacking the idea itself, but instead by assuming it is wrong and attacking the character of the author, is staggeringly hilarious.
- notahacker 3y ago
  I'm replying in agreement with someone who already pointed out the obvious flaw in labelling any questioning of the inspirations or motivations of AI researchers as "Bulverism": none of the stuff they're saying is actually a claim that can be falsified in the first place! I'm unconvinced by the position that the only valid means of casting doubt on a claim is through forensic examination of hard data that may be inaccessible to the interlocutor (like most people's bank accounts...), but whether that is or isn't a generally good approach is irrelevant here as we're talking about claims about courses of action to avoid hypothetical threats. I just noted it was a particularly useful rhetorical flourish when advocating acting on beliefs which aren't readily falsifiable, something CS Lewis was extremely proud of doing and certainly wouldn't have considered a character flaw! Ironically, your reply also failed to falsify anything I said and instead critiqued my assumed motivations for making the comment. It's Bulverism all the way down!
  - deltaninenine 3y ago
    Logical Induction has been successful in predicting future events.
    - notahacker 3y ago
      Sometimes it makes good predictions, sometimes bad. But "advances in AI might lead to Armageddon" isn't the only conclusion induction can reach. Induction can also lead to people concluding certain arguments seem to a mashup of traditional millennialist "end times" preoccupations with the sort of sci-fi they grew up with, or that this looks a lot like a movement towards regulatory capture. Ultimately any (possibly even all) these inferences from past trends and recent actions can be correct, but none of them are falsifiable. So I don't think it's a good idea to insist that people should be falsifying the idea that AI is a risk before we start questioning whether the behaviour of some of the entities on the list says more about their motivations than their words.

duvenaud 3y ago

Here's one of my concrete worries: At some point, humans are going to be outcompeted by AI at basically every important job. At that point, how are we going to maintain political power in the long run? Humanity is going to be like an out-of-touch old person on the internet - we'll either have to delegate everything important (which is risky), or eventually get scammed or extorted out of all our resources and influence. I agree we don't necessarily know the details of how to build such a system, but am pretty sure we will be able to eventually.

Infernal 3y ago
“Humans are going to be outcompeted by AI” is the concrete bit as best I can tell. Historically humans are not outcompeted by new tools, but humans using old tools are outcompeted by humans using new tools. It’s not “all humans vs the new tool”, as the tool has no agency. If you meant “humans using old tools get outcompeted by humans using AI”, then I agree but I don’t see it any differently than previous efficiency improvements with new tooling. If you meant ”all humans get outcompeted by AI”, then I think you have a lot of work to do to demonstrate how AI is going to replace humans in “every important job”, and not simply replace some of the tools in the humans’ toolbox.
- duvenaud 3y ago
  I see what you mean - for a while, the best chess was played by humans aided by chess engines. But that era has passed, and now having a human trying to aid the best chess engines just results in worse chess (or the same, if the human does nothing). But whether there a few humans in the loop doesn't change the likely outcomes, if their actions are constrained by competition. What abilities do humans have that AIs will never have?
  - jvanderbot 3y ago
    Well, in this case, we have the ability to invent chess (a game that will be popular for centuries), invent computers, and invent chess tournaments, and invent programs that can solve chess, and invent all the supporting agriculture, power, telco, silicon boards, etc that allow someone to run a program to beat a person at chess. Then we have bodies to accomplish everything on top of it. The "idea" isn't enough. We have to "do" it. If you take a chess playing robot as the peak of the pyramid, there are probably millions of people and trillions of dollars toiling away to support it. Imagine all the power lines, sewage, HVAC systems, etc that humans crawl around in to keep working. And really, are we "beaten" at chess, or are we now "unbeatable" at chess. If an alien warship came and said "we will destroy earth if you lose at chess", wouldn't we throw our algorithms at it? I say we're now unbeatable at chess.

SpaceManNabs 3y ago

> They can’t point to an existing system that poses existential risk, because it doesn’t exist. There are judges using automated decision systems to excuse away decisions that send people back to jail for recidivism purposes. These systems are just enforcing societal biases at scale. It is clear that we are ready to acquiesce control to AI systems without much care to any extra ethical considerations.

brookst 3y ago
I think you just said that the problem is systemic in our judicial system, and that AI has nothing to do with it.
- SpaceManNabs 3y ago
  AI is the tool that provides "objective to truth" that enables such behavior. It is definite unique in its depth, scale, and implications.
NathanFulton 3y ago
Absolutely. These are the types of pragmatic, real problems we should be focusing on instead of the "risk of extinction from AI". (The statement at hand reads "mitigating the risk of extinction from AI should be a global priority alongside other societal-scale risks such as pandemics and nuclear war.")
- holmesworcester 3y ago
  Einstein's letter to Roosevelt was written before the atomic bomb existed. There's a point where people see a path, and they gain confidence in their intuition from the fact that other members of their field also see a path. Einstein's letter said 'almost certain' and 'in the immediate future' but it makes sense to sound the alarm about AI earlier, both given what we know about the rate of progress of general purpose technologies and given that the AI risk, if real, is greater than the risk Einstein envisioned (total extermination as opposed to military defeat to a mass murderer.)

deltaninenine 3y ago

It's arrived at through induction. Induction is logic involving probability. Probabilistic logic and predictions of the future are valid logic that has demonstrably worked in other situations so if such logic has a level of validity then induction is a candidate for refutation. So we know a human of human intelligence can take over a humans job and endanger other humans. AI has been steadily increasing in intelligence. The latest leap with LLMs crossed certain boundaries of creativity and natural language. By induction the trendline points to machines approaching human intelligence. Also by induction if humans of human intelligence can endanger humanity then a machine of human intelligence should do the same. Now. All of this induction is something you and everyone already knows. We know that this level of progress increases the inductive probabilities of this speculation playing out. None of us needs to be explained any of this logic as we are all well aware of it. What's going on is that humans like to speculate on a future that's more convenient for them. Science shows human psychology is more optimistic then realistic. Hence why so many people are in denial.

ericb 3y ago

What? ChatGPT 4 can already pass the bar exam and is fluent in every language. It is super intelligent. Today. No human can do that, the system is here, and so is an architecture. As for the existential risk, assume nothing other than evil humans will use it to do evil human stuff. Most technology iteratively gets better, so there's no big leaps of imagination required to imagine that we're equipping bad humans with super-human, super-intelligent assistants.

ilaksh 3y ago
Right. And it would be a complete break from the history of computing if human-level GPT doesn't get 100+ times faster in the next few years. Certainly within five years. All it takes is for someone to give an AI that thinks 100 times faster than humans an overly broad goal. Then the only way to counteract it is with another AI with overly broad goals. And you can't tell it to stop and wait for humans to check it's decisions, because while it is waiting for you to come back from your lunch break to try to figure out what it is asking, the competitor's AI did the equivalent of a week of work. So then even if at some level people are "in control" of the AIs, practically speaking they are spectators. And there is no way you will be able to prevent all people from creating fully autonomous lifelike AI with its own goals and instincts. Combine that with hyperspeed and you are truly at it's mercy.
- mrtranscendence 3y ago
  Computational power does not grow at the rate of over 100x within the span of “a few years”. If that were the case we’d have vastly more powerful kit by now.
  - ilaksh 3y ago
    I didn't quite say that. The efficiency of this very specific application absolutely can and almost certainly will increase by more than one order of magnitude within four years. It's got a massive new investment and research focus, is a very specific application, and room for improvement in AI model, software, and hardware. Even if we have to "cheat" to get to 100 times performance in less than five years the effect will be the same. For example, there might be a way to accelerate something like the Tree of Thoughts in hardware. So if the hardware can't actually speed up by that much, the effectiveness of the system still has increased greatly.

a257 3y ago

> They can’t point to an existing system that poses existential risk, because it doesn’t exist. They can’t point to a clear architecture for such a system, because we don’t know how to build it. Inductive reasoning is in favor of their argument being possible. From observing nature, we know that a variety of intelligent species can emerge from physical phenomenon alone. Historically, the dominance of one intelligent species has contributed to the extinction of others. Given this, it can be said that AI might cause our extinction.

Symmetry 3y ago

The idea is that if you build a system that poses an existential risk you want to be reasonably sure it's safe before you turn it on, not afterwards. It would have been irresponsible for the scientists at Los Alamost to do the math on whether an atomic explosion would create a sustained fusion reaction in the atmosphere until after their first test, for example. I don't think it's possible for a large language model, operating in a conventional feed forward way, to really pose a significant danger. But I do think it's hard to say exactly what advances could lead to a dangerous intelligence and with the current state of the art it looks to me at least like we might very well be only one breakthrough away from that. Hence the calls for prudence. The scientists creating the atomic bomb knew a lot more about what they were doing than we do. Their computations sometimes gave the wrong result, see Castle Bravo, but had a good framework for understanding everything that was happening. We're more like cavemen who've learned to reliably make fire but still don't understand it. Why can current versions of GPT reliably add large numbers together when previous versions couldn't? We're still a very long way away from being able to answer questions like that.

lxnn 3y ago

You can't take an empirical approach to existential risk as you don't get the opportunity to learn from your mistakes. You have to prospectively reason about it and plan for it.