6 ms·

We can certainly calculate the probability of an Earth-killing asteroid and it's quite low. In case of AI, it's unclear whether we need any additional scientif

We can certainly calculate the probability of an Earth-killing asteroid and it's quite low. In case of AI, it's unclear whether we need any additional scientific advances at all beyond scaling existing methods. Even if some additional advances are required, the probability that they will happen in the coming decades is at least in tens of percent.

AnimalMuppet 3y ago
> Even if some additional advances are required, the probability that they will happen in the coming decades is at least in tens of percent. You don't have any idea whether additional advances are needed. You don't have any idea what advances are needed. You don't have any idea whether those advances are even possible. But you're confident that they will happen with probability > 10%? You're making very confident assertions, but you have no factual basis for doing so. (Neither does anyone else. We don't know - we're all guessing.)
- comp_throw7 3y ago
  People are actively trying to make it happen! This is literally the mission statement of DeepMind, OpenAI, etc. Obviously things that nobody is trying to make happen are not likely to happen, and sufficiently difficult things that people are trying to make happen might take a while (e.g. fusion). But AGI is a thing that people are trying to make happen, and are making progress on, and basically nobody predicted in advance the progress that has been made so far.
  - AnimalMuppet 3y ago
    "People are actively trying to make it happen" != "people trying can make it happen". People tried to make alchemy work, too - and they achieved some results along the way! But the foundation was wrong, and therefore all the work was not able to lead to the desired goal. Is something GPT-like the right foundation for AGI? That is very far from proven.
    - comp_throw7 3y ago
      I make no claims about any specific architectures, only that human intelligence isn't anything special and so far we've done a pretty good job at blowing past it in a bunch of domains.
- eterevsky 3y ago
  The fact that we don't know some details doesn't mean that we can't make good predictions about _probabilities_. Yes, because there's plenty of datapoints to make prediction by extrapolation. Just look at the progress between GPT-1, GPT-2, GPT-3 and GPT-4 and extrapolate to GPT-5, GPT-6 etc. There could be some roadblocks that would prevent further progress, but the default prediction should be that the trend continues. Metaculus has had predictions for various AI milestones for years and has been consistently too conservative for resolved questions. It now predicts 50% of the invention of full AGI by 2032 which by extension might also be too conservative.
  - danaris 3y ago
    What is the probability that we will achieve FTL travel within the next 100 years? What is the probability that we will be visited openly by intelligent, friendly aliens within the next 50 years? What is the probability that we will develop a device that allows us to seamlessly and accurately converse with dogs and cats within the next 10 years? We can assign probabilities to any of these, but they will be based purely on speculation, because none of these things are known to be possible. Note, not "I can make an argument that they should be possible"; known, with a solid scientific basis and an understanding, at least among experts, of the basic steps needed to get there. On the other hand, "the probability that we will achieve commercially-viable nuclear fusion within 30 years" is a different kind of calculation to make: we understand the physics, we've got a pretty damn good idea of what advances we need to make it possible; the main problem is getting the money, time, and manpower (aka, money, money, and money) to create the materials and build the prototypes. AGI falls into the former category, not the latter.
baxtr 3y ago
We can calculate the probability of earth killing asteroids because it happened “frequently” enough to be calculated. Same goes for pandemics, wars, and super volcano eruptions. By this method there is no way to calculate the risk of an AI extinction simply because it never happened before.
- fsflover 3y ago
  https://news.ycombinator.com/item?id=36147546 https://news.ycombinator.com/item?id=36147546
  - baxtr 3y ago
    Thanks. Yes that's exactly my point. Because there is some data on GPT, we can extrapolate its development. But please read my comment again: There is no way that by this method you can deduce a probability for the extinction by AI. It never happened before. It is really that simple. Same goes for a nuclear war scenario.

AnimalMuppet 3y ago

> Even if some additional advances are required, the probability that they will happen in the coming decades is at least in tens of percent. You don't have any idea whether additional advances are needed. You don't have any idea what advances are needed. You don't have any idea whether those advances are even possible. But you're confident that they will happen with probability > 10%? You're making very confident assertions, but you have no factual basis for doing so. (Neither does anyone else. We don't know - we're all guessing.)

comp_throw7 3y ago
People are actively trying to make it happen! This is literally the mission statement of DeepMind, OpenAI, etc. Obviously things that nobody is trying to make happen are not likely to happen, and sufficiently difficult things that people are trying to make happen might take a while (e.g. fusion). But AGI is a thing that people are trying to make happen, and are making progress on, and basically nobody predicted in advance the progress that has been made so far.
- AnimalMuppet 3y ago
  "People are actively trying to make it happen" != "people trying can make it happen". People tried to make alchemy work, too - and they achieved some results along the way! But the foundation was wrong, and therefore all the work was not able to lead to the desired goal. Is something GPT-like the right foundation for AGI? That is very far from proven.
  - comp_throw7 3y ago
    I make no claims about any specific architectures, only that human intelligence isn't anything special and so far we've done a pretty good job at blowing past it in a bunch of domains.
eterevsky 3y ago
The fact that we don't know some details doesn't mean that we can't make good predictions about _probabilities_. Yes, because there's plenty of datapoints to make prediction by extrapolation. Just look at the progress between GPT-1, GPT-2, GPT-3 and GPT-4 and extrapolate to GPT-5, GPT-6 etc. There could be some roadblocks that would prevent further progress, but the default prediction should be that the trend continues. Metaculus has had predictions for various AI milestones for years and has been consistently too conservative for resolved questions. It now predicts 50% of the invention of full AGI by 2032 which by extension might also be too conservative.
- danaris 3y ago
  What is the probability that we will achieve FTL travel within the next 100 years? What is the probability that we will be visited openly by intelligent, friendly aliens within the next 50 years? What is the probability that we will develop a device that allows us to seamlessly and accurately converse with dogs and cats within the next 10 years? We can assign probabilities to any of these, but they will be based purely on speculation, because none of these things are known to be possible. Note, not "I can make an argument that they should be possible"; known, with a solid scientific basis and an understanding, at least among experts, of the basic steps needed to get there. On the other hand, "the probability that we will achieve commercially-viable nuclear fusion within 30 years" is a different kind of calculation to make: we understand the physics, we've got a pretty damn good idea of what advances we need to make it possible; the main problem is getting the money, time, and manpower (aka, money, money, and money) to create the materials and build the prototypes. AGI falls into the former category, not the latter.

baxtr 3y ago

We can calculate the probability of earth killing asteroids because it happened “frequently” enough to be calculated. Same goes for pandemics, wars, and super volcano eruptions. By this method there is no way to calculate the risk of an AI extinction simply because it never happened before.

fsflover 3y ago
https://news.ycombinator.com/item?id=36147546 https://news.ycombinator.com/item?id=36147546
- baxtr 3y ago
  Thanks. Yes that's exactly my point. Because there is some data on GPT, we can extrapolate its development. But please read my comment again: There is no way that by this method you can deduce a probability for the extinction by AI. It never happened before. It is really that simple. Same goes for a nuclear war scenario.