7 ms·

That's Pascal's Wager. What makes you think that OpenAI is doing anything that has any meaningful impact on AGI? We have no idea what AGI looks like.

by rspeer 10y ago

acgourley 10y ago
You've brought up Pascal's Wager twice in this thread, as if it really can end an argument by itself. Just because you don't like what high impact / low probability EV calculations do in Theology doesn't mean you can wave away high impact / low probability EV calculations in other domains.
- Avshalom 10y ago
  >>high impact / low probability EV calculations in other domains Yeah but there isn't a calculation of the risk/probability of AGI there's just the appeal to the idea.
  - nbouscal 10y ago
    That's… not at all true? There have been numerous thorough attempts at coming up with as reasonable an estimate as possible, involving in-depth conversations with most of the experts in the field. Maybe it's not strictly a "calculation", but it's a hell of a lot more than just an appeal to the idea of AGI.
    - joshuamorton 10y ago
      Except that there's no such thing as the field of "AGI" (or if there is, its a subfield of philosophy). Asking modern ML and Deep Learning researchers about their thoughts on AGI is like asking the wright brothers about a mars mission. Sure, they're the closest thing we have to "experts", but there's not just one, but likely 5 or 10 more field, if not world-changing leaps we need to make before the technologies we have will resemble anything like AGI. And that's ignoring the people who ascribe diety-like powers to some potential AGI. Air gap the computer and control the inputs and outputs. We can formally prove what a system is capable of. That fixes the problem.
      - nbouscal 10y ago
        Your opinion about how far off AGI is is a totally reasonable one, and some AI experts agree with it. Other AI experts disagree, and think that AGI is basically tractable with current research methods plus a breakthrough or two. Yes, there's some additional uncertainty about whether you're even asking the right people. But you can take account of that uncertainty, both by widening your error bars and by asking people from other fields as well (including philosophers, e.g. Bostrom). What you can't do is just throw up your hands and say "it's unknowable". This is all beside the original point, which is that these arguments are much more rigorously grounded than just a wave in the direction of AGI. Could they be better? Sure, and there are a lot of people who'd be interested in seeing some better estimates. But for now, they're the best we have, and they're a completely reasonable thing to base decisions on. > We can formally prove what a system is capable of. That fixes the problem. That's exactly what some of the people working on this problem are trying to do, but it's a hell of a lot harder than you make it sound. Formal methods have come a really long way, but they're not even remotely close to being able to prove that an AGI system is safe (yet).
        joshuamorton 10y ago
        You don't need to prove anything about an AGI system if you can prove things about its I/O capabilities. I recognize that proving things about machine learning models is very hard, I tried to do some research in that area and got practically no where. And I think you and I have very different definitions of Rigorous. Like I said, unless you'd take Wilbur Wright's thoughts on a mars mission seriously, I don't think you should give much thought to Geoff Hinton's thoughts about when we'll get AGI, and I say that having enormous respect for him and his achievements. (and using him as a simple example) Its pseudoscience. We're notoriously bad about predicting the future. I don't see any reason to trust people going on about the dangers of AGI any more than the futurologists of my parents generation, who predicted flying cars and interstellar travel, but missed smartphones.
        schoen 10y ago
        > You don't need to prove anything about an AGI system if you can prove things about its I/O capabilities. But then you can't benefit from it, or you can only benefit from it except in narrowly predefined ways.
    - BaronSamedi 10y ago
      Experts in the field of AI have a long history of being wrong. "AI is 20 years" away has become the standard joke in the field. There is no reason to grant expert opinion on this topic any weight.
      - nbouscal 10y ago
        What's your suggested alternative? It's really easy to criticize, but a lot harder to do any better. If you aren't suggesting any other methods, you're implicitly saying we should just ignore the problem. I really don't think that will end well.
        argonaut 10y ago
        You admit that you don't know what the actual probabilities are. Or you admit you have an extremely low confidence in your estimate of the probabilities.
        nbouscal 10y ago
        Have you read any of the estimates? Basically every expert gives a huge confidence interval, as do most of the people actively working on AI risk.
        argonaut 10y ago
        Saying you are 95% confident AGI will come between 50-100 years (a "huge" confidence interval), is still far too confident.
        nbouscal 10y ago
        I don't recall seeing 95% confidence intervals that narrow. Even the 90% confidence intervals I see are usually like 5-150 or something. (Also it would be super weird to me if the lower bound of someone's 95%CI was 50.)
        rspeer 10y ago
  - deleted 10y ago
    [deleted]
- robbensinger 10y ago
  I agree that "that's Pascal's wager!" isn't a reasonable response to someone arguing that, say, a 1% or 10% extinction risk is worth taking seriously. If you think the probability is infinitesimally small but that we should work on it just in case, then that's more like Pascal's wager. I think the whole discussion thread has a false premise, though. The main argument for working on AGI accident risk is that it's high-probability, not that it's 'low-probability but not super low.' Roughly: it would be surprising if we didn't reach AGI this century; it would be surprising if AGI exhibited roughly human levels of real-world capability (in spite of potential hardware and software improvements over the brain) rather than shooting past human-par performance; and it would be surprising if it were easy to get robustly good outcomes out of AI systems much smarter than humans, operating in environments too complex for it to be feasible to specify desirable v. undesirable properties of outcomes. "It's really difficult to make reliable predictions about when and how people will make conceptual progress on a tough technological challenge, and there's a lot of uncertainty" doesn't imply "the probability of catastrophic accidents is <10%" or even "the probability of catastrophic accidents is <50%".
bermanoid 10y ago
Many people's estimate of the probability-weighted impact OpenAI will have is much greater than anything that I'd ever call a Pascal's Wager. You may disagree, but a lot of us actually have our expectations somewhat quantified and grounded, which is very different from the argument as applied to religion. Personally, I think there's a greater than 10% chance that we'll see AGI that definitively surpasses human ability within the next 50-100 years (growing a lot higher as we get near/past 100). And given what's coming out this early on with minimal funding, I expect that the work that OpenAI does in the near future will have at least a 10% chance of strongly influencing the direction of that AGI work during the critical turning points. A 1% or more chance of their work mattering a lot is plain old betting-on-a-black-swan territory, not Pascal's Wager.
- pjscott 10y ago
  Well put. The Open Philanthropy folks take a broadly similar view of the probabilities, both of when strong AI is likely to arrive and of how much positive influence per dollar they can have on how things turn out: http://www.openphilanthropy.org/blog/potential-risks-advanced-artificial-intelligence-philanthropic-opportunity http://www.openphilanthropy.org/blog/potential-risks-advance...

That's Pascal's Wager. What makes you think that OpenAI is doing anything that has any meaningful impact on AGI? We have no idea what AGI looks like.

by rspeer 10y ago

That's Pascal's Wager. What makes you think that OpenAI is doing anything that has any meaningful impact on AGI? We have no idea what AGI looks like.

acgourley 10y ago

You've brought up Pascal's Wager twice in this thread, as if it really can end an argument by itself. Just because you don't like what high impact / low probability EV calculations do in Theology doesn't mean you can wave away high impact / low probability EV calculations in other domains.

Avshalom 10y ago
>>high impact / low probability EV calculations in other domains Yeah but there isn't a calculation of the risk/probability of AGI there's just the appeal to the idea.
- nbouscal 10y ago
  That's… not at all true? There have been numerous thorough attempts at coming up with as reasonable an estimate as possible, involving in-depth conversations with most of the experts in the field. Maybe it's not strictly a "calculation", but it's a hell of a lot more than just an appeal to the idea of AGI.
  - joshuamorton 10y ago
    Except that there's no such thing as the field of "AGI" (or if there is, its a subfield of philosophy). Asking modern ML and Deep Learning researchers about their thoughts on AGI is like asking the wright brothers about a mars mission. Sure, they're the closest thing we have to "experts", but there's not just one, but likely 5 or 10 more field, if not world-changing leaps we need to make before the technologies we have will resemble anything like AGI. And that's ignoring the people who ascribe diety-like powers to some potential AGI. Air gap the computer and control the inputs and outputs. We can formally prove what a system is capable of. That fixes the problem.
    - nbouscal 10y ago
      Your opinion about how far off AGI is is a totally reasonable one, and some AI experts agree with it. Other AI experts disagree, and think that AGI is basically tractable with current research methods plus a breakthrough or two. Yes, there's some additional uncertainty about whether you're even asking the right people. But you can take account of that uncertainty, both by widening your error bars and by asking people from other fields as well (including philosophers, e.g. Bostrom). What you can't do is just throw up your hands and say "it's unknowable". This is all beside the original point, which is that these arguments are much more rigorously grounded than just a wave in the direction of AGI. Could they be better? Sure, and there are a lot of people who'd be interested in seeing some better estimates. But for now, they're the best we have, and they're a completely reasonable thing to base decisions on. > We can formally prove what a system is capable of. That fixes the problem. That's exactly what some of the people working on this problem are trying to do, but it's a hell of a lot harder than you make it sound. Formal methods have come a really long way, but they're not even remotely close to being able to prove that an AGI system is safe (yet).
      - joshuamorton 10y ago
        You don't need to prove anything about an AGI system if you can prove things about its I/O capabilities. I recognize that proving things about machine learning models is very hard, I tried to do some research in that area and got practically no where. And I think you and I have very different definitions of Rigorous. Like I said, unless you'd take Wilbur Wright's thoughts on a mars mission seriously, I don't think you should give much thought to Geoff Hinton's thoughts about when we'll get AGI, and I say that having enormous respect for him and his achievements. (and using him as a simple example) Its pseudoscience. We're notoriously bad about predicting the future. I don't see any reason to trust people going on about the dangers of AGI any more than the futurologists of my parents generation, who predicted flying cars and interstellar travel, but missed smartphones.
        schoen 10y ago
        > You don't need to prove anything about an AGI system if you can prove things about its I/O capabilities. But then you can't benefit from it, or you can only benefit from it except in narrowly predefined ways.
  - BaronSamedi 10y ago
    Experts in the field of AI have a long history of being wrong. "AI is 20 years" away has become the standard joke in the field. There is no reason to grant expert opinion on this topic any weight.
    - nbouscal 10y ago
      What's your suggested alternative? It's really easy to criticize, but a lot harder to do any better. If you aren't suggesting any other methods, you're implicitly saying we should just ignore the problem. I really don't think that will end well.
      - argonaut 10y ago
        You admit that you don't know what the actual probabilities are. Or you admit you have an extremely low confidence in your estimate of the probabilities.
        nbouscal 10y ago
        Have you read any of the estimates? Basically every expert gives a huge confidence interval, as do most of the people actively working on AI risk.
        argonaut 10y ago
        Saying you are 95% confident AGI will come between 50-100 years (a "huge" confidence interval), is still far too confident.
        nbouscal 10y ago
        I don't recall seeing 95% confidence intervals that narrow. Even the 90% confidence intervals I see are usually like 5-150 or something. (Also it would be super weird to me if the lower bound of someone's 95%CI was 50.)
      - rspeer 10y ago
- deleted 10y ago
  [deleted]
robbensinger 10y ago
I agree that "that's Pascal's wager!" isn't a reasonable response to someone arguing that, say, a 1% or 10% extinction risk is worth taking seriously. If you think the probability is infinitesimally small but that we should work on it just in case, then that's more like Pascal's wager. I think the whole discussion thread has a false premise, though. The main argument for working on AGI accident risk is that it's high-probability, not that it's 'low-probability but not super low.' Roughly: it would be surprising if we didn't reach AGI this century; it would be surprising if AGI exhibited roughly human levels of real-world capability (in spite of potential hardware and software improvements over the brain) rather than shooting past human-par performance; and it would be surprising if it were easy to get robustly good outcomes out of AI systems much smarter than humans, operating in environments too complex for it to be feasible to specify desirable v. undesirable properties of outcomes. "It's really difficult to make reliable predictions about when and how people will make conceptual progress on a tough technological challenge, and there's a lot of uncertainty" doesn't imply "the probability of catastrophic accidents is <10%" or even "the probability of catastrophic accidents is <50%".

bermanoid 10y ago

Many people's estimate of the probability-weighted impact OpenAI will have is much greater than anything that I'd ever call a Pascal's Wager. You may disagree, but a lot of us actually have our expectations somewhat quantified and grounded, which is very different from the argument as applied to religion. Personally, I think there's a greater than 10% chance that we'll see AGI that definitively surpasses human ability within the next 50-100 years (growing a lot higher as we get near/past 100). And given what's coming out this early on with minimal funding, I expect that the work that OpenAI does in the near future will have at least a 10% chance of strongly influencing the direction of that AGI work during the critical turning points. A 1% or more chance of their work mattering a lot is plain old betting-on-a-black-swan territory, not Pascal's Wager.

pjscott 10y ago
Well put. The Open Philanthropy folks take a broadly similar view of the probabilities, both of when strong AI is likely to arrive and of how much positive influence per dollar they can have on how things turn out: http://www.openphilanthropy.org/blog/potential-risks-advanced-artificial-intelligence-philanthropic-opportunity http://www.openphilanthropy.org/blog/potential-risks-advance...