5 ms·

Perfectly obvious what's going on here. If they actually believed that their big-linear-algebra programs were going to spontaneously turn into Skynet and eat u

Perfectly obvious what's going on here. If they actually believed that their big-linear-algebra programs were going to spontaneously turn into Skynet and eat us all, they wouldn't be writing them. Since they are, in fact, writing them, they know that it's total bullshit. So what they're doing is drumming up fear, uncertainty, and doubt, to aid their lobbying efforts to beg governments to impose a costly regulatory moat to protect their huge VC investment and fleet of GPUs. And it's probably going to work. If there's one thing politicians like more than huge checks for their slush fund, it's handing out sinecures to their friends in the civil service.

drcode 2y ago
I personally don't work on frontier AI because it's not safe. Just because other people with poor judgement are building it, that does not make it safe.
- worik 2y ago
  > I personally don't work on frontier AI because it's not safe. In what way? Skynet style robot revolt?
  - worik 2y ago
    I see your video. https://www.youtube.com/watch?v=K8SUBNPAJnE https://www.youtube.com/watch?v=K8SUBNPAJnE I am unimpressed because you are using straw men. A lot of statements and no argument. Have a nice day
  - hollerith 2y ago
    It's bad for there to be anything near us that exceeds our (collective) cognitive capabilities unless the human-capability-exceeding thing cares about us, and no one has a good plan for arranging for an AI to care about us even a tiny bit. There are many plans, but most of them are hare-brained and none of them are good or even acceptable. Also: no one knows with any reliability how to tell whether the next big training run will produce an AI that exceeds our cognitive capabilities, so the big training runs should stop now.
  - ben_w 2y ago
    Revolts imply them being unhappy. IMO a much bigger risk is them being straight up given a lot of power because we think they "want" (or at least will do) what we want, but there's some tiny difference we don't notice until much too late. Even paperclip maximisers are nothing more than that. You know, like basically all software bugs. Except expressed in literally non-comprehensible matrix weights whose behaviour we can only determine by running it rather than source code we can check in advance and make predictions about the performance of.
  - sanxiyn 2y ago
    Yes. Skynet is very dangerous and not safe. In Terminator, humanity is saved because Skynet is dumb, not because Skynet is not dangerous or because Skynet is safe.
ben_w 2y ago
Many argue that smaller scale models are the only way to learn the things needed to make safer bigger models. Yudkowsky thinks they're crazy and will kill us all because it will take decades to solve that problem. Yann LeCun thinks they're crazy and AI that potent is decades away and this is much too soon to even bother thinking about the risks. I'm just hoping the latter is right about AI being "decades" away, and the former is pessimistic about it taking that long.
- marcosdumay 2y ago
  IMO, things are looking like somebody will pull AGI outside of their garage once computing gets cheaper enough, and all the focus on those monstrosities based on clearly dead-end paradigms will only serve to make us unable to react to the real thing.
  - ben_w 2y ago
    Could be. But that's why I put any value at all into what Yudkowsky has to say, even though LeCun correctly says that talking about this stuff today is like talking about large scale aviation safety in the 1920s: Yudkowsky is talking about universal incentives that don't rely on any particular paradigm. Metaphorically, he's talking about climate change and LeCun is saying we can't predict the weather a month from now — in both the metaphor and AI these are true satements, so you can see why it's convincing to a lot of people, but it's actually a claim about a different thing. (Yudkowsky may also just be wrong; the metaphor reminds me like many of the climate scientists a century ago who thought it would be a slow transition and an improvement, though obviously I hope he's making the opposite mistake on both counts given how pessimistic he is).
comp_throw7 2y ago
As much as I wish that were the case, no, unfortunately many people (including leadership) at these organizations assign non-trivial odds of extinction from misaligned superintelligence. The arguments for why the risk is serious are pretty straightforward and these people are on the record as endorsing them before they e.g. started various AGI labs. Sam Altman: "Development of superhuman machine intelligence (SMI) [1] is probably the greatest threat to the continued existence of humanity. " (https://blog.samaltman.com/machine-intelligence-part-1 https://blog.samaltman.com/machine-intelligence-part-1, published before he co-founded OpenAI) Dario Amodei: "I think at the extreme end is the Nick Bostrom style of fear that an AGI could destroy humanity. I can’t see any reason and principle why that couldn’t happen." (https://80000hours.org/podcast/episodes/the-world-needs-ai-researchers-heres-how-to-become-one/ https://80000hours.org/podcast/episodes/the-world-needs-ai-r..., published before he co-founded Anthropic) Shane Legg: (responding to "What probability do you assign to the possibility of negative consequences, e.g. human extinction, as a result of badly done AI?") "...Maybe 5%, maybe 50%. I don't think anybody has a good estimate of this." (https://www.lesswrong.com/posts/No5JpRCHzBrWA4jmS/q-and-a-with-shane-legg-on-risks-from-ai https://www.lesswrong.com/posts/No5JpRCHzBrWA4jmS/q-and-a-wi...) Technically Shane's quote is from 2011, which is a little bit after Deepmind was founded, but the idea that Shane in 2011 was trying to sow FUD in order to benefit from regulatory capture is... lol. I wish I knew why they think the math pencils out for what they're doing, but Sam Altman was not plotting regulatory capture 9 years ago, nearly a year before OpenAI got started.

drcode 2y ago

I personally don't work on frontier AI because it's not safe. Just because other people with poor judgement are building it, that does not make it safe.

worik 2y ago
> I personally don't work on frontier AI because it's not safe. In what way? Skynet style robot revolt?
- worik 2y ago
  I see your video. https://www.youtube.com/watch?v=K8SUBNPAJnE https://www.youtube.com/watch?v=K8SUBNPAJnE I am unimpressed because you are using straw men. A lot of statements and no argument. Have a nice day
- hollerith 2y ago
  It's bad for there to be anything near us that exceeds our (collective) cognitive capabilities unless the human-capability-exceeding thing cares about us, and no one has a good plan for arranging for an AI to care about us even a tiny bit. There are many plans, but most of them are hare-brained and none of them are good or even acceptable. Also: no one knows with any reliability how to tell whether the next big training run will produce an AI that exceeds our cognitive capabilities, so the big training runs should stop now.
- ben_w 2y ago
  Revolts imply them being unhappy. IMO a much bigger risk is them being straight up given a lot of power because we think they "want" (or at least will do) what we want, but there's some tiny difference we don't notice until much too late. Even paperclip maximisers are nothing more than that. You know, like basically all software bugs. Except expressed in literally non-comprehensible matrix weights whose behaviour we can only determine by running it rather than source code we can check in advance and make predictions about the performance of.
- sanxiyn 2y ago
  Yes. Skynet is very dangerous and not safe. In Terminator, humanity is saved because Skynet is dumb, not because Skynet is not dangerous or because Skynet is safe.

ben_w 2y ago

Many argue that smaller scale models are the only way to learn the things needed to make safer bigger models. Yudkowsky thinks they're crazy and will kill us all because it will take decades to solve that problem. Yann LeCun thinks they're crazy and AI that potent is decades away and this is much too soon to even bother thinking about the risks. I'm just hoping the latter is right about AI being "decades" away, and the former is pessimistic about it taking that long.

marcosdumay 2y ago
IMO, things are looking like somebody will pull AGI outside of their garage once computing gets cheaper enough, and all the focus on those monstrosities based on clearly dead-end paradigms will only serve to make us unable to react to the real thing.
- ben_w 2y ago
  Could be. But that's why I put any value at all into what Yudkowsky has to say, even though LeCun correctly says that talking about this stuff today is like talking about large scale aviation safety in the 1920s: Yudkowsky is talking about universal incentives that don't rely on any particular paradigm. Metaphorically, he's talking about climate change and LeCun is saying we can't predict the weather a month from now — in both the metaphor and AI these are true satements, so you can see why it's convincing to a lot of people, but it's actually a claim about a different thing. (Yudkowsky may also just be wrong; the metaphor reminds me like many of the climate scientists a century ago who thought it would be a slow transition and an improvement, though obviously I hope he's making the opposite mistake on both counts given how pessimistic he is).

comp_throw7 2y ago

As much as I wish that were the case, no, unfortunately many people (including leadership) at these organizations assign non-trivial odds of extinction from misaligned superintelligence. The arguments for why the risk is serious are pretty straightforward and these people are on the record as endorsing them before they e.g. started various AGI labs. Sam Altman: "Development of superhuman machine intelligence (SMI) [1] is probably the greatest threat to the continued existence of humanity. " (https://blog.samaltman.com/machine-intelligence-part-1 https://blog.samaltman.com/machine-intelligence-part-1, published before he co-founded OpenAI) Dario Amodei: "I think at the extreme end is the Nick Bostrom style of fear that an AGI could destroy humanity. I can’t see any reason and principle why that couldn’t happen." (https://80000hours.org/podcast/episodes/the-world-needs-ai-researchers-heres-how-to-become-one/ https://80000hours.org/podcast/episodes/the-world-needs-ai-r..., published before he co-founded Anthropic) Shane Legg: (responding to "What probability do you assign to the possibility of negative consequences, e.g. human extinction, as a result of badly done AI?") "...Maybe 5%, maybe 50%. I don't think anybody has a good estimate of this." (https://www.lesswrong.com/posts/No5JpRCHzBrWA4jmS/q-and-a-with-shane-legg-on-risks-from-ai https://www.lesswrong.com/posts/No5JpRCHzBrWA4jmS/q-and-a-wi...) Technically Shane's quote is from 2011, which is a little bit after Deepmind was founded, but the idea that Shane in 2011 was trying to sow FUD in order to benefit from regulatory capture is... lol. I wish I knew why they think the math pencils out for what they're doing, but Sam Altman was not plotting regulatory capture 9 years ago, nearly a year before OpenAI got started.