7 ms·

It’s worth noting also that many academics who signed the statement may face adverse issues like reputational risk as well as funding cut to their research prog

It’s worth noting also that many academics who signed the statement may face adverse issues like reputational risk as well as funding cut to their research programs if AI safety becomes an official policy. For a large number of them, these risks are worth far more than any possible gain from signing it. When a large number of smart, reputable people, including many with expert knowledge and little or negative incentives to act dishonestly, put their names down like this, one should pay attention. Added: Paul Christiano, a brilliant theoretical CS researcher who switched to AI Alignment several years ago, put the risks of “doom” for humanity at 46%. https://www.lesswrong.com/posts/xWMqsvHapP3nwdSW8/my-views-on-doom https://www.lesswrong.com/posts/xWMqsvHapP3nwdSW8/my-views-o...

mehlmao 3y ago
46% is such a precise number. This is why I can't take "rationalists", the Yudkowskys, and the Silvers seriously. Colossal assumptions turned into confidently stated probabilities.
haldujai 3y ago
Subtract OpenAI, Google, StabilityAI and Anthropic affiliated researchers (who have a lot to gain) and not many academic signatories are left. Notably missing representation from the Stanford NLP (edit: I missed that Diyi Yang is a signatory on first read) and NYU groups who’s perspective I’d also be interested in hearing. Not committing one way or another regarding the intent with this but it’s not as diverse an academic crowd as the long list may suggest and for a lot of these names there are incentives to act dishonestly (not claiming that they are).
- nopinsight 3y ago
  Even if it’s just Yoshua Bengio, Geoffrey Hinton, and Stuart Russell, we’d probably agree the risks are not negligible. There are quite a few researchers from Stanford, UC Berkeley, MIT, Carnegie Mellon, Oxford, Cambirdge, Imperial College, Edinburg, Tsinghua, etc who signed as well. Many of whom do not work for those companies. We’re talking about nuclear war level risks here. Even a 1% chance should definitely be addressed. As noted above, Paul Christiano who has worked on AI risk and thought about it for a long time put it at 46%.
  - edgyquant 3y ago
    Id like to see the equation that led to this 46%. Even long time researchers can be overcome by grift
  - haldujai 3y ago
    > There are quite a few researchers from Stanford, UC Berkeley, MIT, Carnegie Mellon, Oxford, Cambirdge, Imperial College, Edinburg, Tsinghua, etc who signed as well. I know the Stanford researchers the most and the “biggest names” in LLMs from HAI and CRFM are absent. It would be useful to have their perspective as well. I’d throw MetaAI in the mix as well. Merely pointing out that healthy skepticism here is not entirely unwarranted. > We’re talking about nuclear war level risks here. Are we? This seems a bit dramatic for LLMs.
    - nopinsight 3y ago
      LLM is already a misnomer. Latest versions are multimodal. Current versions can be used to build agents with limited autonomy. Future versions of LLMs are most likely capable of more independence. Even dumb viruses have caused catastrophic harm. Why? It’s capable of rapid self replication in a massive number of existing vessels. You add in some intelligence, vast store of knowledge, huge bandwidth, and some aid by malicious human actors, what could such a group of future autonomous agents do? More on the risks of “doom”: https://www.lesswrong.com/posts/xWMqsvHapP3nwdSW8/my-views-on-doom https://www.lesswrong.com/posts/xWMqsvHapP3nwdSW8/my-views-o...
      - haldujai 3y ago
        I mean a small group of malicious humans can already bioengineer a deadly virus with CRISPR and open source tech without AI. This is hardly the first time in history a new technological advancement may be used for nefarious purposes. It’s a discussion worth having as AI advances but if [insert evil actor] wants to cause harm there are many cheaper and easier ways to do this right now. To come out and say we need government regulation today does stink at least a little bit of protectionism as practically speaking the “most evil actors” would not adhere to whatever is being proposed, but this would impact the competitive landscape and the corporations yelling the loudest right now have the most to gain, perhaps coincidence but worth questioning.
- chefandy 3y ago
  I just took that list and separated everyone that had any commercial tie listed, regardless of the company. 35 did and 63 did not. > "Subtract OpenAI, Google, StabilityAI and Anthropic affiliated researchers (who have a lot to gain) and not many academic signatories are left." You're putting a lot of effort into painting this list in a bad light without any specific criticism or evidence of malfeasance. Frankly, it sounds like FUD to me.
  - chefandy 3y ago
    With corporate conflicts (that I recognized the names of): Yoshua Bengio: Professor of Computer Science, U. Montreal / Mila, Victoria Krakovna: Research Scientist, Google DeepMind, Mary Phuong: Research Scientist, Google DeepMind, Daniela Amodei: President, Anthropic, Samuel R. Bowman: Associate Professor of Computer Science, NYU and Anthropic, Helen King: Senior Director of Responsibility & Strategic Advisor to Research, Google DeepMind, Mustafa Suleyman: CEO, Inflection AI, Emad Mostaque: CEO, Stability AI, Ian Goodfellow: Principal Scientist, Google DeepMind, Kevin Scott: CTO, Microsoft, Eric Horvitz: Chief Scientific Officer, Microsoft, Mira Murati: CTO, OpenAI, James Manyika: SVP, Research, Technology & Society, Google-Alphabet, Demis Hassabis: CEO, Google DeepMind, Ilya Sutskever: Co-Founder and Chief Scientist, OpenAI, Sam Altman: CEO, OpenAI, Dario Amodei: CEO, Anthropic, Shane Legg: Chief AGI Scientist and Co-Founder, Google DeepMind, John Schulman: Co-Founder, OpenAI, Jaan Tallinn: Co-Founder of Skype, Adam D'Angelo: CEO, Quora, and board member, OpenAI, Simon Last: Cofounder & CTO, Notion, Dustin Moskovitz: Co-founder & CEO, Asana, Miles Brundage: Head of Policy Research, OpenAI, Allan Dafoe: AGI Strategy and Governance Team Lead, Google DeepMind, Jade Leung: Governance Lead, OpenAI, Jared Kaplan: Co-Founder, Anthropic, Chris Olah: Co-Founder, Anthropic, Ryota Kanai: CEO, Araya, Inc., Clare Lyle: Research Scientist, Google DeepMind, Marc Warner: CEO, Faculty, Noah Fiedel: Director, Research & Engineering, Google DeepMind, David Silver: Professor of Computer Science, Google DeepMind and UCL, Lila Ibrahim: COO, Google DeepMind, Marian Rogers Croak: VP Center for Responsible AI and Human Centered Technology, Google Without: Geoffrey Hinton: Emeritus Professor of Computer Science, University of Toronto, Dawn Song: Professor of Computer Science, UC Berkeley, Ya-Qin Zhang: Professor and Dean, AIR, Tsinghua University, Martin Hellman: Professor Emeritus of Electrical Engineering, Stanford, Yi Zeng: Professor and Director of Brain-inspired Cognitive AI Lab, Institute of Automation, Chinese Academy of Sciences, Xianyuan Zhan: Assistant Professor, Tsinghua University, Anca Dragan: Associate Professor of Computer Science, UC Berkeley, Bill McKibben: Schumann Distinguished Scholar, Middlebury College, Alan Robock: Distinguished Professor of Climate Science, Rutgers University, Angela Kane: Vice President, International Institute for Peace, Vienna; former UN High Representative for Disarmament Affairs, Audrey Tang: Minister of Digital Affairs and Chair of National Institute of Cyber Security, Stuart Russell: Professor of Computer Science, UC Berkeley, Andrew Barto: Professor Emeritus, University of Massachusetts, Jaime Fernández Fisac: Assistant Professor of Electrical and Computer Engineering, Princeton University, Diyi Yang: Assistant Professor, Stanford University, Gillian Hadfield: Professor, CIFAR AI Chair, University of Toronto, Vector Institute for AI, Laurence Tribe: University Professor Emeritus, Harvard University, Pattie Maes: Professor, Massachusetts Institute of Technology - Media Lab, Peter Norvig: Education Fellow, Stanford University, Atoosa Kasirzadeh: Assistant Professor, University of Edinburgh, Alan Turing Institute, Erik Brynjolfsson: Professor and Senior Fellow, Stanford Institute for Human-Centered AI, Kersti Kaljulaid: Former President of the Republic of Estonia, David Haussler: Professor and Director of the Genomics Institute, UC Santa Cruz, Stephen Luby: Professor of Medicine (Infectious Diseases), Stanford University, Ju Li: Professor of Nuclear Science and Engineering and Professor of Materials Science and Engineering, Massachusetts Institute of Technology, David Chalmers: Professor of Philosophy, New York University, Daniel Dennett: Emeritus Professor of Philosophy, Tufts University, Peter Railton: Professor of Philosophy at University of Michigan, Ann Arbor, Sheila McIlraith: Professor of Computer Science, University of Toronto, Lex Fridman: Research Scientist, MIT, Sharon Li: Assistant Professor of Computer Science, University of Wisconsin Madison, Phillip Isola: Associate Professor of Electrical Engineering and Computer Science, MIT, David Krueger: Assistant Professor of Computer Science, University of Cambridge, Jacob Steinhardt: Assistant Professor of Computer Science, UC Berkeley, Martin Rees: Professor of Physics, Cambridge University, He He: Assistant Professor of Computer Science and Data Science, New York University, David McAllester: Professor of Computer Science, TTIC, Vincent Conitzer: Professor of Computer Science, Carnegie Mellon University and University of Oxford, Bart Selman: Professor of Computer Science, Cornell University, Michael Wellman: Professor and Chair of Computer Science & Engineering, University of Michigan, Jinwoo Shin: KAIST Endowed Chair Professor, Korea Advanced Institute of Science and Technology, Dae-Shik Kim: Professor of Electrical Engineering, Korea Advanced Institute of Science and Technology (KAIST), Frank Hutter: Professor of Machine Learning, Head of ELLIS Unit, University of Freiburg, Scott Aaronson: Schlumberger Chair of Computer Science, University of Texas at Austin, Max Tegmark: Professor, MIT, Center for AI and Fundamental Interactions, Bruce Schneier: Lecturer, Harvard Kennedy School, Martha Minow: Professor, Harvard Law School, Gabriella Blum: Professor of Human Rights and Humanitarian Law, Harvard Law, Kevin Esvelt: Associate Professor of Biology, MIT, Edward Wittenstein: Executive Director, International Security Studies, Yale Jackson School of Global Affairs, Yale University, Karina Vold: Assistant Professor, University of Toronto, Victor Veitch: Assistant Professor of Data Science and Statistics, University of Chicago, Dylan Hadfield-Menell: Assistant Professor of Computer Science, MIT, Mengye Ren: Assistant Professor of Computer Science, New York University, Shiri Dori-Hacohen: Assistant Professor of Computer Science, University of Connecticut, Jess Whittlestone: Head of AI Policy, Centre for Long-Term Resilience, Sarah Kreps: John L. Wetherill Professor and Director of the Tech Policy Institute, Cornell University, Andrew Revkin: Director, Initiative on Communication & Sustainability, Columbia University - Climate School, Carl Robichaud: Program Officer (Nuclear Weapons), Longview Philanthropy, Leonid Chindelevitch: Lecturer in Infectious Disease Epidemiology, Imperial College London, Nicholas Dirks: President, The New York Academy of Sciences, Tim G. J. Rudner: Assistant Professor and Faculty Fellow, New York University, Jakob Foerster: Associate Professor of Engineering Science, University of Oxford, Michael Osborne: Professor of Machine Learning, University of Oxford, Marina Jirotka: Professor of Human Centred Computing, University of Oxford
joe_the_user 3y ago
I'd guess that a given academic isn't going to face much of a career risk for signing a statement also signed by other very prestigious academics, just the opposite. There's no part of very divided US political spectrum that I can see denouncing AI naysayers, unlike the scientists who signed anti-nuclear statements in 1960s or even people warning about global warming now (indeed, I'd guess the statement doesn't mention climate change 'cause it's still a sore point). Moreover, talking about existential risk involves the assumption the current tech is going to continue to affect more and more fields rather than peaking at some point - this assumption guarantees more funding along with funding for risk. All that said, I don't necessarily think the scientists involved are insincere. Rather, I would expect they're worried and signed this vague statement because it was something that might get traction. While the companies indeed may be "genuine" in the sense they're vaguely [concerned - edit] and also self-serving - "here's a hard problem it's important to have us wise, smart people in charge of and profiting from"
- nopinsight 3y ago
  In interviews, Geoffrey Hinton and Yoshua Bengio certainly expressed serious concerns and even some plausible regret to their life’s work. They did not say anything that can be interpreted as your last sentence suggests at all.
  - joe_the_user 3y ago
    My last sentence currently: "While the companies indeed may be "genuine" in the sense they're vaguely and also self-serving - "here's a hard problem it's important to have us wise, smart people in charge of and profiting from" - IE, I am not referring to the academics there. I'm going to edit the sentence to fill in some missing words but I don't think this will change the meaning involved.
mcguire 3y ago
On the contrary, I suspect "How do we prevent our AIs from killing everyone?" will be a major research question with a great deal of funding involved. Plus, no one seems to be suggesting things like the medical ethics field or institutional review boards, which might have deleterious impacts on their work.
joshuamorton 3y ago
You're putting a weirdly large amount of trust into, functionally, some dude who posted on lesswrong. Sure he has a PhD and is smart, but so is basically everyone else in the field, not just in alignment, and the median person in the field thinks the risk of "doom" is 2-5% (and that's conditioned on the supposed existence of a high level machine intelligence that the median expert believes might exist in 40 years). That still might be higher than you'd like, but it's not actually a huge worry in the grand scheme of things. Like, if I told you that in 40 years, there was a 50% chance of something existing that had a 2% chance of causing extreme harm to the human population, I'm actually not sure that thing should be the biggest priority. Other issues may have more than a 1% chance of leading to terrible outcomes sooner.
- comp_throw7 3y ago
  The median person in field thinks 5-10%, not 2-5%, and median timelines are shorter than 40 years. But this is all a distraction, since unaligned ASI is the default case absent significant efforts (that we aren't currently making), and trying to evaluate risk by averaging out the views of people who haven't actually explored the arguments very carefully (vs. just evaluating the arguments yourself) is a doomed effort.
  - joshuamorton 3y ago
    > The median person in field thinks 5-10%, not 2-5% The median person in the study here, under a particular definition was 5-10%, other comparable studies have found 2%, and similar questions using arguably better definitions in the same study found lower percentages. > median timelines are shorter than 40 years. The median person suggested a 50% chance in 39 years. > since unaligned ASI is the default case I challenge this assertion. Many relatively smart scholars who are more involved in the alignment space than, presumably either you or I, have put forth cogent arguments that alignment-by-default is perfectly reasonable. Dismissing those out of hand seems naive.
    - deleted 3y ago
      [deleted]
thelittleone 3y ago
Aren't we at least equally doomed without AI?
mahathu 3y ago
I think its cringe to both define doom and then come up with some unpredictability precise percentage for that scenario to occur.

mehlmao 3y ago

46% is such a precise number. This is why I can't take "rationalists", the Yudkowskys, and the Silvers seriously. Colossal assumptions turned into confidently stated probabilities.

haldujai 3y ago

Subtract OpenAI, Google, StabilityAI and Anthropic affiliated researchers (who have a lot to gain) and not many academic signatories are left. Notably missing representation from the Stanford NLP (edit: I missed that Diyi Yang is a signatory on first read) and NYU groups who’s perspective I’d also be interested in hearing. Not committing one way or another regarding the intent with this but it’s not as diverse an academic crowd as the long list may suggest and for a lot of these names there are incentives to act dishonestly (not claiming that they are).

nopinsight 3y ago
Even if it’s just Yoshua Bengio, Geoffrey Hinton, and Stuart Russell, we’d probably agree the risks are not negligible. There are quite a few researchers from Stanford, UC Berkeley, MIT, Carnegie Mellon, Oxford, Cambirdge, Imperial College, Edinburg, Tsinghua, etc who signed as well. Many of whom do not work for those companies. We’re talking about nuclear war level risks here. Even a 1% chance should definitely be addressed. As noted above, Paul Christiano who has worked on AI risk and thought about it for a long time put it at 46%.
- edgyquant 3y ago
  Id like to see the equation that led to this 46%. Even long time researchers can be overcome by grift
- haldujai 3y ago
  > There are quite a few researchers from Stanford, UC Berkeley, MIT, Carnegie Mellon, Oxford, Cambirdge, Imperial College, Edinburg, Tsinghua, etc who signed as well. I know the Stanford researchers the most and the “biggest names” in LLMs from HAI and CRFM are absent. It would be useful to have their perspective as well. I’d throw MetaAI in the mix as well. Merely pointing out that healthy skepticism here is not entirely unwarranted. > We’re talking about nuclear war level risks here. Are we? This seems a bit dramatic for LLMs.
  - nopinsight 3y ago
    LLM is already a misnomer. Latest versions are multimodal. Current versions can be used to build agents with limited autonomy. Future versions of LLMs are most likely capable of more independence. Even dumb viruses have caused catastrophic harm. Why? It’s capable of rapid self replication in a massive number of existing vessels. You add in some intelligence, vast store of knowledge, huge bandwidth, and some aid by malicious human actors, what could such a group of future autonomous agents do? More on the risks of “doom”: https://www.lesswrong.com/posts/xWMqsvHapP3nwdSW8/my-views-on-doom https://www.lesswrong.com/posts/xWMqsvHapP3nwdSW8/my-views-o...
    - haldujai 3y ago
      I mean a small group of malicious humans can already bioengineer a deadly virus with CRISPR and open source tech without AI. This is hardly the first time in history a new technological advancement may be used for nefarious purposes. It’s a discussion worth having as AI advances but if [insert evil actor] wants to cause harm there are many cheaper and easier ways to do this right now. To come out and say we need government regulation today does stink at least a little bit of protectionism as practically speaking the “most evil actors” would not adhere to whatever is being proposed, but this would impact the competitive landscape and the corporations yelling the loudest right now have the most to gain, perhaps coincidence but worth questioning.
chefandy 3y ago
I just took that list and separated everyone that had any commercial tie listed, regardless of the company. 35 did and 63 did not. > "Subtract OpenAI, Google, StabilityAI and Anthropic affiliated researchers (who have a lot to gain) and not many academic signatories are left." You're putting a lot of effort into painting this list in a bad light without any specific criticism or evidence of malfeasance. Frankly, it sounds like FUD to me.
- chefandy 3y ago
  With corporate conflicts (that I recognized the names of): Yoshua Bengio: Professor of Computer Science, U. Montreal / Mila, Victoria Krakovna: Research Scientist, Google DeepMind, Mary Phuong: Research Scientist, Google DeepMind, Daniela Amodei: President, Anthropic, Samuel R. Bowman: Associate Professor of Computer Science, NYU and Anthropic, Helen King: Senior Director of Responsibility & Strategic Advisor to Research, Google DeepMind, Mustafa Suleyman: CEO, Inflection AI, Emad Mostaque: CEO, Stability AI, Ian Goodfellow: Principal Scientist, Google DeepMind, Kevin Scott: CTO, Microsoft, Eric Horvitz: Chief Scientific Officer, Microsoft, Mira Murati: CTO, OpenAI, James Manyika: SVP, Research, Technology & Society, Google-Alphabet, Demis Hassabis: CEO, Google DeepMind, Ilya Sutskever: Co-Founder and Chief Scientist, OpenAI, Sam Altman: CEO, OpenAI, Dario Amodei: CEO, Anthropic, Shane Legg: Chief AGI Scientist and Co-Founder, Google DeepMind, John Schulman: Co-Founder, OpenAI, Jaan Tallinn: Co-Founder of Skype, Adam D'Angelo: CEO, Quora, and board member, OpenAI, Simon Last: Cofounder & CTO, Notion, Dustin Moskovitz: Co-founder & CEO, Asana, Miles Brundage: Head of Policy Research, OpenAI, Allan Dafoe: AGI Strategy and Governance Team Lead, Google DeepMind, Jade Leung: Governance Lead, OpenAI, Jared Kaplan: Co-Founder, Anthropic, Chris Olah: Co-Founder, Anthropic, Ryota Kanai: CEO, Araya, Inc., Clare Lyle: Research Scientist, Google DeepMind, Marc Warner: CEO, Faculty, Noah Fiedel: Director, Research & Engineering, Google DeepMind, David Silver: Professor of Computer Science, Google DeepMind and UCL, Lila Ibrahim: COO, Google DeepMind, Marian Rogers Croak: VP Center for Responsible AI and Human Centered Technology, Google Without: Geoffrey Hinton: Emeritus Professor of Computer Science, University of Toronto, Dawn Song: Professor of Computer Science, UC Berkeley, Ya-Qin Zhang: Professor and Dean, AIR, Tsinghua University, Martin Hellman: Professor Emeritus of Electrical Engineering, Stanford, Yi Zeng: Professor and Director of Brain-inspired Cognitive AI Lab, Institute of Automation, Chinese Academy of Sciences, Xianyuan Zhan: Assistant Professor, Tsinghua University, Anca Dragan: Associate Professor of Computer Science, UC Berkeley, Bill McKibben: Schumann Distinguished Scholar, Middlebury College, Alan Robock: Distinguished Professor of Climate Science, Rutgers University, Angela Kane: Vice President, International Institute for Peace, Vienna; former UN High Representative for Disarmament Affairs, Audrey Tang: Minister of Digital Affairs and Chair of National Institute of Cyber Security, Stuart Russell: Professor of Computer Science, UC Berkeley, Andrew Barto: Professor Emeritus, University of Massachusetts, Jaime Fernández Fisac: Assistant Professor of Electrical and Computer Engineering, Princeton University, Diyi Yang: Assistant Professor, Stanford University, Gillian Hadfield: Professor, CIFAR AI Chair, University of Toronto, Vector Institute for AI, Laurence Tribe: University Professor Emeritus, Harvard University, Pattie Maes: Professor, Massachusetts Institute of Technology - Media Lab, Peter Norvig: Education Fellow, Stanford University, Atoosa Kasirzadeh: Assistant Professor, University of Edinburgh, Alan Turing Institute, Erik Brynjolfsson: Professor and Senior Fellow, Stanford Institute for Human-Centered AI, Kersti Kaljulaid: Former President of the Republic of Estonia, David Haussler: Professor and Director of the Genomics Institute, UC Santa Cruz, Stephen Luby: Professor of Medicine (Infectious Diseases), Stanford University, Ju Li: Professor of Nuclear Science and Engineering and Professor of Materials Science and Engineering, Massachusetts Institute of Technology, David Chalmers: Professor of Philosophy, New York University, Daniel Dennett: Emeritus Professor of Philosophy, Tufts University, Peter Railton: Professor of Philosophy at University of Michigan, Ann Arbor, Sheila McIlraith: Professor of Computer Science, University of Toronto, Lex Fridman: Research Scientist, MIT, Sharon Li: Assistant Professor of Computer Science, University of Wisconsin Madison, Phillip Isola: Associate Professor of Electrical Engineering and Computer Science, MIT, David Krueger: Assistant Professor of Computer Science, University of Cambridge, Jacob Steinhardt: Assistant Professor of Computer Science, UC Berkeley, Martin Rees: Professor of Physics, Cambridge University, He He: Assistant Professor of Computer Science and Data Science, New York University, David McAllester: Professor of Computer Science, TTIC, Vincent Conitzer: Professor of Computer Science, Carnegie Mellon University and University of Oxford, Bart Selman: Professor of Computer Science, Cornell University, Michael Wellman: Professor and Chair of Computer Science & Engineering, University of Michigan, Jinwoo Shin: KAIST Endowed Chair Professor, Korea Advanced Institute of Science and Technology, Dae-Shik Kim: Professor of Electrical Engineering, Korea Advanced Institute of Science and Technology (KAIST), Frank Hutter: Professor of Machine Learning, Head of ELLIS Unit, University of Freiburg, Scott Aaronson: Schlumberger Chair of Computer Science, University of Texas at Austin, Max Tegmark: Professor, MIT, Center for AI and Fundamental Interactions, Bruce Schneier: Lecturer, Harvard Kennedy School, Martha Minow: Professor, Harvard Law School, Gabriella Blum: Professor of Human Rights and Humanitarian Law, Harvard Law, Kevin Esvelt: Associate Professor of Biology, MIT, Edward Wittenstein: Executive Director, International Security Studies, Yale Jackson School of Global Affairs, Yale University, Karina Vold: Assistant Professor, University of Toronto, Victor Veitch: Assistant Professor of Data Science and Statistics, University of Chicago, Dylan Hadfield-Menell: Assistant Professor of Computer Science, MIT, Mengye Ren: Assistant Professor of Computer Science, New York University, Shiri Dori-Hacohen: Assistant Professor of Computer Science, University of Connecticut, Jess Whittlestone: Head of AI Policy, Centre for Long-Term Resilience, Sarah Kreps: John L. Wetherill Professor and Director of the Tech Policy Institute, Cornell University, Andrew Revkin: Director, Initiative on Communication & Sustainability, Columbia University - Climate School, Carl Robichaud: Program Officer (Nuclear Weapons), Longview Philanthropy, Leonid Chindelevitch: Lecturer in Infectious Disease Epidemiology, Imperial College London, Nicholas Dirks: President, The New York Academy of Sciences, Tim G. J. Rudner: Assistant Professor and Faculty Fellow, New York University, Jakob Foerster: Associate Professor of Engineering Science, University of Oxford, Michael Osborne: Professor of Machine Learning, University of Oxford, Marina Jirotka: Professor of Human Centred Computing, University of Oxford

joe_the_user 3y ago

I'd guess that a given academic isn't going to face much of a career risk for signing a statement also signed by other very prestigious academics, just the opposite. There's no part of very divided US political spectrum that I can see denouncing AI naysayers, unlike the scientists who signed anti-nuclear statements in 1960s or even people warning about global warming now (indeed, I'd guess the statement doesn't mention climate change 'cause it's still a sore point). Moreover, talking about existential risk involves the assumption the current tech is going to continue to affect more and more fields rather than peaking at some point - this assumption guarantees more funding along with funding for risk. All that said, I don't necessarily think the scientists involved are insincere. Rather, I would expect they're worried and signed this vague statement because it was something that might get traction. While the companies indeed may be "genuine" in the sense they're vaguely [concerned - edit] and also self-serving - "here's a hard problem it's important to have us wise, smart people in charge of and profiting from"

nopinsight 3y ago
In interviews, Geoffrey Hinton and Yoshua Bengio certainly expressed serious concerns and even some plausible regret to their life’s work. They did not say anything that can be interpreted as your last sentence suggests at all.
- joe_the_user 3y ago
  My last sentence currently: "While the companies indeed may be "genuine" in the sense they're vaguely and also self-serving - "here's a hard problem it's important to have us wise, smart people in charge of and profiting from" - IE, I am not referring to the academics there. I'm going to edit the sentence to fill in some missing words but I don't think this will change the meaning involved.

mcguire 3y ago

On the contrary, I suspect "How do we prevent our AIs from killing everyone?" will be a major research question with a great deal of funding involved. Plus, no one seems to be suggesting things like the medical ethics field or institutional review boards, which might have deleterious impacts on their work.

joshuamorton 3y ago

You're putting a weirdly large amount of trust into, functionally, some dude who posted on lesswrong. Sure he has a PhD and is smart, but so is basically everyone else in the field, not just in alignment, and the median person in the field thinks the risk of "doom" is 2-5% (and that's conditioned on the supposed existence of a high level machine intelligence that the median expert believes might exist in 40 years). That still might be higher than you'd like, but it's not actually a huge worry in the grand scheme of things. Like, if I told you that in 40 years, there was a 50% chance of something existing that had a 2% chance of causing extreme harm to the human population, I'm actually not sure that thing should be the biggest priority. Other issues may have more than a 1% chance of leading to terrible outcomes sooner.

comp_throw7 3y ago
The median person in field thinks 5-10%, not 2-5%, and median timelines are shorter than 40 years. But this is all a distraction, since unaligned ASI is the default case absent significant efforts (that we aren't currently making), and trying to evaluate risk by averaging out the views of people who haven't actually explored the arguments very carefully (vs. just evaluating the arguments yourself) is a doomed effort.
- joshuamorton 3y ago
  > The median person in field thinks 5-10%, not 2-5% The median person in the study here, under a particular definition was 5-10%, other comparable studies have found 2%, and similar questions using arguably better definitions in the same study found lower percentages. > median timelines are shorter than 40 years. The median person suggested a 50% chance in 39 years. > since unaligned ASI is the default case I challenge this assertion. Many relatively smart scholars who are more involved in the alignment space than, presumably either you or I, have put forth cogent arguments that alignment-by-default is perfectly reasonable. Dismissing those out of hand seems naive.
  - deleted 3y ago
    [deleted]

thelittleone 3y ago

Aren't we at least equally doomed without AI?

mahathu 3y ago

I think its cringe to both define doom and then come up with some unpredictability precise percentage for that scenario to occur.