4 ms·

> There are quite a few researchers from Stanford, UC Berkeley, MIT, Carnegie Mellon, Oxford, Cambirdge, Imperial College, Edinburg, Tsinghua, etc who signed as

> There are quite a few researchers from Stanford, UC Berkeley, MIT, Carnegie Mellon, Oxford, Cambirdge, Imperial College, Edinburg, Tsinghua, etc who signed as well. I know the Stanford researchers the most and the “biggest names” in LLMs from HAI and CRFM are absent. It would be useful to have their perspective as well. I’d throw MetaAI in the mix as well. Merely pointing out that healthy skepticism here is not entirely unwarranted. > We’re talking about nuclear war level risks here. Are we? This seems a bit dramatic for LLMs.

nopinsight 3y ago
LLM is already a misnomer. Latest versions are multimodal. Current versions can be used to build agents with limited autonomy. Future versions of LLMs are most likely capable of more independence. Even dumb viruses have caused catastrophic harm. Why? It’s capable of rapid self replication in a massive number of existing vessels. You add in some intelligence, vast store of knowledge, huge bandwidth, and some aid by malicious human actors, what could such a group of future autonomous agents do? More on the risks of “doom”: https://www.lesswrong.com/posts/xWMqsvHapP3nwdSW8/my-views-on-doom https://www.lesswrong.com/posts/xWMqsvHapP3nwdSW8/my-views-o...
- haldujai 3y ago
  I mean a small group of malicious humans can already bioengineer a deadly virus with CRISPR and open source tech without AI. This is hardly the first time in history a new technological advancement may be used for nefarious purposes. It’s a discussion worth having as AI advances but if [insert evil actor] wants to cause harm there are many cheaper and easier ways to do this right now. To come out and say we need government regulation today does stink at least a little bit of protectionism as practically speaking the “most evil actors” would not adhere to whatever is being proposed, but this would impact the competitive landscape and the corporations yelling the loudest right now have the most to gain, perhaps coincidence but worth questioning.
  - nopinsight 3y ago
    I’m not sure there is a way for someone to engineer a deadly virus while completely innoculating themselves from it. Short-term AI risk likely comes from a mix of malicious intent and further autonomy that causes harm the perpetrators did not expect. In the longer run, there is a good chance of real autonomy and completely unexpected behaviors from AI.
    - haldujai 3y ago
      Why do you have to inoculate yourself from it to create havoc? Your analogy of “nuclear war” also has no vaccine. AI autonomy is a hypothetical existential risk, especially in the short term. There are many non-hypothetical existential risks including actual nuclear proliferation and escalating great power conflicts happening right now. Again my point being that this is an important discussion but appears overly dramatized, just like there are people screaming doomsday there are also equally qualified people (like Yann LeCun) screaming BS. But let’s entertain this for a second, can you posit a hypothetical where in the short term a nefarious actor can abuse AI or autonomy results in harm? How does this compare to non-AI alternatives for causing harm?
  - bnralt 3y ago
    > I mean a small group of malicious humans can already bioengineer a deadly virus with CRISPR and open source tech without AI. That's what interesting to me. People fearmongering about bioengineering and GMO's were generally dismissed as being anti-science and holding humankind back (or worse, that there opposition to progress meant they had blood on their hands). Yet many of the people who mocked them proved themselves to be even more dogmatic and apocalyptic, while being much closer to influencing regulations. And the technology they're fear-mongering about is even further from being able to harm people than biotech is. We are actually able to create harmful biotech today if we want; we don't know when we'll ever be able to create AGI, and if it would even pose a danger if we did. This mentality - "there could be a slight chance research into this could eventually lead to apocalyptic technology, no I don't have any idea how but the danger is so great we need a lot of regulation" - would severely harm scientific growth if we applied it consistently. Of course everyone is going to say "the technology I'm afraid of is _actually_ dangerous, the technology they're afraid if isn't." But we honestly have no clue when we're talking about technology that we have no idea how to create at the moment.
- CyrsBel 3y ago
  This gets countered by running one (or more) of those same amazing autonomous agents locally for your own defense. Everyone's machine is about to get much more intelligent.
- berniedurfee 3y ago
  “…some intelligence…” appears to be a huge leap from where we seem to be though.
ben_w 3y ago
> > We’re talking about nuclear war level risks here. > Are we? This seems a bit dramatic for LLMs. The signed statement isn't about just LLMs in much the same way that "animal" doesn't just mean "homo sapiens"
- haldujai 3y ago
  I used LLM because the people shouting the loudest come from a LLM company which claimed their newest language model can be used to create bioweapons in their whitepaper. Semantics aside the recent interest in AI risk was clearly stimulated by LLMs and the camp that believes this is the path to AGI which may or may not be true depending who you ask.
  - ben_w 3y ago
    I can only imagine Eleizer Yudkowsky and Rob Miles looking on this conversation with a depressed scream and a facepalm respectively. They've both been loudly concerned about optimisers doing over-optimisation, and society having a Nash equilibrium where everyone's using them as hard as possible regardless of errors, since before it was cool.
    - haldujai 3y ago
      While true the ones doing media tours and speaking the most vocally in May 2023 are the LLM crowd. I don’t think it’s a mischaracterization to say OpenAI has sparked public debate on this topic.
      - ben_w 3y ago
        To the extent that this may be true (I've not exactly studied which public thinkpiece writers care about which AI so it might easily be the image generators that get this crown for all I know), that's because ChatGPT actually does something that a normal person understands. A paper titled "Dual use of artificial-intelligence-powered drug discovery" (published last year) got a few angst pieces and is mostly forgotten by the general public and media so far as I can tell; but the people behind it both talked directly to regulators and other labs to help advise them how many other precursor chemicals were now potential threats, and also went onto the usual podcasts and other public forums to raise awareness of the risk to other AI researchers. The story behind that was "ugh, they want us to think about risks… what if we ask it to find dangerous chemicals instead of safe ones? *overnight* oh no!"

nopinsight 3y ago

LLM is already a misnomer. Latest versions are multimodal. Current versions can be used to build agents with limited autonomy. Future versions of LLMs are most likely capable of more independence. Even dumb viruses have caused catastrophic harm. Why? It’s capable of rapid self replication in a massive number of existing vessels. You add in some intelligence, vast store of knowledge, huge bandwidth, and some aid by malicious human actors, what could such a group of future autonomous agents do? More on the risks of “doom”: https://www.lesswrong.com/posts/xWMqsvHapP3nwdSW8/my-views-on-doom https://www.lesswrong.com/posts/xWMqsvHapP3nwdSW8/my-views-o...

haldujai 3y ago
I mean a small group of malicious humans can already bioengineer a deadly virus with CRISPR and open source tech without AI. This is hardly the first time in history a new technological advancement may be used for nefarious purposes. It’s a discussion worth having as AI advances but if [insert evil actor] wants to cause harm there are many cheaper and easier ways to do this right now. To come out and say we need government regulation today does stink at least a little bit of protectionism as practically speaking the “most evil actors” would not adhere to whatever is being proposed, but this would impact the competitive landscape and the corporations yelling the loudest right now have the most to gain, perhaps coincidence but worth questioning.
- nopinsight 3y ago
  I’m not sure there is a way for someone to engineer a deadly virus while completely innoculating themselves from it. Short-term AI risk likely comes from a mix of malicious intent and further autonomy that causes harm the perpetrators did not expect. In the longer run, there is a good chance of real autonomy and completely unexpected behaviors from AI.
  - haldujai 3y ago
    Why do you have to inoculate yourself from it to create havoc? Your analogy of “nuclear war” also has no vaccine. AI autonomy is a hypothetical existential risk, especially in the short term. There are many non-hypothetical existential risks including actual nuclear proliferation and escalating great power conflicts happening right now. Again my point being that this is an important discussion but appears overly dramatized, just like there are people screaming doomsday there are also equally qualified people (like Yann LeCun) screaming BS. But let’s entertain this for a second, can you posit a hypothetical where in the short term a nefarious actor can abuse AI or autonomy results in harm? How does this compare to non-AI alternatives for causing harm?
- bnralt 3y ago
  > I mean a small group of malicious humans can already bioengineer a deadly virus with CRISPR and open source tech without AI. That's what interesting to me. People fearmongering about bioengineering and GMO's were generally dismissed as being anti-science and holding humankind back (or worse, that there opposition to progress meant they had blood on their hands). Yet many of the people who mocked them proved themselves to be even more dogmatic and apocalyptic, while being much closer to influencing regulations. And the technology they're fear-mongering about is even further from being able to harm people than biotech is. We are actually able to create harmful biotech today if we want; we don't know when we'll ever be able to create AGI, and if it would even pose a danger if we did. This mentality - "there could be a slight chance research into this could eventually lead to apocalyptic technology, no I don't have any idea how but the danger is so great we need a lot of regulation" - would severely harm scientific growth if we applied it consistently. Of course everyone is going to say "the technology I'm afraid of is _actually_ dangerous, the technology they're afraid if isn't." But we honestly have no clue when we're talking about technology that we have no idea how to create at the moment.
CyrsBel 3y ago
This gets countered by running one (or more) of those same amazing autonomous agents locally for your own defense. Everyone's machine is about to get much more intelligent.
berniedurfee 3y ago
“…some intelligence…” appears to be a huge leap from where we seem to be though.

ben_w 3y ago

> > We’re talking about nuclear war level risks here. > Are we? This seems a bit dramatic for LLMs. The signed statement isn't about just LLMs in much the same way that "animal" doesn't just mean "homo sapiens"

haldujai 3y ago
I used LLM because the people shouting the loudest come from a LLM company which claimed their newest language model can be used to create bioweapons in their whitepaper. Semantics aside the recent interest in AI risk was clearly stimulated by LLMs and the camp that believes this is the path to AGI which may or may not be true depending who you ask.
- ben_w 3y ago
  I can only imagine Eleizer Yudkowsky and Rob Miles looking on this conversation with a depressed scream and a facepalm respectively. They've both been loudly concerned about optimisers doing over-optimisation, and society having a Nash equilibrium where everyone's using them as hard as possible regardless of errors, since before it was cool.
  - haldujai 3y ago
    While true the ones doing media tours and speaking the most vocally in May 2023 are the LLM crowd. I don’t think it’s a mischaracterization to say OpenAI has sparked public debate on this topic.
    - ben_w 3y ago
      To the extent that this may be true (I've not exactly studied which public thinkpiece writers care about which AI so it might easily be the image generators that get this crown for all I know), that's because ChatGPT actually does something that a normal person understands. A paper titled "Dual use of artificial-intelligence-powered drug discovery" (published last year) got a few angst pieces and is mostly forgotten by the general public and media so far as I can tell; but the people behind it both talked directly to regulators and other labs to help advise them how many other precursor chemicals were now potential threats, and also went onto the usual podcasts and other public forums to raise awareness of the risk to other AI researchers. The story behind that was "ugh, they want us to think about risks… what if we ask it to find dangerous chemicals instead of safe ones? *overnight* oh no!"