5 ms·

Kevin Esvelt says open source models could soon be used by terrorists to create bioweapons. https://nitter.net/kesvelt/status/1720440451059335520 https://nitte

Kevin Esvelt says open source models could soon be used by terrorists to create bioweapons. https://nitter.net/kesvelt/status/1720440451059335520 https://nitter.net/kesvelt/status/1720440451059335520 https://en.wikipedia.org/wiki/Kevin_M._Esvelt https://en.wikipedia.org/wiki/Kevin_M._Esvelt

dmw_ng 3y ago
There's been instructions for manufacturing weapons useful for terrorism floating around since the BBS days, nothing new here
- cubefox 3y ago
  It's a big difference when you have an expert which you can ask questions.
  - xvector 3y ago
    Terrorists already have all the information they need to build some heinous shit with ~no external guidance aside from what's already on the internet.
    - cubefox 3y ago
      Engineered viruses could cause far more deaths than conventional weapons. Even more than nuclear weapons, and they are easier to manufacture.
      - snypher 3y ago
        This is the same thinking that wants to ban my 3d printer.
  - hollowturtle 3y ago
    expert?! can’t do math why should suggest weapon instructions better? at the first hallucination you explode on the spot
    - cubefox 3y ago
      Even ChatGPT-3.5 could do more than a little math: https://www.lesswrong.com/posts/qy5dF7bQcFjSKaW58/bad-at-arithmetic-promising-at-math https://www.lesswrong.com/posts/qy5dF7bQcFjSKaW58/bad-at-ari...
    - ben_w 3y ago
      That's why we're not already dead. If anyone releases all the weights of a model that does everything perfectly (or at least can use the right tools which I suspect is much easier), that model is far too valuable to make it disappear, and dangerous enough to do all the things people get worried about. The only way to prevent that is to have a culture of "don't release unless we're sure it's safe" well before you reach that threshold. I'm happy with the imperfections of gpt-3.5 and 4, both for this reason and for my own job security. But chatGPT hasn't even reached its first birthday yet, it's very early days for this.
      - riversflow 3y ago
        > The only way to prevent that You mean a complete hypothetical outside of scifi? Lets start worrying about alien invasions too? Our planet is actually, not hypothetically, becoming uninhabitable due to pollutiom. I am so tired of ML people thinking they are god and have created something of infinite power. The hubris. The birds eye view is that we need tons of major breakthroughs to allow us to overcome this climate disaster while also figuring out how to make 8 Billion+ comfortable and happy without turning the earth into a toxic waste dump, and we need this ASAP. This nonsense about AI safety is going to have a negative net affect on the lives of Billions of people, by slowing down the progress thay could be made. AI X-risk is a complete sham being used to try and control a new, powerful tool. Science requires the scientific method, which requires physical embodiment, trial and error, and disciplined observation and measurment. AI has 0 ability to do any of that, we don't even have online learning(I think that's the term, where the model learns from its usage) in any of these large models.
        ben_w 3y ago
        > You mean a complete hypothetical outside of scifi? 18 months ago, so was having an AI make even so much as a toy website by drawing a sketch on a sheet of paper, taking a photo, captioning it "make me a website that looks like this", and pressing the "go" button. > Our planet is actually, not hypothetically, becoming uninhabitable due to pollutiom. I am so tired of ML people thinking they are god and have created something of infinite power. The hubris. So much irony there. No, the planet is not becoming uninhabitable. Bits of it are, and this is bad, and this is leading to mass migration which is causing political drama. Lots of people out there get benefits from the things that cause all the various kinds of pollution, from hyper-local things like littering and fly tipping to global things like CO2 and CFCs, and the arguments they use are sometimes the same ones you just used — things like "I am so tired of these Greta Thunberg people thinking humans can change the environment. The hubris." Also, no, nobody thinks we've already created a machine god. We think we might, eventually, with a lot of concerted effort, be able to make something that's somewhat better at every cognitive task than any human, but not only do even the most optimistic estimates place that several years away, but quite a lot of people are already going "that has so many ways it can go wrong, let's not do that". Finally, one of the ways it can go wrong is basically hyper-capitalism: an AI tasked with making as much money as possible, doesn't necessarily come with the sort of mind that we have which feels shame and embarrassment when their face is put on an effigy and burned by people that would like their environment to not be polluted. > The birds eye view is that we need tons of major breakthroughs to allow us to overcome this climate disaster while also figuring out how to make 8 Billion+ comfortable and happy without turning the earth into a toxic waste dump, and we need this ASAP. This nonsense about AI safety is going to have a negative net affect on the lives of Billions of people, by slowing down the progress thay could be made. Nah, don't need a single breakthrough, we've got sufficient known solutions to solve it all already even if there's not a single new idea. Just building out the existing research-level tech for storage and renewables is more than good enough for energy and transport, similarly there already exists solutions for other domains. Also, AI isn't just LLMs and non-LLM AIs do actually help with this kind of research, it's just not exciting to the general public because 50 pages of "here's how we Navier-Stoked ourselves a new turbine design" will have most people's eyes glaze over. Unfortunately, and directly relevant to your concerns about pollution, the fact AI means more than LLMs also means that last year a team working on using AI to test chemicals for safety before they get manufactured… found 40,000 new chemical weapons in 6 hours by flipping a sign from "find safe" to "find unsafe": https://www.theverge.com/2022/3/17/22983197/ai-new-possible-chemical-weapons-generative-models-vx https://www.theverge.com/2022/3/17/22983197/ai-new-possible-... > Science requires the scientific method, which requires physical embodiment, trial and error, and disciplined observation and measurment. Yes. > AI has 0 ability to do any of that, we don't even have online learning(I think that's the term, where the model learns from its usage) in any of these large models. False, all false. AI can easily follow the scientific method, and indeed AI is basically just applied statistics so it does this by default and the hard part is to give it heuristics so it doesn't have to on things we are literally born knowing, like faces. Likewise, trial and error: that's what almost every model is doing almost all the time during their training. Only the most trivial ones can have weights calculated directly. Also, physical embodiment is a huge field all by itself. Tesla's cars and robots, Boston Dynamics — and, surprisingly, there's even a lot of research connecting robots to LLMs: https://github.com/GT-RIPL/Awesome-LLM-Robotics https://github.com/GT-RIPL/Awesome-LLM-Robotics Finally, "online learning" is only one of many ways to update models from usage; ChatGPT does something (not necessarily online learning but it could be) with the signals from the thumbs up/down and regenerate buttons to update either the model or the RLHF layer in response to them. Even the opposite of online learning, offline learning (AKA batch learning), can update models in response to new data. The term you were probably after is "incremental learning" (which can be implemented in either a batched or online fashion), and one way you can tell that an LLM (OpenAI or other) is doing this by watching the version number change over time.
  - Vetch 3y ago
    An AI that would be like an Illustrated Primer or the AIs from Fire Upon Deep is the dream from which we are currently far, doubly so for open source models. I wouldn't trust one with a sauerkraut recipe, let alone the instructions for a doomsday device. For the forseeable future, models cannot be relied upon without external resources to augment them. Yet even augmented with references, it's still proving to be a bigger challenge than expected to get reliable results.
Roark66 3y ago
Seriously? This is just silly. Everyone knows the barrier to terrorists using bio weapons is not specialist knowledge, but access to labs, equipment, reagents etc. It's the whole Guttenberg's printing press argument. "Whoaa hold on now, what do you mean you want knowledge to be freely available to the vulgar masses?" The only difference with LLMs is that you do not have to search for this knowledge by yourself, you get a very much hallucination prone AI to tell you the answers. If we extend this argument further why don't we restrict access to public libraries, scientific research and neuter Google even more. And what about Wikipedia?
- cubefox 3y ago
  Neither Wikipedia nor public libraries allow instructions to make weapons of mass destruction.
  - xcdzvyn 3y ago
    All of the information AI regurgitates is either already available online as part of its corpus (and therefore the AI plays no particular role in access to that information), or completely made up (which is likely to kill more terrorists than anyone else!) Reiterating other comments, terrorists can't make bioweapons because they lack the facilities and prerequisites, not because they're incompetent.
    - cubefox 3y ago
      Top AI researchers like Geoffrey Hinton say that large language models likely have an internal world model and aren't just stochastic parrots. Which means they can do more than just repeating strings from the training distribution. Facilities are a major hurdle for nuclear weapons. For bioweapons they are much less of a problem. The main constraint is competency.
      - junofan 3y ago
        I think you might want to take a look at some of the history here, and particularly the cyclical nature of the AI field for the past 50–60 years. It’s helpful to put what everyone’s saying in context.
matkoniecz 3y ago
> Would sharing future model weights give everyone an amoral biotech-expert tutor? > Yes. claim seems dubious to me Is he explaining somewhere why it is worse than virology scientists publishing research? Or is he proposing to ban virology as a field? Also, if AI can actually synthesize knowledge at expert level - then we have far larger problems than this anyway.
- cubefox 3y ago
  Which far larger problems? A synthetic virus could kill a large fraction of humanity.
junofan 3y ago
That thread is simply unhinged. There is no terrorist with a wet lab who outright refuses to read papers and instead relies on a chatbot to work with dangerous agents.
- cubefox 3y ago
  Would you have called the possibility of large language models helping millions of people "unhinged" a few years ago as well?
  - junofan 3y ago
    No? That’s been the goal of NLP and information retrieval research for decades.
    - cubefox 3y ago
      The goal is also to develop systems that are significantly more capable than current systems. And those systems could be misused when terrorists gain access to them. What about that is "unhinged"?
      - wavemode 3y ago
        It's unhinged because one could make slippery slope arguments about any technology killing millions of people. In the cold war era, the government didn't even want cryptography to become generally available. I mean, what if Soviet spies use it to communicate with each other and the government can't decode what they're saying? Legislators who are worried about technology killing people ought to focus their efforts on the technologies that we actually know kill people, like guns and cigarettes. (Oh but, those industries are donating money to the politicians, so they conveniently don't care much.)
        cubefox 3y ago
        Cryptography can't be used to produce weapons of mass destruction. It's a purely defensive technology. Engineered superviruses are a whole different caliber.
deleted 3y ago
[deleted]
war321 3y ago
The bottleneck for bioterrorism isn't AI telling you how to do something, it's producing the final result. You wanna curtail bioweapons, monitor the BSL labs, biowarfare labs, bioreactors, and organic 3D printers. ChatGPT telling me how to shoot someone isn't gonna help me if I can't get a gun.
- cubefox 3y ago
  I think it's mainly an infohazard. You certainly don't need large facilities like for nuclear weapons that could easily be monitored by spy satellites. The virus could be produced in any normal building. And the ingredients are likely dual use for medical applications. This stuff isn't easy to control.

dmw_ng 3y ago

There's been instructions for manufacturing weapons useful for terrorism floating around since the BBS days, nothing new here

cubefox 3y ago
It's a big difference when you have an expert which you can ask questions.
- xvector 3y ago
  Terrorists already have all the information they need to build some heinous shit with ~no external guidance aside from what's already on the internet.
  - cubefox 3y ago
    Engineered viruses could cause far more deaths than conventional weapons. Even more than nuclear weapons, and they are easier to manufacture.
    - snypher 3y ago
      This is the same thinking that wants to ban my 3d printer.
- hollowturtle 3y ago
  expert?! can’t do math why should suggest weapon instructions better? at the first hallucination you explode on the spot
  - cubefox 3y ago
    Even ChatGPT-3.5 could do more than a little math: https://www.lesswrong.com/posts/qy5dF7bQcFjSKaW58/bad-at-arithmetic-promising-at-math https://www.lesswrong.com/posts/qy5dF7bQcFjSKaW58/bad-at-ari...
  - ben_w 3y ago
    That's why we're not already dead. If anyone releases all the weights of a model that does everything perfectly (or at least can use the right tools which I suspect is much easier), that model is far too valuable to make it disappear, and dangerous enough to do all the things people get worried about. The only way to prevent that is to have a culture of "don't release unless we're sure it's safe" well before you reach that threshold. I'm happy with the imperfections of gpt-3.5 and 4, both for this reason and for my own job security. But chatGPT hasn't even reached its first birthday yet, it's very early days for this.
    - riversflow 3y ago
      > The only way to prevent that You mean a complete hypothetical outside of scifi? Lets start worrying about alien invasions too? Our planet is actually, not hypothetically, becoming uninhabitable due to pollutiom. I am so tired of ML people thinking they are god and have created something of infinite power. The hubris. The birds eye view is that we need tons of major breakthroughs to allow us to overcome this climate disaster while also figuring out how to make 8 Billion+ comfortable and happy without turning the earth into a toxic waste dump, and we need this ASAP. This nonsense about AI safety is going to have a negative net affect on the lives of Billions of people, by slowing down the progress thay could be made. AI X-risk is a complete sham being used to try and control a new, powerful tool. Science requires the scientific method, which requires physical embodiment, trial and error, and disciplined observation and measurment. AI has 0 ability to do any of that, we don't even have online learning(I think that's the term, where the model learns from its usage) in any of these large models.
      - ben_w 3y ago
        > You mean a complete hypothetical outside of scifi? 18 months ago, so was having an AI make even so much as a toy website by drawing a sketch on a sheet of paper, taking a photo, captioning it "make me a website that looks like this", and pressing the "go" button. > Our planet is actually, not hypothetically, becoming uninhabitable due to pollutiom. I am so tired of ML people thinking they are god and have created something of infinite power. The hubris. So much irony there. No, the planet is not becoming uninhabitable. Bits of it are, and this is bad, and this is leading to mass migration which is causing political drama. Lots of people out there get benefits from the things that cause all the various kinds of pollution, from hyper-local things like littering and fly tipping to global things like CO2 and CFCs, and the arguments they use are sometimes the same ones you just used — things like "I am so tired of these Greta Thunberg people thinking humans can change the environment. The hubris." Also, no, nobody thinks we've already created a machine god. We think we might, eventually, with a lot of concerted effort, be able to make something that's somewhat better at every cognitive task than any human, but not only do even the most optimistic estimates place that several years away, but quite a lot of people are already going "that has so many ways it can go wrong, let's not do that". Finally, one of the ways it can go wrong is basically hyper-capitalism: an AI tasked with making as much money as possible, doesn't necessarily come with the sort of mind that we have which feels shame and embarrassment when their face is put on an effigy and burned by people that would like their environment to not be polluted. > The birds eye view is that we need tons of major breakthroughs to allow us to overcome this climate disaster while also figuring out how to make 8 Billion+ comfortable and happy without turning the earth into a toxic waste dump, and we need this ASAP. This nonsense about AI safety is going to have a negative net affect on the lives of Billions of people, by slowing down the progress thay could be made. Nah, don't need a single breakthrough, we've got sufficient known solutions to solve it all already even if there's not a single new idea. Just building out the existing research-level tech for storage and renewables is more than good enough for energy and transport, similarly there already exists solutions for other domains. Also, AI isn't just LLMs and non-LLM AIs do actually help with this kind of research, it's just not exciting to the general public because 50 pages of "here's how we Navier-Stoked ourselves a new turbine design" will have most people's eyes glaze over. Unfortunately, and directly relevant to your concerns about pollution, the fact AI means more than LLMs also means that last year a team working on using AI to test chemicals for safety before they get manufactured… found 40,000 new chemical weapons in 6 hours by flipping a sign from "find safe" to "find unsafe": https://www.theverge.com/2022/3/17/22983197/ai-new-possible-chemical-weapons-generative-models-vx https://www.theverge.com/2022/3/17/22983197/ai-new-possible-... > Science requires the scientific method, which requires physical embodiment, trial and error, and disciplined observation and measurment. Yes. > AI has 0 ability to do any of that, we don't even have online learning(I think that's the term, where the model learns from its usage) in any of these large models. False, all false. AI can easily follow the scientific method, and indeed AI is basically just applied statistics so it does this by default and the hard part is to give it heuristics so it doesn't have to on things we are literally born knowing, like faces. Likewise, trial and error: that's what almost every model is doing almost all the time during their training. Only the most trivial ones can have weights calculated directly. Also, physical embodiment is a huge field all by itself. Tesla's cars and robots, Boston Dynamics — and, surprisingly, there's even a lot of research connecting robots to LLMs: https://github.com/GT-RIPL/Awesome-LLM-Robotics https://github.com/GT-RIPL/Awesome-LLM-Robotics Finally, "online learning" is only one of many ways to update models from usage; ChatGPT does something (not necessarily online learning but it could be) with the signals from the thumbs up/down and regenerate buttons to update either the model or the RLHF layer in response to them. Even the opposite of online learning, offline learning (AKA batch learning), can update models in response to new data. The term you were probably after is "incremental learning" (which can be implemented in either a batched or online fashion), and one way you can tell that an LLM (OpenAI or other) is doing this by watching the version number change over time.
- Vetch 3y ago
  An AI that would be like an Illustrated Primer or the AIs from Fire Upon Deep is the dream from which we are currently far, doubly so for open source models. I wouldn't trust one with a sauerkraut recipe, let alone the instructions for a doomsday device. For the forseeable future, models cannot be relied upon without external resources to augment them. Yet even augmented with references, it's still proving to be a bigger challenge than expected to get reliable results.

Roark66 3y ago

Seriously? This is just silly. Everyone knows the barrier to terrorists using bio weapons is not specialist knowledge, but access to labs, equipment, reagents etc. It's the whole Guttenberg's printing press argument. "Whoaa hold on now, what do you mean you want knowledge to be freely available to the vulgar masses?" The only difference with LLMs is that you do not have to search for this knowledge by yourself, you get a very much hallucination prone AI to tell you the answers. If we extend this argument further why don't we restrict access to public libraries, scientific research and neuter Google even more. And what about Wikipedia?

cubefox 3y ago
Neither Wikipedia nor public libraries allow instructions to make weapons of mass destruction.
- xcdzvyn 3y ago
  All of the information AI regurgitates is either already available online as part of its corpus (and therefore the AI plays no particular role in access to that information), or completely made up (which is likely to kill more terrorists than anyone else!) Reiterating other comments, terrorists can't make bioweapons because they lack the facilities and prerequisites, not because they're incompetent.
  - cubefox 3y ago
    Top AI researchers like Geoffrey Hinton say that large language models likely have an internal world model and aren't just stochastic parrots. Which means they can do more than just repeating strings from the training distribution. Facilities are a major hurdle for nuclear weapons. For bioweapons they are much less of a problem. The main constraint is competency.
    - junofan 3y ago
      I think you might want to take a look at some of the history here, and particularly the cyclical nature of the AI field for the past 50–60 years. It’s helpful to put what everyone’s saying in context.

matkoniecz 3y ago

> Would sharing future model weights give everyone an amoral biotech-expert tutor? > Yes. claim seems dubious to me Is he explaining somewhere why it is worse than virology scientists publishing research? Or is he proposing to ban virology as a field? Also, if AI can actually synthesize knowledge at expert level - then we have far larger problems than this anyway.

cubefox 3y ago
Which far larger problems? A synthetic virus could kill a large fraction of humanity.

junofan 3y ago

That thread is simply unhinged. There is no terrorist with a wet lab who outright refuses to read papers and instead relies on a chatbot to work with dangerous agents.

cubefox 3y ago
Would you have called the possibility of large language models helping millions of people "unhinged" a few years ago as well?
- junofan 3y ago
  No? That’s been the goal of NLP and information retrieval research for decades.
  - cubefox 3y ago
    The goal is also to develop systems that are significantly more capable than current systems. And those systems could be misused when terrorists gain access to them. What about that is "unhinged"?
    - wavemode 3y ago
      It's unhinged because one could make slippery slope arguments about any technology killing millions of people. In the cold war era, the government didn't even want cryptography to become generally available. I mean, what if Soviet spies use it to communicate with each other and the government can't decode what they're saying? Legislators who are worried about technology killing people ought to focus their efforts on the technologies that we actually know kill people, like guns and cigarettes. (Oh but, those industries are donating money to the politicians, so they conveniently don't care much.)
      - cubefox 3y ago
        Cryptography can't be used to produce weapons of mass destruction. It's a purely defensive technology. Engineered superviruses are a whole different caliber.

deleted 3y ago

[deleted]

war321 3y ago

The bottleneck for bioterrorism isn't AI telling you how to do something, it's producing the final result. You wanna curtail bioweapons, monitor the BSL labs, biowarfare labs, bioreactors, and organic 3D printers. ChatGPT telling me how to shoot someone isn't gonna help me if I can't get a gun.

cubefox 3y ago
I think it's mainly an infohazard. You certainly don't need large facilities like for nuclear weapons that could easily be monitored by spy satellites. The virus could be produced in any normal building. And the ingredients are likely dual use for medical applications. This stuff isn't easy to control.