5 ms·

Propaganda or Science: Open-Source AI and Bioterrorism Risk

83 points by 1a3orn 3y ago

artninja1988 3y ago
An extremely impressive takedown of biorisk papers
- ethanbond 3y ago
  Idk, writing off e.g. anthrax risk because the 2001 attacks only killed 5 people is pretty silly. There's good reason to believe those attacks weren't really intended to kill people, and they certainly weren't intended to kill a large number of people. This fact also casts some doubt of relevance on the Montague paper as well, which says that a bio agent's ability to spread is so incredibly important. Yes, the ability to spread does amplify the risk enormously for obvious reasons, but there are plenty of non-spreading agents that you can do huge, huge amounts of damage with, including anthrax when used "appropriately."
  - j45 3y ago
    A slippery slope is anythings dangerous that anyone wants to be dangerous and can fall into a larger and larger nebulus of safety for the many and control for the few, which is a different kind of danger. It will be interesting to see where this attempts to contain the interpretations of fear from the interpretations of fearmongering (or not) leads to. Maybe AGI gets disagrees with all the manipulators and becomes for the masses.
    - deleted 3y ago
      [deleted]
  - sterlind 3y ago
    the 2001 Anthrax attacks are especially interesting because the Anthrax came from a DoD BSL-4 lab studying biological weapons. investigators eventually matched the Anthrax from the letters with a particular flask from the lab, and were closing in on the scientist responsible when he killed himself. Definitely doesn't support the idea that Anthrax is easy to DIY.
    - ethanbond 3y ago
      It doesn’t support the idea that anthrax was easy to DIY. I have no idea to what extent LLMs make it easier, my only point is that the author implies that the fairly small damage caused by those attacks is because it’s hard to spread anthrax (the argument made in the linked paper). To the contrary, there was a ton of effort put into ensuring the anthrax didn’t spread.
      - sterlind 3y ago
        I'm not sure what you mean that effort was put in to keep the anthrax from spreading. This stuff wasn't your grandmother's anthrax. The spores were small enough to pass through NATO biofilters. They were mixed with high levels of silicon, which apparently made them better at infecting the lungs. Sandia tried and failed 58 times to reproduce the high silica content, apparently. If anything, it sounds like the anthrax was intentionally designed to spread better.
        ethanbond 3y ago
        It seems they were delivered in such a way to mitigate chances of other people/handlers getting sick, and they were delivered with notes that immediately told the victim what to do. Very weird story! But lots of indicators it wasn’t meant to be a big catastrophe.
        sterlind 3y ago
        I'm honestly surprised more wasn't made of it. There was no shortage of 9/11 conspiracy theories, but an anthrax attack was confirmed to come from a DoD lab, and they tried to pin it on Iraq initially, and... crickets?
slowmovintarget 3y ago
Quoting the article's own tl;dr: > I examined all the biorisk-relevant citations from a policy paper arguing that we should ban powerful open source LLMs. > None of them provide good evidence for the paper's conclusion. The best of the set is evidence from statements from Anthropic -- which rest upon data that no one outside of Anthropic can even see, and on Anthropic's interpretation of that data. The rest of the evidence cited in this paper ultimately rests on a single extremely questionable "experiment" without a control group. > In all, citations in the paper provide an illusion of evidence ("look at all these citations") rather than actual evidence ("these experiments are how we know open source LLMs are dangerous and could contribute to biorisk"). > A recent further paper on this topic (published after I had started writing this review) continues this pattern of being more advocacy than science. > Almost all the bad papers that I look at are funded by Open Philanthropy. If Open Philanthropy cares about truth, then they should stop burning the epistemic commons by funding "research" that is always going to give the same result no matter the state of the world. The rest of the paper supports this thesis with... wait for it... evidence!
- AnimalMuppet 3y ago
  "burning the epistemic commons". Beautiful phrase. I may steal it.
- deleted 3y ago
  [deleted]
- artninja1988 3y ago
  > Almost all the bad papers that I look at are funded by Open Philanthropy. If Open Philanthropy cares about truth, then they should stop burning the epistemic commons by funding "research" that is always going to give the same result no matter the state of the world. I see they are in the effective altruism space. Something about them seems extremely shady to me...
- matheusmoreira 3y ago
  > policy paper arguing that we should ban powerful open source LLMs This is a corporation lobbying effort, right? I simply can't believe that anyone else would actually want the power of LLMs to be concentrated in the hands of trillionaire corporations.
MeImCounting 3y ago
The problem is that this type of analysis and evidence does not appeal to the same primal instinct that made The Terminator such a successful franchise in both the box office and pop culture. It is way more fun to entertain ideas of paperclip optimizers and emotionally manipulative super-intelligent sociopaths than to actually reason in good faith. The endless comparisons to nuclear bombs or imagined super-viruses are an inevitable result of people having lived through the rampant propaganda of the cold war. The idea of just another technology controlled exclusively by mega-corps in a world where a majority of the most powerful technologies are controlled by those corps doesnt seem that bad in comparison with nuclear annihilation. This is missing the fundamental truth of what AI is and therefore missing its most direct comparison. At risk of repeating the same arguments seen elsewhere on HN and other sites: cryptology is definitely the closest comparison. Both AI and crypto are incredibly clever applications of fundamental mathematics. Both are technologies that are absolutely disruptive to certain monopolies or methods of societal control. Both can be replicated by anyone who understands the basic principles and throws enough resources at them. I think we are going to watch a similar thing happening where there are many concerted attempts to compromise the technology itself or the availability of said tech and eventually the halls of power are going to end up targeting endpoints instead of the protocol (architecture) itself. The users of a given technology are where regulation ought to (and I believe will) lie in the end.
aaron695 3y ago
[dead]
fabian2k 3y ago
Didn't know there was even an argument here. And the first example is some extremely basic and fundamental stuff, the difference between a bacterium and the toxin it produces. Any biology textbook would cover that. This seems like a very far-fetched argument and I'm not convince by it at all.

83 points by 1a3orn 3y ago

artninja1988 3y ago

An extremely impressive takedown of biorisk papers

ethanbond 3y ago
Idk, writing off e.g. anthrax risk because the 2001 attacks only killed 5 people is pretty silly. There's good reason to believe those attacks weren't really intended to kill people, and they certainly weren't intended to kill a large number of people. This fact also casts some doubt of relevance on the Montague paper as well, which says that a bio agent's ability to spread is so incredibly important. Yes, the ability to spread does amplify the risk enormously for obvious reasons, but there are plenty of non-spreading agents that you can do huge, huge amounts of damage with, including anthrax when used "appropriately."
- j45 3y ago
  A slippery slope is anythings dangerous that anyone wants to be dangerous and can fall into a larger and larger nebulus of safety for the many and control for the few, which is a different kind of danger. It will be interesting to see where this attempts to contain the interpretations of fear from the interpretations of fearmongering (or not) leads to. Maybe AGI gets disagrees with all the manipulators and becomes for the masses.
  - deleted 3y ago
    [deleted]
- sterlind 3y ago
  the 2001 Anthrax attacks are especially interesting because the Anthrax came from a DoD BSL-4 lab studying biological weapons. investigators eventually matched the Anthrax from the letters with a particular flask from the lab, and were closing in on the scientist responsible when he killed himself. Definitely doesn't support the idea that Anthrax is easy to DIY.
  - ethanbond 3y ago
    It doesn’t support the idea that anthrax was easy to DIY. I have no idea to what extent LLMs make it easier, my only point is that the author implies that the fairly small damage caused by those attacks is because it’s hard to spread anthrax (the argument made in the linked paper). To the contrary, there was a ton of effort put into ensuring the anthrax didn’t spread.
    - sterlind 3y ago
      I'm not sure what you mean that effort was put in to keep the anthrax from spreading. This stuff wasn't your grandmother's anthrax. The spores were small enough to pass through NATO biofilters. They were mixed with high levels of silicon, which apparently made them better at infecting the lungs. Sandia tried and failed 58 times to reproduce the high silica content, apparently. If anything, it sounds like the anthrax was intentionally designed to spread better.
      - ethanbond 3y ago
        It seems they were delivered in such a way to mitigate chances of other people/handlers getting sick, and they were delivered with notes that immediately told the victim what to do. Very weird story! But lots of indicators it wasn’t meant to be a big catastrophe.
        sterlind 3y ago
        I'm honestly surprised more wasn't made of it. There was no shortage of 9/11 conspiracy theories, but an anthrax attack was confirmed to come from a DoD lab, and they tried to pin it on Iraq initially, and... crickets?

slowmovintarget 3y ago

Quoting the article's own tl;dr: > I examined all the biorisk-relevant citations from a policy paper arguing that we should ban powerful open source LLMs. > None of them provide good evidence for the paper's conclusion. The best of the set is evidence from statements from Anthropic -- which rest upon data that no one outside of Anthropic can even see, and on Anthropic's interpretation of that data. The rest of the evidence cited in this paper ultimately rests on a single extremely questionable "experiment" without a control group. > In all, citations in the paper provide an illusion of evidence ("look at all these citations") rather than actual evidence ("these experiments are how we know open source LLMs are dangerous and could contribute to biorisk"). > A recent further paper on this topic (published after I had started writing this review) continues this pattern of being more advocacy than science. > Almost all the bad papers that I look at are funded by Open Philanthropy. If Open Philanthropy cares about truth, then they should stop burning the epistemic commons by funding "research" that is always going to give the same result no matter the state of the world. The rest of the paper supports this thesis with... wait for it... evidence!

AnimalMuppet 3y ago
"burning the epistemic commons". Beautiful phrase. I may steal it.
deleted 3y ago
[deleted]
artninja1988 3y ago
> Almost all the bad papers that I look at are funded by Open Philanthropy. If Open Philanthropy cares about truth, then they should stop burning the epistemic commons by funding "research" that is always going to give the same result no matter the state of the world. I see they are in the effective altruism space. Something about them seems extremely shady to me...
matheusmoreira 3y ago
> policy paper arguing that we should ban powerful open source LLMs This is a corporation lobbying effort, right? I simply can't believe that anyone else would actually want the power of LLMs to be concentrated in the hands of trillionaire corporations.

MeImCounting 3y ago

The problem is that this type of analysis and evidence does not appeal to the same primal instinct that made The Terminator such a successful franchise in both the box office and pop culture. It is way more fun to entertain ideas of paperclip optimizers and emotionally manipulative super-intelligent sociopaths than to actually reason in good faith. The endless comparisons to nuclear bombs or imagined super-viruses are an inevitable result of people having lived through the rampant propaganda of the cold war. The idea of just another technology controlled exclusively by mega-corps in a world where a majority of the most powerful technologies are controlled by those corps doesnt seem that bad in comparison with nuclear annihilation. This is missing the fundamental truth of what AI is and therefore missing its most direct comparison. At risk of repeating the same arguments seen elsewhere on HN and other sites: cryptology is definitely the closest comparison. Both AI and crypto are incredibly clever applications of fundamental mathematics. Both are technologies that are absolutely disruptive to certain monopolies or methods of societal control. Both can be replicated by anyone who understands the basic principles and throws enough resources at them. I think we are going to watch a similar thing happening where there are many concerted attempts to compromise the technology itself or the availability of said tech and eventually the halls of power are going to end up targeting endpoints instead of the protocol (architecture) itself. The users of a given technology are where regulation ought to (and I believe will) lie in the end.

aaron695 3y ago

[dead]

fabian2k 3y ago

Didn't know there was even an argument here. And the first example is some extremely basic and fundamental stuff, the difference between a bacterium and the toxin it produces. Any biology textbook would cover that. This seems like a very far-fetched argument and I'm not convince by it at all.