5 ms·

This sounds suspiciously like manufacturing weapons? I’m most interested to understand how this project was permitted in the first place. Even the public discl

This sounds suspiciously like manufacturing weapons? I’m most interested to understand how this project was permitted in the first place. Even the public disclosure of this research feels negligent.

spupe 5y ago
No one has to apply for a license to use machine learning models. They are merely communicating the fact that it's so trivially easy to do something like this, that pretty much anyone with an interest will be able to do so. The question is whether it was already trivially easy to design such compounds before or not, which I think it was, but I would be glad to hear a counterpoint.
- jjoonathan 5y ago
  Synthesis was, and still is, the hard part. It was already easy to find public information on seriously nasty nerve agents. For some intuition on the difficulty in synthesis, note that explosive chemistry and synthesis is comparatively trivial, yet there are relatively few terror attacks that go beyond commercial-off-the-shelf compounds and almost none that do it well. Personally, I think this model would be a boon for public safety because contract synthesis operations could use it to screen incoming requests for "nerve gas but changed up a bit." Basic chemical intuition probably already gets them far in this regard, but a published model could be standardized and mandated. Someone else in the thread asked about next steps. Those would be good next steps.
  - monkeybutton 5y ago
    IIRC Germany experienced some rather horrific accidents in their facilities for manufacturing chemical weapons.
  - semi-extrinsic 5y ago
    > there are relatively few terror attacks that go beyond commercial-off-the-shelf compounds and almost none that do it well The 2011 Norway attacks, 2002 Bali bombings, 2005 London underground bombings, 2008 Mumbai attacks... There is a long list of deadly terrorist attacks using homemade explosives, I don't think this is something to be dismissed.
    - civilized 5y ago
      > There is a long list of deadly terrorist attacks using homemade explosives But they used well-known and readily available compounds. They don't attempt to use novel ones. It would just make the attack way more difficult to execute.
bpodgursky 5y ago
I would rather than policymakers hear about the destructive potential of this technology before, and not after, an actual bad actor uses it.
jwuphysics 5y ago
They've certainly given this some thought: > We are but one very small company in a universe of many hundreds of companies using AI software for drug discovery and de novo design. How many of them have even considered repurposing, or misuse, possibilities? Most will work on small molecules, and many of the companies are very well funded and likely using the global chemistry network to make their AI-designed molecules. How many people have the know-how to find the pockets of chemical space that can be filled with molecules predicted to be orders of magnitude more toxic than VX?
dekhn 5y ago
What do you mean, "permitted"? It's not illegal to run these sorts of computational jobs or publish the results.
deleted 5y ago
[deleted]
jhgb 5y ago
There's a major difference between drafting and manufacturing.
numpad0 5y ago
Freedom of speech. In the free world, you’re free to plan almost anything from planning grocery procurement to dreaming global thermonuclear war, given you DON’T PUT BAD STUFF IN ACTION. That’s where and how the line is drawn. There are dangerous extensions to the definition of action but that’s the theory. Also if it’s not put it in action but rather discussed publicly, that’s just helping the society prepare and evolve.
- randcraw 5y ago
  No, the US has long classified speech on weapons of mass destruction, like how to configure shaped charges to trigger atom bombs. You're not free to circumvent those restraints. It's likely that identifying molecules suited only to become bioweapons or publishing how to synthesize them will also fall under the same strictures, as necessary to ensure public safety. Free speech is a principle to guide laws, not a law unto itself. Protecting speech that makes people less free is oxymoronic.
  - Teever 5y ago
    Yeah but how is this going to be enforced? Are you going to spell out in the law exactly what people aren't allowed to research? You've just implemented a holding pattern while giving them a road map on what to research. So your holding pattern is going to be set up to buy you x number of years, where you assume it will take your adversary x number of years to fill in the gaps on the map you just gave them. Is that enough?
    - randcraw 5y ago
      No, from what I've observed, such laws most directly constrain those who fund research as well as those who solicit/promote it. Often they accompany other criminal charges rather than motivate prosecution as a primary charge. Security and espionage laws are often not directly enforceable. Their value is to "pile on", adding severity to related matters, such as charges committing or aiding past terrorist acts or planning therefor. Ideally the threat of direct prosecution for a crime would deter possible perpetrators, but making penalties more severe and extending incarceration are useful roles for laws too.

spupe 5y ago

No one has to apply for a license to use machine learning models. They are merely communicating the fact that it's so trivially easy to do something like this, that pretty much anyone with an interest will be able to do so. The question is whether it was already trivially easy to design such compounds before or not, which I think it was, but I would be glad to hear a counterpoint.

jjoonathan 5y ago
Synthesis was, and still is, the hard part. It was already easy to find public information on seriously nasty nerve agents. For some intuition on the difficulty in synthesis, note that explosive chemistry and synthesis is comparatively trivial, yet there are relatively few terror attacks that go beyond commercial-off-the-shelf compounds and almost none that do it well. Personally, I think this model would be a boon for public safety because contract synthesis operations could use it to screen incoming requests for "nerve gas but changed up a bit." Basic chemical intuition probably already gets them far in this regard, but a published model could be standardized and mandated. Someone else in the thread asked about next steps. Those would be good next steps.
- monkeybutton 5y ago
  IIRC Germany experienced some rather horrific accidents in their facilities for manufacturing chemical weapons.
- semi-extrinsic 5y ago
  > there are relatively few terror attacks that go beyond commercial-off-the-shelf compounds and almost none that do it well The 2011 Norway attacks, 2002 Bali bombings, 2005 London underground bombings, 2008 Mumbai attacks... There is a long list of deadly terrorist attacks using homemade explosives, I don't think this is something to be dismissed.
  - civilized 5y ago
    > There is a long list of deadly terrorist attacks using homemade explosives But they used well-known and readily available compounds. They don't attempt to use novel ones. It would just make the attack way more difficult to execute.

bpodgursky 5y ago

I would rather than policymakers hear about the destructive potential of this technology before, and not after, an actual bad actor uses it.

jwuphysics 5y ago

They've certainly given this some thought: > We are but one very small company in a universe of many hundreds of companies using AI software for drug discovery and de novo design. How many of them have even considered repurposing, or misuse, possibilities? Most will work on small molecules, and many of the companies are very well funded and likely using the global chemistry network to make their AI-designed molecules. How many people have the know-how to find the pockets of chemical space that can be filled with molecules predicted to be orders of magnitude more toxic than VX?

dekhn 5y ago

What do you mean, "permitted"? It's not illegal to run these sorts of computational jobs or publish the results.

deleted 5y ago

[deleted]

jhgb 5y ago

There's a major difference between drafting and manufacturing.

numpad0 5y ago

Freedom of speech. In the free world, you’re free to plan almost anything from planning grocery procurement to dreaming global thermonuclear war, given you DON’T PUT BAD STUFF IN ACTION. That’s where and how the line is drawn. There are dangerous extensions to the definition of action but that’s the theory. Also if it’s not put it in action but rather discussed publicly, that’s just helping the society prepare and evolve.

randcraw 5y ago
No, the US has long classified speech on weapons of mass destruction, like how to configure shaped charges to trigger atom bombs. You're not free to circumvent those restraints. It's likely that identifying molecules suited only to become bioweapons or publishing how to synthesize them will also fall under the same strictures, as necessary to ensure public safety. Free speech is a principle to guide laws, not a law unto itself. Protecting speech that makes people less free is oxymoronic.
- Teever 5y ago
  Yeah but how is this going to be enforced? Are you going to spell out in the law exactly what people aren't allowed to research? You've just implemented a holding pattern while giving them a road map on what to research. So your holding pattern is going to be set up to buy you x number of years, where you assume it will take your adversary x number of years to fill in the gaps on the map you just gave them. Is that enough?
  - randcraw 5y ago
    No, from what I've observed, such laws most directly constrain those who fund research as well as those who solicit/promote it. Often they accompany other criminal charges rather than motivate prosecution as a primary charge. Security and espionage laws are often not directly enforceable. Their value is to "pile on", adding severity to related matters, such as charges committing or aiding past terrorist acts or planning therefor. Ideally the threat of direct prosecution for a crime would deter possible perpetrators, but making penalties more severe and extending incarceration are useful roles for laws too.