6 ms·
From the paper: > In less than 6 hours after starting on our in-house server, our model generated 40,000 molecules that scored within our desired threshold. In
by heliostatic 5y ago
From the paper:
> In less than 6 hours after starting on our in-house server, our model generated 40,000 molecules that scored within our desired threshold. In the process, the AI designed not only VX, but also many other known chemical warfare agents that we identified through visual confirmation with structures in public chemistry databases. Many new molecules were also designed that looked equally plausible. These new molecules were predicted to be more toxic, based on the predicted LD50 values, than publicly known chemical warfare agents (Fig. 1). This was unexpected because the datasets we used for training the AI did not include these nerve agents. The virtual molecules even occupied a region of molecular property space that was entirely separate from the many thousands of molecules in the organism-specific LD50 model, which comprises mainly pesticides, environmental toxins and drugs (Fig. 1). By inverting the use of our machine learning models, we had transformed our innocuous generative model from a helpful tool of medicine to a generator of likely deadly molecules.
- bottled_poe 5y agoThis sounds suspiciously like manufacturing weapons? I’m most interested to understand how this project was permitted in the first place. Even the public disclosure of this research feels negligent.
- spupe 5y agoNo one has to apply for a license to use machine learning models. They are merely communicating the fact that it's so trivially easy to do something like this, that pretty much anyone with an interest will be able to do so. The question is whether it was already trivially easy to design such compounds before or not, which I think it was, but I would be glad to hear a counterpoint.
- jjoonathan 5y agoSynthesis was, and still is, the hard part. It was already easy to find public information on seriously nasty nerve agents. For some intuition on the difficulty in synthesis, note that explosive chemistry and synthesis is comparatively trivial, yet there are relatively few terror attacks that go beyond commercial-off-the-shelf compounds and almost none that do it well. Personally, I think this model would be a boon for public safety because contract synthesis operations could use it to screen incoming requests for "nerve gas but changed up a bit." Basic chemical intuition probably already gets them far in this regard, but a published model could be standardized and mandated. Someone else in the thread asked about next steps. Those would be good next steps.
- monkeybutton 5y agoIIRC Germany experienced some rather horrific accidents in their facilities for manufacturing chemical weapons.
- semi-extrinsic 5y ago> there are relatively few terror attacks that go beyond commercial-off-the-shelf compounds and almost none that do it well The 2011 Norway attacks, 2002 Bali bombings, 2005 London underground bombings, 2008 Mumbai attacks... There is a long list of deadly terrorist attacks using homemade explosives, I don't think this is something to be dismissed.
- civilized 5y ago> There is a long list of deadly terrorist attacks using homemade explosives But they used well-known and readily available compounds. They don't attempt to use novel ones. It would just make the attack way more difficult to execute.
- bpodgursky 5y agoI would rather than policymakers hear about the destructive potential of this technology before, and not after, an actual bad actor uses it.
- jwuphysics 5y agoThey've certainly given this some thought: > We are but one very small company in a universe of many hundreds of companies using AI software for drug discovery and de novo design. How many of them have even considered repurposing, or misuse, possibilities? Most will work on small molecules, and many of the companies are very well funded and likely using the global chemistry network to make their AI-designed molecules. How many people have the know-how to find the pockets of chemical space that can be filled with molecules predicted to be orders of magnitude more toxic than VX?
- dekhn 5y agoWhat do you mean, "permitted"? It's not illegal to run these sorts of computational jobs or publish the results.
- deleted 5y ago[deleted]
- jhgb 5y agoThere's a major difference between drafting and manufacturing.
- numpad0 5y agoFreedom of speech. In the free world, you’re free to plan almost anything from planning grocery procurement to dreaming global thermonuclear war, given you DON’T PUT BAD STUFF IN ACTION. That’s where and how the line is drawn. There are dangerous extensions to the definition of action but that’s the theory. Also if it’s not put it in action but rather discussed publicly, that’s just helping the society prepare and evolve.
- randcraw 5y agoNo, the US has long classified speech on weapons of mass destruction, like how to configure shaped charges to trigger atom bombs. You're not free to circumvent those restraints. It's likely that identifying molecules suited only to become bioweapons or publishing how to synthesize them will also fall under the same strictures, as necessary to ensure public safety. Free speech is a principle to guide laws, not a law unto itself. Protecting speech that makes people less free is oxymoronic.
- Teever 5y agoYeah but how is this going to be enforced? Are you going to spell out in the law exactly what people aren't allowed to research? You've just implemented a holding pattern while giving them a road map on what to research. So your holding pattern is going to be set up to buy you x number of years, where you assume it will take your adversary x number of years to fill in the gaps on the map you just gave them. Is that enough?
- randcraw 5y agoNo, from what I've observed, such laws most directly constrain those who fund research as well as those who solicit/promote it. Often they accompany other criminal charges rather than motivate prosecution as a primary charge. Security and espionage laws are often not directly enforceable. Their value is to "pile on", adding severity to related matters, such as charges committing or aiding past terrorist acts or planning therefor. Ideally the threat of direct prosecution for a crime would deter possible perpetrators, but making penalties more severe and extending incarceration are useful roles for laws too.
- ReptileMan 5y agoThat is really in the attack of captain obvious territory. Next we can expect - we turn our algorithm to find energy storage molecules and what we got is explosives.
- spicybright 5y agoResearchers seemed surprised by the results. And, from a lay person like myself, I wouldn't have thought "make medications for specific illnesses, and make sure they don't effect the rest of the body" is the logical opposite of "make chemicals that kill people as effectively as possible".
- echelon 5y agoThis is more evidence for why I think the "Vulnerable World Hypothesis" is not only a real framework, but that it describes the most eminent danger to society. It's far more dangerous than nuclear MAD, in my opinion. We're entering into a future where average educated people will be able to synthesize biochemical agents, delivery mechanisms, viruses, and more. I've thought up half a dozen low-hanging fruit that I think anyone could build today. You can probably do the same if you think about it. You don't even have to attack humans. Our society is dependent on a lot of assumptions. And just as VWH states, I don't think it can be defended against. It's scary.
- qq66 5y agoI don't even think you need a hostile actor, you could just have an incompetent actor who flips the sign on a value, and instead of designing a biological agent to maximize wheat production, designs one that minimizes wheat production, and ends up exterminating all global wheat.
- spicybright 5y agoTo be fair, at least for your example, you'd need a really long chain of incompetent actors to kill off wheat. We should really be scared of the lone and bored grad student making a wheat killer and accidentally dropping the test tube.
- astrange 5y agoI'd be more worried about someone developing a plastic-recycling microbe that works too well. There's already lots of viruses that attack plants, but the plants have immune systems for that.
- deleted 5y ago[deleted]