6 ms·
Dual use of artificial-intelligence-powered drug discovery
- heliostatic 5y agoFrom the paper: > In less than 6 hours after starting on our in-house server, our model generated 40,000 molecules that scored within our desired threshold. In the process, the AI designed not only VX, but also many other known chemical warfare agents that we identified through visual confirmation with structures in public chemistry databases. Many new molecules were also designed that looked equally plausible. These new molecules were predicted to be more toxic, based on the predicted LD50 values, than publicly known chemical warfare agents (Fig. 1). This was unexpected because the datasets we used for training the AI did not include these nerve agents. The virtual molecules even occupied a region of molecular property space that was entirely separate from the many thousands of molecules in the organism-specific LD50 model, which comprises mainly pesticides, environmental toxins and drugs (Fig. 1). By inverting the use of our machine learning models, we had transformed our innocuous generative model from a helpful tool of medicine to a generator of likely deadly molecules.
- bottled_poe 5y agoThis sounds suspiciously like manufacturing weapons? I’m most interested to understand how this project was permitted in the first place. Even the public disclosure of this research feels negligent.
- spupe 5y agoNo one has to apply for a license to use machine learning models. They are merely communicating the fact that it's so trivially easy to do something like this, that pretty much anyone with an interest will be able to do so. The question is whether it was already trivially easy to design such compounds before or not, which I think it was, but I would be glad to hear a counterpoint.
- jjoonathan 5y agoSynthesis was, and still is, the hard part. It was already easy to find public information on seriously nasty nerve agents. For some intuition on the difficulty in synthesis, note that explosive chemistry and synthesis is comparatively trivial, yet there are relatively few terror attacks that go beyond commercial-off-the-shelf compounds and almost none that do it well. Personally, I think this model would be a boon for public safety because contract synthesis operations could use it to screen incoming requests for "nerve gas but changed up a bit." Basic chemical intuition probably already gets them far in this regard, but a published model could be standardized and mandated. Someone else in the thread asked about next steps. Those would be good next steps.
- monkeybutton 5y agoIIRC Germany experienced some rather horrific accidents in their facilities for manufacturing chemical weapons.
- semi-extrinsic 5y ago> there are relatively few terror attacks that go beyond commercial-off-the-shelf compounds and almost none that do it well The 2011 Norway attacks, 2002 Bali bombings, 2005 London underground bombings, 2008 Mumbai attacks... There is a long list of deadly terrorist attacks using homemade explosives, I don't think this is something to be dismissed.
- civilized 5y ago> There is a long list of deadly terrorist attacks using homemade explosives But they used well-known and readily available compounds. They don't attempt to use novel ones. It would just make the attack way more difficult to execute.
- bpodgursky 5y agoI would rather than policymakers hear about the destructive potential of this technology before, and not after, an actual bad actor uses it.
- jwuphysics 5y agoThey've certainly given this some thought: > We are but one very small company in a universe of many hundreds of companies using AI software for drug discovery and de novo design. How many of them have even considered repurposing, or misuse, possibilities? Most will work on small molecules, and many of the companies are very well funded and likely using the global chemistry network to make their AI-designed molecules. How many people have the know-how to find the pockets of chemical space that can be filled with molecules predicted to be orders of magnitude more toxic than VX?
- dekhn 5y agoWhat do you mean, "permitted"? It's not illegal to run these sorts of computational jobs or publish the results.
- deleted 5y ago[deleted]
- jhgb 5y agoThere's a major difference between drafting and manufacturing.
- numpad0 5y agoFreedom of speech. In the free world, you’re free to plan almost anything from planning grocery procurement to dreaming global thermonuclear war, given you DON’T PUT BAD STUFF IN ACTION. That’s where and how the line is drawn. There are dangerous extensions to the definition of action but that’s the theory. Also if it’s not put it in action but rather discussed publicly, that’s just helping the society prepare and evolve.
- randcraw 5y agoNo, the US has long classified speech on weapons of mass destruction, like how to configure shaped charges to trigger atom bombs. You're not free to circumvent those restraints. It's likely that identifying molecules suited only to become bioweapons or publishing how to synthesize them will also fall under the same strictures, as necessary to ensure public safety. Free speech is a principle to guide laws, not a law unto itself. Protecting speech that makes people less free is oxymoronic.
- Teever 5y agoYeah but how is this going to be enforced? Are you going to spell out in the law exactly what people aren't allowed to research? You've just implemented a holding pattern while giving them a road map on what to research. So your holding pattern is going to be set up to buy you x number of years, where you assume it will take your adversary x number of years to fill in the gaps on the map you just gave them. Is that enough?
- randcraw 5y agoNo, from what I've observed, such laws most directly constrain those who fund research as well as those who solicit/promote it. Often they accompany other criminal charges rather than motivate prosecution as a primary charge. Security and espionage laws are often not directly enforceable. Their value is to "pile on", adding severity to related matters, such as charges committing or aiding past terrorist acts or planning therefor. Ideally the threat of direct prosecution for a crime would deter possible perpetrators, but making penalties more severe and extending incarceration are useful roles for laws too.
- ReptileMan 5y agoThat is really in the attack of captain obvious territory. Next we can expect - we turn our algorithm to find energy storage molecules and what we got is explosives.
- spicybright 5y agoResearchers seemed surprised by the results. And, from a lay person like myself, I wouldn't have thought "make medications for specific illnesses, and make sure they don't effect the rest of the body" is the logical opposite of "make chemicals that kill people as effectively as possible".
- echelon 5y agoThis is more evidence for why I think the "Vulnerable World Hypothesis" is not only a real framework, but that it describes the most eminent danger to society. It's far more dangerous than nuclear MAD, in my opinion. We're entering into a future where average educated people will be able to synthesize biochemical agents, delivery mechanisms, viruses, and more. I've thought up half a dozen low-hanging fruit that I think anyone could build today. You can probably do the same if you think about it. You don't even have to attack humans. Our society is dependent on a lot of assumptions. And just as VWH states, I don't think it can be defended against. It's scary.
- qq66 5y agoI don't even think you need a hostile actor, you could just have an incompetent actor who flips the sign on a value, and instead of designing a biological agent to maximize wheat production, designs one that minimizes wheat production, and ends up exterminating all global wheat.
- spicybright 5y agoTo be fair, at least for your example, you'd need a really long chain of incompetent actors to kill off wheat. We should really be scared of the lone and bored grad student making a wheat killer and accidentally dropping the test tube.
- astrange 5y agoI'd be more worried about someone developing a plastic-recycling microbe that works too well. There's already lots of viruses that attack plants, but the plants have immune systems for that.
- deleted 5y ago[deleted]
- krageon 5y agoQuite an interesting paper in it's impact study, but the questions it asks all boil down to "the masses are not to be trusted with our vaunted knowledge, because harm can be done. How can we prevent them from moving outside the boundaries of knowledge we like" and then the example used is how GPT-3 is managed. This is without a doubt a terrific example of bad technological stewardship: Any malicious state actor gets to use what you make for evil, while at the same time keeping it away from the folks you're making it for. I wish they had had a more substantial and innovative thing to say about it, but as it stands it reads as regressive.
- verisimi 5y agoBut .. "Risk of misuse The thought had never previously struck us. We were vaguely aware of security concerns around work with pathogens or toxic chemicals, but that did not relate to us; we primarily operate in a virtual setting."!!! Operating in the rarefied air of 'science' they were unaware of the risk of misuse! I don't think they can even conceive of the idea that the government might misuse this technology.
- krageon 5y ago> Operation in the rarefied air of 'science' they were unaware of the risk of misuse! The thundering irony of this concept in a timeline where a (group of) scientist(s) discovered fission cannot be overstated.
- jjoonathan 5y agoThe thundering irony of blaming the group of people who made you aware of a problem for being unaware of the problem is even more difficult to overstate.
- deleted 5y ago[deleted]
- jjoonathan 5y agoThey wrote a paper on it. They weren't just aware, they were demonstrably aware. Or are you suggesting that we blame people for having a finite mental processing speed?
- catchclose8919 5y ago...heh, can't believe it we're not yet at "Siri, use the TensorFlow 4.6 OmegaFold model to design potentially stable variants of COVID-25 with active BSE prionic inserts. ...DONE. Great, now place an order of the best generated sequence to the cheapest gene synthesis provider in <unregulared country x>. Air-mail delivery."
- jostmey 5y agoYou should read "Oryx and Crake" by Atwood
- joshmarlow 5y agoHave you read the sequel? I've been debating picking it up.
- jdrc 5y agoSiri, print 10B vaccines
- catchclose8919 5y agoUnfortunately things are not that symmetrical... just as it is much easier to make a missile that can get through any given missile defense system than it was to make that defense system, it's many orders of magnitude easier/cheaper (if at all practically possible) to create stuff that causes an incurable lethal condition than it is to create an antidote. (And the root causes of it all are very likely quite fundamental and unbypassable, consequences of the 2nd law bubbling up the matter complexity chain...) I'd advise against doing more search on these topics if you want to sleep well at night and/or if you don't think you're already on one "naughty list" or another ;)
- deleted 5y ago[deleted]
- zanethomas 5y agoSomeone somewhere is now generating new psychoactive drug candidates.
- ellimilial 5y agoTry a nociception, drug design or pharmacy group at your local university. Chances are they’ve been doing this for years.
- rocgf 5y agoWell, we can definitely hope so.
- spupe 5y agoHonestly, it sounded too over-dramatic and did not offer meaningful actionable consequences. The actual data is thin and not surprising at all, and it's also unclear how difficult it would be to design such harmful compounds without AI. I thought any capable chemist could easily come up with hundreds of harmful substances, old and new. I guess the AI is removing the chemist from the equation?
- scratcheee 5y agoQuantity is a quality all of its own. Being able to identify many such compounds very quickly implies some new threat models that are a bit disturbing. Of the 40000 substances identified, perhaps one of them has another interesting characteristic we've never seen in a toxic compound before? Perhaps such characteristics could be idenfified and filtered for, and perhaps someone could pin down something entirely unique and outside of any threat-assessment anyone's produced. >and not surprising at all it's obvious in retrospect, but they claim that the threat vector had not been seriously concidered before, and I see no reason to doubt their assessment. Many things aren't surprising once they've been pointed out, but were missed for far too long beforehand, and I personally had never concidered this before, despite seeing multiple articles about AI-generated substances for positive purposes.
- wombatpm 5y agoFor example, is one of the super VX alternatives a good candidate for a binary type delivery? Where two or more compounds react on deployment to create the toxic version. Now you can have people safely (for some value of safe when performing organic chemistry) producing the components in many more laboratories
- SamBam 5y agoThis is also currently being discussed here: https://news.ycombinator.com/item?id=30698803 https://news.ycombinator.com/item?id=30698803 (Top of the front page, 56 comments.)
- amelius 5y agoUse of these drugs would go against the Geneva Protocol.
- csee 5y agoThe people who would use them don't care about the Geneva Protocol.
- wolverine876 5y ago> The people who would use them don't care about the Geneva Protocol. Unless you mean that, by definition, anyone who violates the Geneva Conventions doesn't care about them, you are overlooking that most major militaries observe the Geneva Conventions to a great extent. They are cleverly written to make it in everyone's interests.
- 323 5y agoPutin did it in Salisbury and nothing happened to him.
- deleted 5y ago[deleted]
- 323 5y agoThis particular issue, toxins, doesn't seem like a huge problem in practice. You can already buy extremely potent toxins that also work against humans at the hardware store (some pesticides/insecticides/rat poisons). New toxins might be interesting for covert operations, for example killing someone without being detected by a toxicological report, but this seem to have limited usage.
- deleted 5y ago[deleted]
- tginart 5y agoIt is trivial that you can flip a sign on any network optimizing for something good and now it’s optimizing for something bad… But it is still interesting that the network generalized to these toxic nerve agents without having them in the training set
- hackerfromthefu 5y agoI believe we are heading towards personal safety bubbles, where dangerous molecular detection and warning is done in realtime via smartphone or other personal device such as glasses or implant. It might take awhile coming, but that seems the natural conclusion to having supercomputers available for individuals. Would the detection be possible via advanced mmwave/field detection, or would it have to have a molecular mini lab system of some sort ala pcr? Great market opportunity here. This seems inevitable because the rise of various threats is also inevitable via technology, and us humans like to be safe.
- imranq 5y agoAI discovers drugs to make researchers better at AI discovery
- optimalsolver 5y agoAnd away we go.