25 ms·

Deliberately optimizing for harm

445 points by herodotus 5y ago

h2odragon 5y ago
> ricin is (fortunately) not all that easy to turn into a weapon, and the people who try to do it are generally somewhat disconnected from reality (and also from technical proficiency). Of course that fact was no barrier to much hype about the "dangers" it posed, either. I suspect the same now; that we have more to fear from the fear junkie propaganda than the actual facts.
- once_inc 5y ago
  I personally fear the lone-wolf attack drastically reducing in cost and effort. Where it would once be cost-prohibitive to design and manufacture your own nerve gas or lethal virus, these days with AI/ML and Crispr-cas and the like, it feels like any intelligent, deranged person wanting to take as many people to the grave with him has the tools to do just that.
  - bitexploder 5y ago
    I think this is inevitable and something we will grapple with in coming decades. Especially around genetic engineering of viruses.
  - WJW 5y ago
    Intelligent and deranged persons already have the tools to make way more casualties with way less effort using guns and/or explosives. The "problem" for them is that people who get sufficiently deranged to think that killing a few hundred (or even thousand) people will meaningfully solve the problem they are upset about will also be sufficiently deranged that their ability to reason coherently will be drastically reduced.
    - wpietri 5y ago
      Would they? I'm not seeing that as necessarily true. The Unabomber seems like a good example: https://en.wikipedia.org/wiki/Ted_Kaczynski https://en.wikipedia.org/wiki/Ted_Kaczynski Or look at mass shooting incidents: https://en.wikipedia.org/wiki/Mass_shootings_in_the_United_States https://en.wikipedia.org/wiki/Mass_shootings_in_the_United_S... The Las Vegas shooting was rationally planned and carried out. He managed to shoot nearly 500 people, killing 60. They did happen to pick conventional weapons. But is that because of rational choice, or just familiarity and availability? Imagine somebody like Kaczynski, but instead of being an award-winning young mathematician, he was an award-winning industrial chemist or genomics student.
      - uxp100 5y ago
        Kaczynski did not optimize for death, really following the lead of the shockingly common political bombing campaigns of the 70s. The Las Vegas shooter might be a better example.
        wpietri 5y ago
        Sure, but I don't think that was a necessary outcome. Consider this quote: "I felt disgusted about what my uncontrolled sexual cravings had almost led me to do. And I felt humiliated, and I violently hated the psychiatrist. Just then there came a major turning point in my life. Like a Phoenix, I burst from the ashes of my despair to a glorious new hope." I agree he went with something common to the time. But I don't think that was a necessary outcome. After all, his approach didn't achieve his goals, so we can't say his sort of terrorism is any more rational than aiming for something bigger. Indeed, the nominal goals he ended up with, one could argue that mass-death terrorism is more rational.
      - Houshalter 5y ago
        None of these examples got away with it, or even acheived any of their stated goals. If terrorism actually worked, it would be a lot scarier. Also fortunately most political extremists don't want to kill just random people. Weapons of mass destruction don't discriminate, making them poor choices.
nonrandomstring 5y ago
Many years ago a colleague who works in defence told me about a job posting he'd seen but was having a moral struggle with. The opening was for "Lethality Engineer": Ideal candidate with good physics and medical background. I said that the main perk was that at least on Halloween he wouldn't need to buy a costume. He could just go out as himself. He didn't take the job.
- stickfigure 5y ago
  I hope recent events have illustrated that if it weren't for the people who develop lethal weapons, we (as in you and I) would be helpless against the bullies of the world. Unilateral pacifism is cute philosophy only when there are rough men standing ready to do violence on their behalf.
  - MattGaiser 5y ago
    Especially since we cannot really help Ukraine with anything but tech and they are outnumbered, so the only advantage we can give is how much better our weapons are than Russian ones.
  - marvin 5y ago
    I strongly agree with this sentiment. However, it is hard for an ethical person to participate in developing war technology when possession and usage of the weapons is purely a political question, and history also has seen our side of geopolitics commit atrocities. My stance has previously been that I am unwilling to work on weapons technology, because history has shown that these weapons sometimes end up being used for an indefensible cause. Then all of a sudden you're an accomplice to murder, and getting away with it. In the light of Russia's invasion of Ukraine, which is just a continuation of its historical imperialism, working on weapons is something I would be perfectly okay with and probably even motivated to do. But stop for a moment and think what a history of aggressive military actions does to our society's ability to recruit for this important job.
    - gonehome 5y ago
xycombinator 5y ago
Is anybody searching for compounds that reduce evil intent? Something that would mellow people out without causing hallucinations. A mass tranquilizer? Not effective against lone operatives but able to be deployed against an invading army.
- rossdavidh 5y ago
  I believe that's called a sedative. Most armies aren't filled with people with evil intent; they're filled with draftees who couldn't get out of it.
- ansible 5y ago
  Uh, I don't know about that... https://en.wikipedia.org/wiki/Serenity_(2005_film) https://en.wikipedia.org/wiki/Serenity_(2005_film) On a more serious note, anything that's going to affect behavior is going to have a dosage range. Too little absorbed, and there won't be enough effect. Too much, and that will harm / kill people in interesting ways. With chemical weapons, you only worry about the bio-accumulating enough to kill your enemies. An enemy receiving more than a lethal dose isn't a problem.
- okasaki 5y ago
  It's been suggested. eg. https://www.vice.com/en/article/akzyeb/link-between-lithium-in-drinking-water-suicide-study https://www.vice.com/en/article/akzyeb/link-between-lithium-... > The report states: “These findings, which are consistent with the finding in clinical trials that lithium reduces suicide and related behaviours in people with a mood disorder, suggest that naturally occurring lithium in drinking water may have the potential to reduce the risk of suicide and may possibly help in mood stabilisation, particularly in populations with relatively high suicide rates and geographical areas with a greater range of lithium concentration in the drinking water.”
- jerf 5y ago
  Of course they are. Among the "evil intent" they would reduce is any desire to rebel against your government, so you bet all big intelligence agencies are looking into it, for instance. Science fiction wrote about this decades ago. Fortunately, there's a lot of considerations involved in deployment of anything. It's easier said than done to get something of a medical nature into a population surreptitiously, because it's hard to get a certain dose into one person without someone else getting not enough and yet someone else getting way too much. You'd have to come up with a way of delivering a medical dose in a controlled fashion and lie about it or something, you couldn't just sneak it into the food/water reliably. Further, just because someone can name the exact complicated effect they'd like doesn't mean there's a drug that corresponds to it. Serenity, already mentioned, is a bit of silly example in my opinion because such a large effect should have been found during testing. But it does no good to pacify the population such that they'd never dream of so much as peacefully voting out the current leaders if the end result is that nobody would ever dream of so much as having enough ambition to show up to their jobs and you end up conquered by the next country over without them even trying, simply because they economically run circles around you. Or any number of other possible second-order effects. In a nutshell, it's dangerous to try to undercut evolution just to stay in power if not everywhere decides to do so equally, because you'll be evolved right out along with the society you putative rule. Evolution is alive and well and anyone who thinks it's asleep and they can screw around without consequences is liable to get a lethal wakeup call.
rossdavidh 5y ago
While it's worrying and worth thinking about, the track record of using AI to generate pharmaceuticals to do good has been "mixed", except really it's just been a bust. It may someday do great things, but not much yet, and one silver lining is that AI-generated toxins are unlikely to improve on the human-designed ones, either. "That is, I'm not sure that anyone needs to deploy a new compound in order to wreak havoc - they can save themselves a lot of trouble by just making Sarin or VX, God help us."
- starwind 5y ago
  > the track record of using AI to generate pharmaceuticals to do good has been "mixed", except really it's just been a bust. Researchers have only been using AI for drug development for like 6 years, I think it's way to early to call it a bust
  - rossdavidh 5y ago
    I guess I should have said "...thus far".
ansible 5y ago
The article assumes that full development of a new chemical weapon would require more development effort. With regards to military usage: storable at room temperature, relatively easy to manufacture from commonly available precursor chemicals, etc. [1] How true is that? Are there components of this process that make things easier now? Where I have chemical structure X, and a system generates the process steps and chemicals needed to produce X. How much of the domain in chemistry / chemical engineering has been automated these days? What are the future prospects for this? [1] I assume one of the design goals for a new chemical weapon for military use is that it breaks down in the environment, but not too quickly (like say in a week or a month). Though I suppose if you want to just destroy civilization you would design for longevity in the environment instead. And being able to seep through many kinds of plastic if possible.
- deleted 5y ago
  [deleted]
- logifail 5y ago
  > Where I have chemical structure X, and a system generates the process steps and chemicals needed to produce X. Undergraduate chemistry students spend a fair amount of time learning how to look at a novel structure X and by disconnecting "backwards" it into simpler components, deduce a route by which it might be synthesed "forward" in the laboratory from readily available starting materials. There's an excellent book on this, "Organic Synthesis: The Disconnection Approach", by Stuart Warren.
  - dredmorbius 5y ago
    Interesting. Were / are you a chem major? Any other major topics or readings you could recommend for someone wanting a general understanding of key concepts in modern chemistry? I'd suppose generally: materials, synthesis, o-chem, and chem-eng. My own background: began a hard-science degree. One year undergrad uni chem.
    - 323 5y ago
      The field is called "process chemistry". A very big thing in pharma: > Process chemists take compounds that were discovered by research chemists and turn them into commercial products. They “scale up” reactions by making larger and larger quantities, first for testing, then for commercial production. The goal of a process chemist is to develop synthetic routes that are safe, cost-effective, environmentally friendly, and efficient. https://www.acs.org/content/acs/en/careers/chemical-sciences/fields/process-chemistry.html https://www.acs.org/content/acs/en/careers/chemical-sciences...
toss1 5y ago
>>The thought had never previously struck us. We were vaguely aware of security concerns around work with pathogens or toxic chemicals, ...We have spent decades using computers and AI to improve human health—not to degrade it. We were naive in thinking about the potential misuse of our trade... Of course now, the next step is to use the technology to preemptively search for and develop antidotes to the new potential weapons their tool has discovered.
openasocket 5y ago
Fortunately we don't see any real work in the chemical and biological weapons space anymore. While it would still be pretty handy for terrorist groups, in actual warfare chemical weapons aren't super useful. See https://acoup.blog/2020/03/20/collections-why-dont-we-use-chemical-weapons-anymore/ https://acoup.blog/2020/03/20/collections-why-dont-we-use-ch... .
- bsedlm 5y ago
  that we don't see it doesn't mean it is not happening.
- WinterMount223 5y ago
  I guess developments are not published on Nature.
- cogman10 5y ago
  Just doesn't seem like it's worth it even for terrorists. Why invest a bunch of time and effort making more and more deadly poisons when we've already got a wide variety of them that are cheap to manufacture, well known in how they work, and don't cost a bunch of research money to uncover?
  - openasocket 5y ago
    True, though you have to remember that threat is a social construct and isn't necessarily a rational measure. The 2001 anthrax attacks killed 5 people, injuring 17, and shocked the nation. As a direct result Congress put billions into funding for new vaccines and drugs and bio-terrorism preparedness. If 5 people were killed and 17 wounded in a mass shooting by a terrorist, would we really have reacted as strongly? If you wanted to install fear into a country, I think being attacked by some custom, previously unknown chemical weapon would scarier than sarin.
  - at_a_remove 5y ago
    At the risk of putting myself on a watchlist, They already know that, so They have an eye on certain kinds of labware, different precursors, and such. And They already have antidotes to some of these poisons. One could optimize for compounds with hard-to-monitor precursors. Compounds that can be transported with low vapor pressure and volatility, so they cannot be easily sniffed out. Or imagine a lethal compound with a high delay factor. Or something with specifically panic-inducing effects, perhaps hemorrhagic with a side effect of your skin sliding off in great slick sheets. Another interesting high delay factor compound might induce psychosis: have fun tracking where these gibbering maniacs were a week ago. With a sufficiently dark imagination, "needs" could be identified for all sorts of compounds. Remember, the goal is to throw a monkey wrench into the gearwork of an opposing civilization, not necessarily to kill. Fear of the unknown is very effective for this.
ChrisMarshallNY 5y ago
We already (sort of) do this. AI/ML is probably used for simulating nuclear explosions, and is [arguably] even more useful and accurate than actually setting off a bomb, and measuring it. It makes sense that it could be weaponized. When Skynet becomes self-aware, it would probably just design a chemical that kills us all, and would aerosol that into the atmosphere. No need for terminators, just big pesticide cans.
- wanda 5y ago
  I'm quite sure we have already invented several chemicals that match your description -- like sarin gas, invented in 1938 by someone who, indeed, wanted to create a decent pesticide. A lethal dose of sarin gas is something like 28-35m³/min over 2 minutes exposure, according to wikipedia. [0] Hitler was well aware of its creation, and I believe quite a lot of the stuff was produced for the purpose of warfare. There were several in the Nazi military who wanted to use it, but Hitler declined. That seems rather odd, given his indifference to exterminating people with gas on an industrial scale beyond the theater of war. It has been suggested that Hitler was probably aware that to use sarin gas would be to invite the allies to do so in response, which would result in a dramatic loss of life on the German side due to the sheer lethality of such chemical weapons. [1] Perhaps he thought it easier to stick to conventional warfare, in which the pace is more manageable than with WMDs, where you would start going down the road of mutually assured destruction but without the strategic framework in place to prevent anyone from actually wiping out a population before realising how bad an idea it would be. And I think this reluctance to change the game, this seemingly deliberate moderation, perhaps best demonstrates the true difference between the machine and human in warfare. It is not a difference in innovation -- we have always been very good at inventing highly optimised ways to end life. The difference is that a machine intelligence will not hesitate. It will not ask for confirmation, pause or break a sweat. It will pull the trigger first, it will point the bombs at anything that is an adversary and anything that could theoretically be or become an adversary, and it will not miss. And it will not have to face ethical criticism and historical condemnation afterwards. [2] [0]: https://en.m.wikipedia.org/wiki/Sarin https://en.m.wikipedia.org/wiki/Sarin [1]: https://www.washingtonpost.com/news/retropolis/wp/2017/04/11/hitler-refused-to-use-sarin-gas-during-wwii-the-mystery-is-why/ https://www.washingtonpost.com/news/retropolis/wp/2017/04/11... [2]: Assuming this is a Skynet-like machine intelligence, which doesn't really have the capacity for remorse or negotiation and seems primarily, indeed solely occupied with the task of ending human life. Obviously, a true AI that is essentially a conscious mind equivalent to our own minds, may experience the same hesitancy that most of us would, were our fingers to be over the buzzer. Unless the AI independently arrives at a different set of values to us, like the Borg or something.
hengheng 5y ago
So, their tool will draw molecules that are good at doing harm, and that is it? No word on stabilization (which makes it safe to handle), synthesis, purification and such. I'd wager that most of these substances have at some point been on somebody's blackboard, but deemed impractical or infeasible, and then not pursued, and that's why we don't know them by name today. Still a scary lesson though.
ck2 5y ago
Why make a chemical weapon when you can just tweak a virus which self-replicates? BA.2 is even more infectious than BA.1 which is saying something, imagine an engineered BA.3 with even more spread and then make it as even more deadly. You might even be able to target it to one race or region if there is a gene specific to that area. Always hoped the future would be Star-Trek-like but it seems all it takes is one dictator or terrorist to end the world, slowly at first but then it would double every other day and impossible to stop.
- danuker 5y ago
  If you make it too deadly, maybe it doesn't spread as far (because the hosts die). Make it just the right amount of deadly!
photochemsyn 5y ago
This is essentially what the pesticide and herbicide industries have been doing since their inception, i.e. designing molecules that efficiently kill animals, insects and plants. It seemed like a miracle at first, but the long-term consequences of things like persistent chlorinated aromatics and their derivatives (Agent Orange and dioxin for example) eventually appeared in human populations. The development of the toxic nerve agents (organo-phosphate compouds mostly) in particular was a side effect of research into insect toxins. The nerve agents were discovered in this manner, they worked too well. Nevertheless, these pesticides were deemed safer than the organochlorines because they degraded fairly rapidly after application (although they are implicated in nerve damage related diseases like Parkinson's in agricultural areas). Insect infestations are indeed a big issue in agriculture and can wipe out entire crops if not dealt with, but there are plenty of options that don't require applications of highly toxic or persistent chemicals. Otherwise, this is just another of the many issues modern technology has created. Smallpox is another one - in the late 1990s, there was a great debate over whether to destroy the last smallpox samples - and then in the mid 2000's, someone demonstrated you could recreate smallpox by ordering the appropriate DNA sequences online and assembling them in a host cell. Then there's the past ten years of CRISPR and gain-of-function research with pathogenic viruses, a very contentious topic indeed, and still unresolved.
cryptofistMonk 5y ago
This is not really that worrying IMO - we already have weaponized toxins, viruses, and enough explosives to blow up the entire planet. So what if an AI can come up with something a little bit worse? It isn't the existence of these things that's stopping us all from killing each other.
taurusnoises 5y ago
Anyone wanna ELI5? It's useful for bother explainer and receiver. ;)
- danuker 5y ago
  Before, computers were used to make less poisonous chemicals. Now, the people asking computers to do that realized they can ask the computers to make more poisonous chemicals.
- Traubenfuchs 5y ago
  Software is able to simulate the effect of chemical compounds / molecules on the human body. This can be used to find drugs that do specific things, or stronger versions of existing drugs. For example, you could look for very strong but very short acting sleeping pills that immediately make you fall asleep, but cause zero grogginess the next day. Or you could optimize antibiotics to have a high half life, so you only have to take them once, instead of 3 times a day for a week, which you can easily forget. Now think about nerve gas. We have discovered lots of different nerve gas agents and know pretty well how much of each type you need to kill a human. Said software can be used to find new versions of nerve gas that kill with even lesser concentrations. You could also optimize for other variables: Nerve gas that remains on surfaces and doesn't decay by itself for example.
- xondono 5y ago
  They had an AI that looked for safe drugs by minimizing an estimate of lethality, changed it to ‘maximize’ and the computer spewed known nerve gas agents.
VLM 5y ago
This will probably come in handy for industrial espionage type tasks. Lets say you had a nation-state enemy who eats a lot of some ethnic ingredient. Come up with a cheap artificial flavor/color or process that is optimize to give heavy consumers cancer in 30 years. Not in one year, that will show up in the approval process. Then have an agent in the target country "discover" thru random chance this really excellent food dye or whatever. Now you kill half the population with cancer, you're gonna get nuked in response, even non-nuke countries will be pissed off enough to get nukes just to nuke the perpetrator. But lets say you make the victims fat and sick and die a little younger just enough to get 1% hit on economic growth... Some people would say this is how we ended up with trans-fats and margarine and vegetable oils in general or certain veg oils in specific. Certainly, corn syrup has caused more human and economic devastation that fission, nerve agents, or most any WMD I can think of...
- 0xdeadbeefbabe 5y ago
  This is far more fun than believing in an emerging accident. You don't have to eat corn syrup, btw.
- deleted 5y ago
  [deleted]
- tenebrisalietum 5y ago
  The problem with this is that this is basically genetic engineering, you might successfully make a low-level economic growth impact now, but future generations will be resistant to the poisons as those weak against them die off. You are securing your own demise long-term if you don't subject your population to the same.
- tgtweak 5y ago
  I guess if it's tasty then it's fair game.
jleyank 5y ago
Chemical weapons above a certain level, bio weapons and nucs are all seen as weapons of mass destruction and are not really that useful tactically. Introducing strategic-destabilizing elements to a conflict greatly increases its unpredictability and probably is a health risk for the leaders involved.
busyant 5y ago
I'm sure I'm not the first person to consider this, but ... RNA molecules can often be "evolved" in vitro to bind/inhibit target molecules with high specificity (e.g., https://en.wikipedia.org/wiki/Systematic_evolution_of_ligands_by_exponential_enrichment https://en.wikipedia.org/wiki/Systematic_evolution_of_ligand...) I imagine it would not be difficult to create RNAs that inhibit some essential human enzyme and then use the RNAs for targeted assassination. I mean, if you're doing an autopsy, you might run standard drug tests for poisons, but who's gonna screen for a highly specific RNA?
- foobarbecue 5y ago
  Have you seen the latest Bond movie?
  - busyant 5y ago
    No. Is that part of the plot? edit: just read the wiki for the latest Bond movie. Apparently, there is nothing new under the Sun. Thank you.
    - Computeiful 5y ago
- deleted 5y ago
  [deleted]
sjdegraeve 5y ago
.
- deleted 5y ago
  [deleted]
- ______-_-______ 5y ago
  You might be looking for this thread https://news.ycombinator.com/item?id=30699673 https://news.ycombinator.com/item?id=30699673
Barrera 5y ago
If this were really a practical concern, machine learning would be designing drugs that fly through the clinic today. They aren't and so this paper, though click-grabbing, is probably of no practical consequence. One reason is lack of data. Chemical data sets are extremely difficult to collect and as such tend to be siloed on creation. Synthesis of the target compounds and testing using uniform, validated protocols are non-trivial activities. They can only be undertaken by deep pockets. Those deep pockets are interested in return on investment. So, into the silo it goes. This might not always be the case, though. For now, the paper does raise the question of the goals and ethics around machine learning research. But unintended and/or malevolent consequences of new discoveries have been a problem for a long time. Just ask Shelley.
- philipkglass 5y ago
  A successful drug candidate must be useful in the treatment of human medical problems and not have harmful side effects that outweigh its benefits. A weaponized poison may have any number of harmful effects without diminishing its utility. A compound with really indiscriminate biochemical effects, like fluoroethyl fluoroacetate, makes a potent poison without any specific tuning for humans. It's much easier to discover compounds that genuinely harm people than those that genuinely help them.
xkcd-sucks 5y ago
The "dual use" paper this is commenting on is the clickbait equivalent of "encryption is for pedos", and maybe Derek's "not too surprising" is code for "Science editors are not discerning enough". Like, this is the whole point of pharmacology: Predicting the biological interactions of chemicals (what they do to biological targets, how potent), and their ancillary physical properties (solubility, volatility, stability etc). For example, Optimizing for mu-opioid agonist activity gives you super potent painkillers, drugs of abuse, and that stuff Russia gassed a theater with to knock out / kill hostages and kidnappers (i.e. fentanyl analogues) Optimizing for inhibition of various proteases might give you chemotherapy drugs with nasty side effects, or stuff with nasty side effects and no known therapeutic use (i.e. ricin) Optimizing for acetylcholinesterase inhibitor activity will turn up nasty poisons which could be purposed as "nerve agents" or "pesticides" Optimizing for 5HT2a activity will give compounds that are great for mapping receptor locations in brains, which are also drugs of abuse, and which are also lethal to humans in small doses. And the "predicted compounds not included in the training set" thing is just table stakes for any predictive model!
- Scoundreller 5y ago
  > 5HT2a You sure you don’t mean 5HT2b? I mean, anything can be toxic with enough dose, but the b-subtype agonists seem a lot more toxic than the a-subtype agonists. (Fun fact: 6-APB, a “research chemical” recreational substance became an actual research chemical because of it had better 5HT2b selectivity than what was previously used in lab research)
  - xkcd-sucks 5y ago
    Was thinking of halogenated NBOMe series - observed in humans to have a pretty narrow therapeutic index re: death, cheapish synth, can be vaporized But yeah 2b could be worse. Or many other targets as well Funny thing, optimizing "research chemicals" for (1) uncontrolled synthetic pathway and (2) potency is common to Institutional, Druggie, and Terrorist researchers. None of them want to go through the bureaucracy for controlled substances and potency is good for [better controlled experiments / smaller quantities to transport / more killing power]
teekert 5y ago
Interesting exercise, perhaps the harmful molecule generating AI still generates helpful molecules because molecules harmful at a certain dose may sometimes be very beneficial in a (much) lower dose. And the other way around of course. Perhaps we should simply have one “biologically active molecule” generating network. The dose will ultimately determine the toxicity.
- slivanes 5y ago
  I couldn't help but think of homeopathy with the above sentence.
  - teekert 5y ago
    Some snake venoms will stop your heart… but at a lower dose they will simply ease the heart and lower your blood pressure. For some examples: [0] [0]: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC6832721/ https://www.ncbi.nlm.nih.gov/pmc/articles/PMC6832721/
- l33t2328 5y ago
  > And the other way around of course. Whaaat? Are you saying there exist molecules which are very harmful in a (much) lower dose, but are beneficial at a higher dose? Do you have any examples?
  - teekert 5y ago
    I am saying that the network that comes up with “good” molecules will produce molecules that are very harmful as well, presumably at higher doses. I mean take some beta blockers (helpful molecules) at 100x normale dose: pretty harmful. Edit: Yeah my original comment didn’t come out right, I agree.
  - teekert 5y ago
    So, as I said my remark didn’t come out right, but, some molecules may be considered harmful at low dose and harmless at high dose if they stabilize a deteriorating conduction at or over some threshold concentration. Yeah I know it’s a fetch but you got me thinking… It’s not that clear cut. I mean the urine of someone on chemotherapy is pretty toxic, still we consider the molecules beneficial to the patient overall (the patient-tumor system if you will, not the patient by themselves).
alpineidyll3 5y ago
Hype over crap like this grinds my gears. Organophosphines like VX are ALL toxic. There's about a zillion such toxic molecules all containing the same functional group. This study does not demonstrate that this tool is better generator of toxic molecules than anything that includes the basic rules of valence and rudimentary understanding of shape similarity. When thinking about whether ML does something novel, we must always compare with some simple alternative. I would be impressed if it'd predicted something like Palytoxin, a highly specific molecule with extraordinary toxic activity. There's no way the tools of this paper would though. -- director of ML at a drug company.
verisimi 5y ago
Wow - I picked up on this earlier today, and even quoted the same as in this article. I was amazed that the scientists had not considered that AI could be/is being used for harm. (Was downvoted for this, but whatevs.) It struck me as incredibly naive, but then - what would someone else do in their situation? Most of us work in silos without awareness of how our work is used, and I suspect we are often causing (unintentional) harm to others whether we are scientists, programmers, in finance, in health, in government, etc. If we realise our predicament, there isn't an moral authority to make things right. There is only the legislation that was been written by lobbyists paid by the corporations we work for. Putting the article in broader context, perhaps it is about the creation of a moral framework for AI intended to pacify our disgust at the system we find. I expect that we will be expected to look away as AI "ethics" committees justify the unjustifiable, but call it ethical. As whatever-it-is is found to be ethical after all by ethical authorities, most of us we will wave this through and consider that we have acted judiciously. IMO.
MayeulC 5y ago
Providing chemical plants with models to estimate lethality of orders could be a great use case for this work.
empiricus 5y ago
Does this fall into the category of research "try not to make public"? Or is this category only wishful thinking on my part.
cryptonector 5y ago
The use of nuclear weapons would be... obvious: if an explosion in the 10Ktn or bigger happens, it's a nuclear weapon. There aren't enough nuclear powers to make the use of nuclear weapons plausibly deniable. The use of chemical weapons might not be as obvious if they are slow acting. And the production of chemical weapons is much easier than that of nuclear weapons. Though, the dispersion of chemical weapons is non-trivial. The use of biological weapons need not be obvious at all -- "it's a naturally-evolved pathogen, this happens!". The development and production of biological weapons is much easier than that of nuclear weapons. Human and animal bodies can be made to help spread biological weapons, so their dispersion can be trivial. The only thing that a bioweapons user might need ahead of time is treatment / vaccines, unless the bioweapon is weak and the real weapon is psychological. Sobering thoughts.
adultSwim 5y ago
I'm most worried about state actors.
tgtweak 5y ago
"Now, keep in mind that we can't deliberately design our way to drugs so easily, so we won't be able to design horrible compounds in one shot, either. " I would discount this, heavily and concerningly, as a false sense of security. The reality is that prohibitive factors in creating new drugs from compounds discovered similarly (by AI or other automated process) is almost entirely due to testing safety procedures and regulations... If the bad actors are trying to find the most lethal compound with no such oversight - and chances are very high that they aren't bound by any such regulation if they're state-level labs operating under impunity - there is nothing but the synthesis that would make the formulation and testing of these as impractical as the author claims. Take away the years-long, heavily scrutinized and regulated multi-stage billion-dollar path to drug approvals and you'll find that barrier is not so high. I would like to think this data could be helpful to any organizations looking to proactively develop detectors or antidotes for such compounds - especially if the threat was previously unknown to them. Let's say an entirely novel class of toxin was found in a cluster of these predictions that has no existing references in private or public records - it could be that another organization has discovered and synthesized something similar through one of many other paths. Many lines are drawn between this type of approach and that of whitehat hackers. You must necessarily create the vulnerability to mitigate it. It feels like "white hat" biolabs claiming the same are operating on the same conundrum and that the difference between "studying for the sake of mitigating" and "creating a weapon" are fundamentally indistinguishable without an absolute knowledge of intent - such is impossible from the outside.
- jcranmer 5y ago
  > The reality is that prohibitive factors in creating new drugs from compounds discovered similarly (by AI or other automated process) is almost entirely due to testing safety procedures and regulations Most drug candidates fail because they don't work, not because of any regulatory procedure. About 50% of drug candidates that enter Phase III trials--the final clinical trial before approval--fail, and that's almost always because they failed to meet clinical endpoints (i.e., they don't do what they're supposed to do), and not because they're not safe (toxicity is Phase I trials).
  - 5y ago
varajelle 5y ago
Part of my job is optimizing for ARM.
dsign 5y ago
So, the gist of this article is "oh look, machine learning can be used to build super-weapons!". Of course it can, and we have no shortage of (other) tech to make our life miserable. And of course the problem is not the tech itself but the people and their institutions. That paper from Urbina et. al. is at best academic click-bait. They are optimizing for publishing something and get cited, not for social good. They should stop.

445 points by herodotus 5y ago

> ricin is (fortunately) not all that easy to turn into a weapon, and the people who try to do it are generally somewhat disconnected from reality (and also from technical proficiency). Of course that fact was no barrier to much hype about the "dangers" it posed, either. I suspect the same now; that we have more to fear from the fear junkie propaganda than the actual facts.

once_inc 5y ago
I personally fear the lone-wolf attack drastically reducing in cost and effort. Where it would once be cost-prohibitive to design and manufacture your own nerve gas or lethal virus, these days with AI/ML and Crispr-cas and the like, it feels like any intelligent, deranged person wanting to take as many people to the grave with him has the tools to do just that.
- bitexploder 5y ago
  I think this is inevitable and something we will grapple with in coming decades. Especially around genetic engineering of viruses.
- WJW 5y ago
  Intelligent and deranged persons already have the tools to make way more casualties with way less effort using guns and/or explosives. The "problem" for them is that people who get sufficiently deranged to think that killing a few hundred (or even thousand) people will meaningfully solve the problem they are upset about will also be sufficiently deranged that their ability to reason coherently will be drastically reduced.
  - wpietri 5y ago
    Would they? I'm not seeing that as necessarily true. The Unabomber seems like a good example: https://en.wikipedia.org/wiki/Ted_Kaczynski https://en.wikipedia.org/wiki/Ted_Kaczynski Or look at mass shooting incidents: https://en.wikipedia.org/wiki/Mass_shootings_in_the_United_States https://en.wikipedia.org/wiki/Mass_shootings_in_the_United_S... The Las Vegas shooting was rationally planned and carried out. He managed to shoot nearly 500 people, killing 60. They did happen to pick conventional weapons. But is that because of rational choice, or just familiarity and availability? Imagine somebody like Kaczynski, but instead of being an award-winning young mathematician, he was an award-winning industrial chemist or genomics student.
    - uxp100 5y ago
      Kaczynski did not optimize for death, really following the lead of the shockingly common political bombing campaigns of the 70s. The Las Vegas shooter might be a better example.
      - wpietri 5y ago
        Sure, but I don't think that was a necessary outcome. Consider this quote: "I felt disgusted about what my uncontrolled sexual cravings had almost led me to do. And I felt humiliated, and I violently hated the psychiatrist. Just then there came a major turning point in my life. Like a Phoenix, I burst from the ashes of my despair to a glorious new hope." I agree he went with something common to the time. But I don't think that was a necessary outcome. After all, his approach didn't achieve his goals, so we can't say his sort of terrorism is any more rational than aiming for something bigger. Indeed, the nominal goals he ended up with, one could argue that mass-death terrorism is more rational.
    - Houshalter 5y ago
      None of these examples got away with it, or even acheived any of their stated goals. If terrorism actually worked, it would be a lot scarier. Also fortunately most political extremists don't want to kill just random people. Weapons of mass destruction don't discriminate, making them poor choices.

nonrandomstring 5y ago

Many years ago a colleague who works in defence told me about a job posting he'd seen but was having a moral struggle with. The opening was for "Lethality Engineer": Ideal candidate with good physics and medical background. I said that the main perk was that at least on Halloween he wouldn't need to buy a costume. He could just go out as himself. He didn't take the job.

stickfigure 5y ago
I hope recent events have illustrated that if it weren't for the people who develop lethal weapons, we (as in you and I) would be helpless against the bullies of the world. Unilateral pacifism is cute philosophy only when there are rough men standing ready to do violence on their behalf.
- MattGaiser 5y ago
  Especially since we cannot really help Ukraine with anything but tech and they are outnumbered, so the only advantage we can give is how much better our weapons are than Russian ones.
- marvin 5y ago
  I strongly agree with this sentiment. However, it is hard for an ethical person to participate in developing war technology when possession and usage of the weapons is purely a political question, and history also has seen our side of geopolitics commit atrocities. My stance has previously been that I am unwilling to work on weapons technology, because history has shown that these weapons sometimes end up being used for an indefensible cause. Then all of a sudden you're an accomplice to murder, and getting away with it. In the light of Russia's invasion of Ukraine, which is just a continuation of its historical imperialism, working on weapons is something I would be perfectly okay with and probably even motivated to do. But stop for a moment and think what a history of aggressive military actions does to our society's ability to recruit for this important job.
  - gonehome 5y ago

xycombinator 5y ago

Is anybody searching for compounds that reduce evil intent? Something that would mellow people out without causing hallucinations. A mass tranquilizer? Not effective against lone operatives but able to be deployed against an invading army.

rossdavidh 5y ago
I believe that's called a sedative. Most armies aren't filled with people with evil intent; they're filled with draftees who couldn't get out of it.
ansible 5y ago
Uh, I don't know about that... https://en.wikipedia.org/wiki/Serenity_(2005_film) https://en.wikipedia.org/wiki/Serenity_(2005_film) On a more serious note, anything that's going to affect behavior is going to have a dosage range. Too little absorbed, and there won't be enough effect. Too much, and that will harm / kill people in interesting ways. With chemical weapons, you only worry about the bio-accumulating enough to kill your enemies. An enemy receiving more than a lethal dose isn't a problem.
okasaki 5y ago
It's been suggested. eg. https://www.vice.com/en/article/akzyeb/link-between-lithium-in-drinking-water-suicide-study https://www.vice.com/en/article/akzyeb/link-between-lithium-... > The report states: “These findings, which are consistent with the finding in clinical trials that lithium reduces suicide and related behaviours in people with a mood disorder, suggest that naturally occurring lithium in drinking water may have the potential to reduce the risk of suicide and may possibly help in mood stabilisation, particularly in populations with relatively high suicide rates and geographical areas with a greater range of lithium concentration in the drinking water.”
jerf 5y ago
Of course they are. Among the "evil intent" they would reduce is any desire to rebel against your government, so you bet all big intelligence agencies are looking into it, for instance. Science fiction wrote about this decades ago. Fortunately, there's a lot of considerations involved in deployment of anything. It's easier said than done to get something of a medical nature into a population surreptitiously, because it's hard to get a certain dose into one person without someone else getting not enough and yet someone else getting way too much. You'd have to come up with a way of delivering a medical dose in a controlled fashion and lie about it or something, you couldn't just sneak it into the food/water reliably. Further, just because someone can name the exact complicated effect they'd like doesn't mean there's a drug that corresponds to it. Serenity, already mentioned, is a bit of silly example in my opinion because such a large effect should have been found during testing. But it does no good to pacify the population such that they'd never dream of so much as peacefully voting out the current leaders if the end result is that nobody would ever dream of so much as having enough ambition to show up to their jobs and you end up conquered by the next country over without them even trying, simply because they economically run circles around you. Or any number of other possible second-order effects. In a nutshell, it's dangerous to try to undercut evolution just to stay in power if not everywhere decides to do so equally, because you'll be evolved right out along with the society you putative rule. Evolution is alive and well and anyone who thinks it's asleep and they can screw around without consequences is liable to get a lethal wakeup call.

rossdavidh 5y ago

While it's worrying and worth thinking about, the track record of using AI to generate pharmaceuticals to do good has been "mixed", except really it's just been a bust. It may someday do great things, but not much yet, and one silver lining is that AI-generated toxins are unlikely to improve on the human-designed ones, either. "That is, I'm not sure that anyone needs to deploy a new compound in order to wreak havoc - they can save themselves a lot of trouble by just making Sarin or VX, God help us."

starwind 5y ago
> the track record of using AI to generate pharmaceuticals to do good has been "mixed", except really it's just been a bust. Researchers have only been using AI for drug development for like 6 years, I think it's way to early to call it a bust
- rossdavidh 5y ago
  I guess I should have said "...thus far".

ansible 5y ago

The article assumes that full development of a new chemical weapon would require more development effort. With regards to military usage: storable at room temperature, relatively easy to manufacture from commonly available precursor chemicals, etc. [1] How true is that? Are there components of this process that make things easier now? Where I have chemical structure X, and a system generates the process steps and chemicals needed to produce X. How much of the domain in chemistry / chemical engineering has been automated these days? What are the future prospects for this? [1] I assume one of the design goals for a new chemical weapon for military use is that it breaks down in the environment, but not too quickly (like say in a week or a month). Though I suppose if you want to just destroy civilization you would design for longevity in the environment instead. And being able to seep through many kinds of plastic if possible.

deleted 5y ago
[deleted]
logifail 5y ago
> Where I have chemical structure X, and a system generates the process steps and chemicals needed to produce X. Undergraduate chemistry students spend a fair amount of time learning how to look at a novel structure X and by disconnecting "backwards" it into simpler components, deduce a route by which it might be synthesed "forward" in the laboratory from readily available starting materials. There's an excellent book on this, "Organic Synthesis: The Disconnection Approach", by Stuart Warren.
- dredmorbius 5y ago
  Interesting. Were / are you a chem major? Any other major topics or readings you could recommend for someone wanting a general understanding of key concepts in modern chemistry? I'd suppose generally: materials, synthesis, o-chem, and chem-eng. My own background: began a hard-science degree. One year undergrad uni chem.
  - 323 5y ago
    The field is called "process chemistry". A very big thing in pharma: > Process chemists take compounds that were discovered by research chemists and turn them into commercial products. They “scale up” reactions by making larger and larger quantities, first for testing, then for commercial production. The goal of a process chemist is to develop synthetic routes that are safe, cost-effective, environmentally friendly, and efficient. https://www.acs.org/content/acs/en/careers/chemical-sciences/fields/process-chemistry.html https://www.acs.org/content/acs/en/careers/chemical-sciences...

toss1 5y ago

>>The thought had never previously struck us. We were vaguely aware of security concerns around work with pathogens or toxic chemicals, ...We have spent decades using computers and AI to improve human health—not to degrade it. We were naive in thinking about the potential misuse of our trade... Of course now, the next step is to use the technology to preemptively search for and develop antidotes to the new potential weapons their tool has discovered.

openasocket 5y ago

Fortunately we don't see any real work in the chemical and biological weapons space anymore. While it would still be pretty handy for terrorist groups, in actual warfare chemical weapons aren't super useful. See https://acoup.blog/2020/03/20/collections-why-dont-we-use-chemical-weapons-anymore/ https://acoup.blog/2020/03/20/collections-why-dont-we-use-ch... .

bsedlm 5y ago
that we don't see it doesn't mean it is not happening.
WinterMount223 5y ago
I guess developments are not published on Nature.
cogman10 5y ago
Just doesn't seem like it's worth it even for terrorists. Why invest a bunch of time and effort making more and more deadly poisons when we've already got a wide variety of them that are cheap to manufacture, well known in how they work, and don't cost a bunch of research money to uncover?
- openasocket 5y ago
  True, though you have to remember that threat is a social construct and isn't necessarily a rational measure. The 2001 anthrax attacks killed 5 people, injuring 17, and shocked the nation. As a direct result Congress put billions into funding for new vaccines and drugs and bio-terrorism preparedness. If 5 people were killed and 17 wounded in a mass shooting by a terrorist, would we really have reacted as strongly? If you wanted to install fear into a country, I think being attacked by some custom, previously unknown chemical weapon would scarier than sarin.
- at_a_remove 5y ago
  At the risk of putting myself on a watchlist, They already know that, so They have an eye on certain kinds of labware, different precursors, and such. And They already have antidotes to some of these poisons. One could optimize for compounds with hard-to-monitor precursors. Compounds that can be transported with low vapor pressure and volatility, so they cannot be easily sniffed out. Or imagine a lethal compound with a high delay factor. Or something with specifically panic-inducing effects, perhaps hemorrhagic with a side effect of your skin sliding off in great slick sheets. Another interesting high delay factor compound might induce psychosis: have fun tracking where these gibbering maniacs were a week ago. With a sufficiently dark imagination, "needs" could be identified for all sorts of compounds. Remember, the goal is to throw a monkey wrench into the gearwork of an opposing civilization, not necessarily to kill. Fear of the unknown is very effective for this.

ChrisMarshallNY 5y ago

We already (sort of) do this. AI/ML is probably used for simulating nuclear explosions, and is [arguably] even more useful and accurate than actually setting off a bomb, and measuring it. It makes sense that it could be weaponized. When Skynet becomes self-aware, it would probably just design a chemical that kills us all, and would aerosol that into the atmosphere. No need for terminators, just big pesticide cans.

wanda 5y ago
I'm quite sure we have already invented several chemicals that match your description -- like sarin gas, invented in 1938 by someone who, indeed, wanted to create a decent pesticide. A lethal dose of sarin gas is something like 28-35m³/min over 2 minutes exposure, according to wikipedia. [0] Hitler was well aware of its creation, and I believe quite a lot of the stuff was produced for the purpose of warfare. There were several in the Nazi military who wanted to use it, but Hitler declined. That seems rather odd, given his indifference to exterminating people with gas on an industrial scale beyond the theater of war. It has been suggested that Hitler was probably aware that to use sarin gas would be to invite the allies to do so in response, which would result in a dramatic loss of life on the German side due to the sheer lethality of such chemical weapons. [1] Perhaps he thought it easier to stick to conventional warfare, in which the pace is more manageable than with WMDs, where you would start going down the road of mutually assured destruction but without the strategic framework in place to prevent anyone from actually wiping out a population before realising how bad an idea it would be. And I think this reluctance to change the game, this seemingly deliberate moderation, perhaps best demonstrates the true difference between the machine and human in warfare. It is not a difference in innovation -- we have always been very good at inventing highly optimised ways to end life. The difference is that a machine intelligence will not hesitate. It will not ask for confirmation, pause or break a sweat. It will pull the trigger first, it will point the bombs at anything that is an adversary and anything that could theoretically be or become an adversary, and it will not miss. And it will not have to face ethical criticism and historical condemnation afterwards. [2] [0]: https://en.m.wikipedia.org/wiki/Sarin https://en.m.wikipedia.org/wiki/Sarin [1]: https://www.washingtonpost.com/news/retropolis/wp/2017/04/11/hitler-refused-to-use-sarin-gas-during-wwii-the-mystery-is-why/ https://www.washingtonpost.com/news/retropolis/wp/2017/04/11... [2]: Assuming this is a Skynet-like machine intelligence, which doesn't really have the capacity for remorse or negotiation and seems primarily, indeed solely occupied with the task of ending human life. Obviously, a true AI that is essentially a conscious mind equivalent to our own minds, may experience the same hesitancy that most of us would, were our fingers to be over the buzzer. Unless the AI independently arrives at a different set of values to us, like the Borg or something.

hengheng 5y ago

So, their tool will draw molecules that are good at doing harm, and that is it? No word on stabilization (which makes it safe to handle), synthesis, purification and such. I'd wager that most of these substances have at some point been on somebody's blackboard, but deemed impractical or infeasible, and then not pursued, and that's why we don't know them by name today. Still a scary lesson though.

ck2 5y ago

Why make a chemical weapon when you can just tweak a virus which self-replicates? BA.2 is even more infectious than BA.1 which is saying something, imagine an engineered BA.3 with even more spread and then make it as even more deadly. You might even be able to target it to one race or region if there is a gene specific to that area. Always hoped the future would be Star-Trek-like but it seems all it takes is one dictator or terrorist to end the world, slowly at first but then it would double every other day and impossible to stop.

danuker 5y ago
If you make it too deadly, maybe it doesn't spread as far (because the hosts die). Make it just the right amount of deadly!

photochemsyn 5y ago

This is essentially what the pesticide and herbicide industries have been doing since their inception, i.e. designing molecules that efficiently kill animals, insects and plants. It seemed like a miracle at first, but the long-term consequences of things like persistent chlorinated aromatics and their derivatives (Agent Orange and dioxin for example) eventually appeared in human populations. The development of the toxic nerve agents (organo-phosphate compouds mostly) in particular was a side effect of research into insect toxins. The nerve agents were discovered in this manner, they worked too well. Nevertheless, these pesticides were deemed safer than the organochlorines because they degraded fairly rapidly after application (although they are implicated in nerve damage related diseases like Parkinson's in agricultural areas). Insect infestations are indeed a big issue in agriculture and can wipe out entire crops if not dealt with, but there are plenty of options that don't require applications of highly toxic or persistent chemicals. Otherwise, this is just another of the many issues modern technology has created. Smallpox is another one - in the late 1990s, there was a great debate over whether to destroy the last smallpox samples - and then in the mid 2000's, someone demonstrated you could recreate smallpox by ordering the appropriate DNA sequences online and assembling them in a host cell. Then there's the past ten years of CRISPR and gain-of-function research with pathogenic viruses, a very contentious topic indeed, and still unresolved.

cryptofistMonk 5y ago

This is not really that worrying IMO - we already have weaponized toxins, viruses, and enough explosives to blow up the entire planet. So what if an AI can come up with something a little bit worse? It isn't the existence of these things that's stopping us all from killing each other.

taurusnoises 5y ago

Anyone wanna ELI5? It's useful for bother explainer and receiver. ;)

danuker 5y ago
Before, computers were used to make less poisonous chemicals. Now, the people asking computers to do that realized they can ask the computers to make more poisonous chemicals.
Traubenfuchs 5y ago
Software is able to simulate the effect of chemical compounds / molecules on the human body. This can be used to find drugs that do specific things, or stronger versions of existing drugs. For example, you could look for very strong but very short acting sleeping pills that immediately make you fall asleep, but cause zero grogginess the next day. Or you could optimize antibiotics to have a high half life, so you only have to take them once, instead of 3 times a day for a week, which you can easily forget. Now think about nerve gas. We have discovered lots of different nerve gas agents and know pretty well how much of each type you need to kill a human. Said software can be used to find new versions of nerve gas that kill with even lesser concentrations. You could also optimize for other variables: Nerve gas that remains on surfaces and doesn't decay by itself for example.
xondono 5y ago
They had an AI that looked for safe drugs by minimizing an estimate of lethality, changed it to ‘maximize’ and the computer spewed known nerve gas agents.

VLM 5y ago

This will probably come in handy for industrial espionage type tasks. Lets say you had a nation-state enemy who eats a lot of some ethnic ingredient. Come up with a cheap artificial flavor/color or process that is optimize to give heavy consumers cancer in 30 years. Not in one year, that will show up in the approval process. Then have an agent in the target country "discover" thru random chance this really excellent food dye or whatever. Now you kill half the population with cancer, you're gonna get nuked in response, even non-nuke countries will be pissed off enough to get nukes just to nuke the perpetrator. But lets say you make the victims fat and sick and die a little younger just enough to get 1% hit on economic growth... Some people would say this is how we ended up with trans-fats and margarine and vegetable oils in general or certain veg oils in specific. Certainly, corn syrup has caused more human and economic devastation that fission, nerve agents, or most any WMD I can think of...

0xdeadbeefbabe 5y ago
This is far more fun than believing in an emerging accident. You don't have to eat corn syrup, btw.
deleted 5y ago
[deleted]
tenebrisalietum 5y ago
The problem with this is that this is basically genetic engineering, you might successfully make a low-level economic growth impact now, but future generations will be resistant to the poisons as those weak against them die off. You are securing your own demise long-term if you don't subject your population to the same.
tgtweak 5y ago
I guess if it's tasty then it's fair game.

jleyank 5y ago

Chemical weapons above a certain level, bio weapons and nucs are all seen as weapons of mass destruction and are not really that useful tactically. Introducing strategic-destabilizing elements to a conflict greatly increases its unpredictability and probably is a health risk for the leaders involved.

busyant 5y ago

I'm sure I'm not the first person to consider this, but ... RNA molecules can often be "evolved" in vitro to bind/inhibit target molecules with high specificity (e.g., https://en.wikipedia.org/wiki/Systematic_evolution_of_ligands_by_exponential_enrichment https://en.wikipedia.org/wiki/Systematic_evolution_of_ligand...) I imagine it would not be difficult to create RNAs that inhibit some essential human enzyme and then use the RNAs for targeted assassination. I mean, if you're doing an autopsy, you might run standard drug tests for poisons, but who's gonna screen for a highly specific RNA?

foobarbecue 5y ago
Have you seen the latest Bond movie?
- busyant 5y ago
  No. Is that part of the plot? edit: just read the wiki for the latest Bond movie. Apparently, there is nothing new under the Sun. Thank you.
  - Computeiful 5y ago
deleted 5y ago
[deleted]

sjdegraeve 5y ago

deleted 5y ago
[deleted]
______-_-______ 5y ago
You might be looking for this thread https://news.ycombinator.com/item?id=30699673 https://news.ycombinator.com/item?id=30699673

Barrera 5y ago

If this were really a practical concern, machine learning would be designing drugs that fly through the clinic today. They aren't and so this paper, though click-grabbing, is probably of no practical consequence. One reason is lack of data. Chemical data sets are extremely difficult to collect and as such tend to be siloed on creation. Synthesis of the target compounds and testing using uniform, validated protocols are non-trivial activities. They can only be undertaken by deep pockets. Those deep pockets are interested in return on investment. So, into the silo it goes. This might not always be the case, though. For now, the paper does raise the question of the goals and ethics around machine learning research. But unintended and/or malevolent consequences of new discoveries have been a problem for a long time. Just ask Shelley.

philipkglass 5y ago
A successful drug candidate must be useful in the treatment of human medical problems and not have harmful side effects that outweigh its benefits. A weaponized poison may have any number of harmful effects without diminishing its utility. A compound with really indiscriminate biochemical effects, like fluoroethyl fluoroacetate, makes a potent poison without any specific tuning for humans. It's much easier to discover compounds that genuinely harm people than those that genuinely help them.

xkcd-sucks 5y ago

The "dual use" paper this is commenting on is the clickbait equivalent of "encryption is for pedos", and maybe Derek's "not too surprising" is code for "Science editors are not discerning enough". Like, this is the whole point of pharmacology: Predicting the biological interactions of chemicals (what they do to biological targets, how potent), and their ancillary physical properties (solubility, volatility, stability etc). For example, Optimizing for mu-opioid agonist activity gives you super potent painkillers, drugs of abuse, and that stuff Russia gassed a theater with to knock out / kill hostages and kidnappers (i.e. fentanyl analogues) Optimizing for inhibition of various proteases might give you chemotherapy drugs with nasty side effects, or stuff with nasty side effects and no known therapeutic use (i.e. ricin) Optimizing for acetylcholinesterase inhibitor activity will turn up nasty poisons which could be purposed as "nerve agents" or "pesticides" Optimizing for 5HT2a activity will give compounds that are great for mapping receptor locations in brains, which are also drugs of abuse, and which are also lethal to humans in small doses. And the "predicted compounds not included in the training set" thing is just table stakes for any predictive model!

Scoundreller 5y ago
> 5HT2a You sure you don’t mean 5HT2b? I mean, anything can be toxic with enough dose, but the b-subtype agonists seem a lot more toxic than the a-subtype agonists. (Fun fact: 6-APB, a “research chemical” recreational substance became an actual research chemical because of it had better 5HT2b selectivity than what was previously used in lab research)
- xkcd-sucks 5y ago
  Was thinking of halogenated NBOMe series - observed in humans to have a pretty narrow therapeutic index re: death, cheapish synth, can be vaporized But yeah 2b could be worse. Or many other targets as well Funny thing, optimizing "research chemicals" for (1) uncontrolled synthetic pathway and (2) potency is common to Institutional, Druggie, and Terrorist researchers. None of them want to go through the bureaucracy for controlled substances and potency is good for [better controlled experiments / smaller quantities to transport / more killing power]

teekert 5y ago

Interesting exercise, perhaps the harmful molecule generating AI still generates helpful molecules because molecules harmful at a certain dose may sometimes be very beneficial in a (much) lower dose. And the other way around of course. Perhaps we should simply have one “biologically active molecule” generating network. The dose will ultimately determine the toxicity.

slivanes 5y ago
I couldn't help but think of homeopathy with the above sentence.
- teekert 5y ago
  Some snake venoms will stop your heart… but at a lower dose they will simply ease the heart and lower your blood pressure. For some examples: [0] [0]: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC6832721/ https://www.ncbi.nlm.nih.gov/pmc/articles/PMC6832721/
l33t2328 5y ago
> And the other way around of course. Whaaat? Are you saying there exist molecules which are very harmful in a (much) lower dose, but are beneficial at a higher dose? Do you have any examples?
- teekert 5y ago
  I am saying that the network that comes up with “good” molecules will produce molecules that are very harmful as well, presumably at higher doses. I mean take some beta blockers (helpful molecules) at 100x normale dose: pretty harmful. Edit: Yeah my original comment didn’t come out right, I agree.
- teekert 5y ago
  So, as I said my remark didn’t come out right, but, some molecules may be considered harmful at low dose and harmless at high dose if they stabilize a deteriorating conduction at or over some threshold concentration. Yeah I know it’s a fetch but you got me thinking… It’s not that clear cut. I mean the urine of someone on chemotherapy is pretty toxic, still we consider the molecules beneficial to the patient overall (the patient-tumor system if you will, not the patient by themselves).

alpineidyll3 5y ago

Hype over crap like this grinds my gears. Organophosphines like VX are ALL toxic. There's about a zillion such toxic molecules all containing the same functional group. This study does not demonstrate that this tool is better generator of toxic molecules than anything that includes the basic rules of valence and rudimentary understanding of shape similarity. When thinking about whether ML does something novel, we must always compare with some simple alternative. I would be impressed if it'd predicted something like Palytoxin, a highly specific molecule with extraordinary toxic activity. There's no way the tools of this paper would though. -- director of ML at a drug company.

verisimi 5y ago

Wow - I picked up on this earlier today, and even quoted the same as in this article. I was amazed that the scientists had not considered that AI could be/is being used for harm. (Was downvoted for this, but whatevs.) It struck me as incredibly naive, but then - what would someone else do in their situation? Most of us work in silos without awareness of how our work is used, and I suspect we are often causing (unintentional) harm to others whether we are scientists, programmers, in finance, in health, in government, etc. If we realise our predicament, there isn't an moral authority to make things right. There is only the legislation that was been written by lobbyists paid by the corporations we work for. Putting the article in broader context, perhaps it is about the creation of a moral framework for AI intended to pacify our disgust at the system we find. I expect that we will be expected to look away as AI "ethics" committees justify the unjustifiable, but call it ethical. As whatever-it-is is found to be ethical after all by ethical authorities, most of us we will wave this through and consider that we have acted judiciously. IMO.

MayeulC 5y ago

Providing chemical plants with models to estimate lethality of orders could be a great use case for this work.

empiricus 5y ago

Does this fall into the category of research "try not to make public"? Or is this category only wishful thinking on my part.

cryptonector 5y ago

The use of nuclear weapons would be... obvious: if an explosion in the 10Ktn or bigger happens, it's a nuclear weapon. There aren't enough nuclear powers to make the use of nuclear weapons plausibly deniable. The use of chemical weapons might not be as obvious if they are slow acting. And the production of chemical weapons is much easier than that of nuclear weapons. Though, the dispersion of chemical weapons is non-trivial. The use of biological weapons need not be obvious at all -- "it's a naturally-evolved pathogen, this happens!". The development and production of biological weapons is much easier than that of nuclear weapons. Human and animal bodies can be made to help spread biological weapons, so their dispersion can be trivial. The only thing that a bioweapons user might need ahead of time is treatment / vaccines, unless the bioweapon is weak and the real weapon is psychological. Sobering thoughts.

adultSwim 5y ago

I'm most worried about state actors.

tgtweak 5y ago

"Now, keep in mind that we can't deliberately design our way to drugs so easily, so we won't be able to design horrible compounds in one shot, either. " I would discount this, heavily and concerningly, as a false sense of security. The reality is that prohibitive factors in creating new drugs from compounds discovered similarly (by AI or other automated process) is almost entirely due to testing safety procedures and regulations... If the bad actors are trying to find the most lethal compound with no such oversight - and chances are very high that they aren't bound by any such regulation if they're state-level labs operating under impunity - there is nothing but the synthesis that would make the formulation and testing of these as impractical as the author claims. Take away the years-long, heavily scrutinized and regulated multi-stage billion-dollar path to drug approvals and you'll find that barrier is not so high. I would like to think this data could be helpful to any organizations looking to proactively develop detectors or antidotes for such compounds - especially if the threat was previously unknown to them. Let's say an entirely novel class of toxin was found in a cluster of these predictions that has no existing references in private or public records - it could be that another organization has discovered and synthesized something similar through one of many other paths. Many lines are drawn between this type of approach and that of whitehat hackers. You must necessarily create the vulnerability to mitigate it. It feels like "white hat" biolabs claiming the same are operating on the same conundrum and that the difference between "studying for the sake of mitigating" and "creating a weapon" are fundamentally indistinguishable without an absolute knowledge of intent - such is impossible from the outside.

jcranmer 5y ago
> The reality is that prohibitive factors in creating new drugs from compounds discovered similarly (by AI or other automated process) is almost entirely due to testing safety procedures and regulations Most drug candidates fail because they don't work, not because of any regulatory procedure. About 50% of drug candidates that enter Phase III trials--the final clinical trial before approval--fail, and that's almost always because they failed to meet clinical endpoints (i.e., they don't do what they're supposed to do), and not because they're not safe (toxicity is Phase I trials).
- 5y ago

varajelle 5y ago

Part of my job is optimizing for ARM.

dsign 5y ago

So, the gist of this article is "oh look, machine learning can be used to build super-weapons!". Of course it can, and we have no shortage of (other) tech to make our life miserable. And of course the problem is not the tech itself but the people and their institutions. That paper from Urbina et. al. is at best academic click-bait. They are optimizing for publishing something and get cited, not for social good. They should stop.