12 ms·
AI in drug discovery is overhyped: examples from AstraZeneca, Harvard, Stanford
- chris_va 9y agoAs someone who has worked on AI for drug discovery, I would say that the title is correct, but not for the reasons stated. There is also some annoying speculation in this document that is completely incorrect, and unsupported, so I caution people when reading it. Anyway, the primary reason that AI for drug discover is overhyped is that the sort of problems AI is good at solving don't line up well with the unsolved problems in the drug discovery pipeline. This article, for example, focuses a lot on lead generation. Lead generation is the easiest aspect of the problem to tackle using AI, and so most people doing research start out trying to build a foundation in this space. However, it doesn't actually represent the majority of the cost. Drug makers typically spend about ~$800M on failed drugs for every ~$900M in revenue. They aren't spending that $800M on leads, finding leads is fairly easy. They are spending that money on drugs that fail in Phase 2 and Phase 3, which is more about off-target side effects, bulk formulation and synthesis, patient population differences, drug-drug interactions, etc. It would be nice having better leads, but there aren't a shortage of them that look good in vitro or even in vivo. It isn't until much later in the pipeline that the costs really add up, and failures there are expensive. If we could solve off-target side effects using AI, then we'd be in a whole different ballgame. Having banged my head against it for a while, I think it is possible, but will take a huge amount of investment. The work this article talks about is more foundational, which is necessary but should not really be taken as anything more.
- lowglow 9y ago> If we could solve off-target side effects using AI, then we'd be in a whole different ballgame. Having banged my head against it for a while, I think it is possible, but will take a huge amount of investment. Can you talk more about what you've been thinking about here?
- cing 9y agoNot sure what OP is thinking, but you can look at this example of a commercial product designed for the prediction of off-target effects (https://cyclicarx.com/ligandexpress/ https://cyclicarx.com/ligandexpress/).
- chris_va 9y agoThat route is doubly tough: 1) SAAS in the pharma world is mostly a waste of time. Culturally, they don't want to pay for anything except drugs. There is also a culture of sunk cost, where they do not want to prune drugs from their pipeline based on what some piece of software says. 2) This is a boil the ocean approach, which does not work statistically. There are 20,000 targets. Predicting bioavailability at each target is very difficult, and different populations have different expression patterns. Even if you have 99% precision/recall for each one, odds that you can help with selection enrichment are infinitesimal. Even if you restrict it to a handful of targets with strong known side effects, the state of the art predications are still not good enough to meaningfully improve the outcomes. There are better approaches, nothing easy though.
- Eridrus 9y agoI'm not an expert in this space, but lead generation doesn't seem like a fundamentally bad place to do this - if you could accurately rank leads this would deal with the failed drugs - it just hasn't achieved the results people would like.
- aaavl2821 9y agothat is a great point. for those not familiar with where the money really goes in R&D, and where the biggest opportunities for improvement are, check out these charts [1] and [2] from the seminal paper on calculating the cost of getting a drug approved [3]. the biggest areas to improve R&D productivity lie in 1) picking more validated targets to reduce Phase 2 and 3 failure rate and 2) reducing the cost of lead optimization (basically the process of turning a compound that has the desired impact on the target (target is a molecule implicated in disease that you want to effect with a drug) into a molecule with drug like properties (ie gets where you need it in the body, is safe, can be manufactured and delivered efficiently)) im not an AI person, but could AI play a role here? there are plenty of validated targets that are "undruggable"; could AI help find as-yet-undiscovered molecules that could engage these targets? or could AI somehow make the med chem / lead optimization process easier? [1] https://media.nature.com/m685/nature-assets/nrd/journal/v9/n3/images/nrd3078-f2.jpg https://media.nature.com/m685/nature-assets/nrd/journal/v9/n... [2] https://media.nature.com/m685/nature-assets/nrd/journal/v9/n3/images/nrd3078-f3.jpg https://media.nature.com/m685/nature-assets/nrd/journal/v9/n... [3] https://www.nature.com/articles/nrd3078 https://www.nature.com/articles/nrd3078
- randcraw 9y agoThe primary missions in drug development are efficacy and safety. AI can help answer clear well-formed efficacy questions like, "Does this molecule fit a chosen target molecule"? But it can't help with bigger efficacies like, "Will hitting this target make a sufficient difference in managing this disease"? Or any safety questions like, "Does the molecule also hit any other molecules somewhere in the body (or population) that might screw up something else"? And unfortunately it's safety failures that eat up 90% of drug development costs (esp. in Phase III). Until Wall St (or Sand Hill Rd) understands that domain-agnostic low-info approaches like AI are incapable of answering complex questions that require teams of PhDs steeped in decades of doing both chemistry and biology, the notion of CADD will continue to miss the mark and waste megabucks.
- msamwald 9y agoWell, as far as I know the state-of-the-art results on current drug toxicity prediction benchmarks (e.g., Tox21) are held by deep neural networks; it seems like recent AI advances HAVE proven useful in that regard.
- digitalzombie 9y agoDo you have a source for that? I'm super interested into reading more. I had to do Tox21 I was using Bayesian Network for drug toxicity at the FDA we used the DILI database.
- deleted 9y ago[deleted]
- _untom_ 9y agoThis is the paper from the winners of the Tox21 Challenge: https://www.frontiersin.org/articles/10.3389/fenvs.2015.0008.. https://www.frontiersin.org/articles/10.3389/fenvs.2015.0008.... (fun fact: one of the authors is the inventor of the LSTM)
- qaq 9y ago"~$800M on failed drugs for every ~$900M in revenue" Do you have a source for this? Looking at big Pharma the marketing budget is at least 2X the R&D budget so the numbers really look off from what you are posting?
- qaq 9y agoLooking at Pfizer last q. for example revenue 53 bil. R&D 7.5 bil
- chris_va 9y agoMost of the actual R&D is done by smaller companies, and then the large companies (e.g. Pfizer) buy up the compounds in Phase ~2. They do have some of their own development, but it isn't the majority for most players. They'll primarily take drugs through Phase 3, then deal with synthesis, distribution, and marketing. It is hard to look at a single company to see how the money is spent on failed drugs.
- ouid 9y agocalling any existing technology "AI" is overhyped
- theophrastus 9y agoI don't know about "any existing technology" but in any aspect of drug discovery with which i'm aware, i'd agree. My job amounts to using statistics and linear programming to winnow drug candidates out of databases of hundreds of millions of small molecules, (sometimes: "chemoinformatics"). And i've lost count of how many times i've had the marketing department insist on calling what i do "AI". (and let me assure you, as far as pushing back against marketing is concerned "all resistance is futile")
- mkagenius 9y agototally random thought: do you have pictures of molecules by any chance - can be a good source to train some CNN on it and existing drugs. Disclaimer: I have zero experience in drug discovery.
- theophrastus 9y agoI don't know how it can help train whatever 'CNN' stands for (i'm guessing not the news network, but something Neural Network?) Yet there are vast sources of various ("pictures") representations of small molecules all over the internet; from line-drawings, 2d stick models[1] to those providing rotation to space filling models[2]. As a very personal aside, neural networks are just a means of doing curve fitting without necessary learning anything. [1] https://en.wikipedia.org/wiki/Morphine https://en.wikipedia.org/wiki/Morphine [2] https://www.cancerquest.org/patients/drug-reference/morphine https://www.cancerquest.org/patients/drug-reference/morphine
- AmericanSeal 9y agoIt stands for convolutional neural network, which is a NN architecture that works well on image data. They use CNN's in object detection and other computer vision related tasks.
- JacobiX 9y agoThe article mention the name of a new startup too many times to be convincing.
- mostafab 9y agoSorry for this inconvenience. Content without ads is rarely free.
- anon1253 9y ago1) As someone who does NLP and AI for Evidence Based medicine [1] I can only agree. I build natural language processing tools, together with the team, for EBM. I have /no/ idea what to do next. Please. Help me. 2) Rant: here's the thing, we have SVMs, LSTM, BiLSTM-CRF, Knowledge Graphs, 28M articles, deploy infrastructure, tests, you know all the cool stuff. We can't sell it. It's PubMed this, PubMed that. There's no breaking that spell. So instead of building a better search engine, we focused on better analytics. Is it cool, sure. Is it as good as all the KoL stuff from LexusNexis, nope. Medicine is /hard/, if you think your app is hard: medicine is /harder/. Am I trying to cover my ass, sure. But seriously, deal with everything being non-standard and proprietary for long enough and you'll feel the same way. 80% of my job is saying "no" to people who aren't going to pay in the first place (Sanofi, Novartis, etc). They don't care. I'm just going to ride this out, throw some blockchain in somewhere where it makes absolutely 0% sense, call it a day. Seriously, NLP/AI for medicine: it's crap 100%. And, I'm a "senior developer" with publications behind my ass on this. I don't know what to do instead of riding the hype train. Seriously: anyone out there that knows what to do with 100M+ medical publications in a 100M+ knowledge graph, be my guest: I'd hire you in a heart beat. But it's hard and there is nothing to be gained AFAIK. [1]: http://growthevidence.com/joel-kuiper/ http://growthevidence.com/joel-kuiper/
- epmaybe 9y agoYou probably already know, but there are teams with the national library of medicine that work specifically on natural language processing within the scope of pubmed, but also on new ways to get to the publications. I almost interned there for a summer, very fascinating stuff.
- anon1253 9y agoyep, we're trying to outcompete them. But it's a hard thing. MetaMap: we beat it, cTakes, we can do it faster. But really how many < 80ms concept recognition systems do you need that work on 28M documents. And if you have that: what are you going to do with it. We have, internally, systems that predict dense vectors based on CUI graphs of concepts in text. Great for similarity search: but utterly useless in the end.
- brndnmtthws 9y agoAnyone who has actually built software using what the masses think is 'artificial intelligence' knows exactly how artificial and overhyped ML and AI are. Neural networks are a far cry from the magic they've been made out to be, and are nowhere close to as sophisticated as the brains of most animals. Cherry picking results makes for good marketing material, but that's pretty much where the usefulness ends. We still don't even understand how exactly brains work.
- tensor 9y agoNeural networks really are pretty magical for a lot of image based problems these days. The impact for text problems is so far decidedly not magical, though there are starting to be some tantalizing results in my opinion.
- brndnmtthws 9y agoMore gimmick than magic. At best they can do a crappy job of emulating specific human behaviours on carefully curated training datasets.
- oh-kumudo 9y ago> At best they can do a crappy job of emulating specific human behaviours ...With better performance than human, that is pretty magical.
- brndnmtthws 9y agoIn certain cases, perhaps. But that's the exception, not the rule. You can't simply apply a neural net to every problem and magically get great results.
- infinite8s 9y agoWhat about Alpha Go Zeros approach of reinforced learning based on self play? Arguably the game of go has really simple, explicit rules, and evaluating the final end state is also easy, but it shows that with enough compute NNs can identify important features without supervision or labeling.
- jcoffland 9y agoAI in X is overhyped. For all X.
- toddwprice 9y agoWas jumping on to say exactly that. We are most likely somewhere close to the "Peak of Inflated Expectations" for the current iteration of AI. It's useful, just not a magic bullet to solve all problems. https://en.wikipedia.org/wiki/Hype_cycle#/media/File:Gartner_Hype_Cycle.svg https://en.wikipedia.org/wiki/Hype_cycle#/media/File:Gartner...
- deleted 9y ago[deleted]
- jostmey 9y agoFor the most part, I agree with the article's premise. But they were pretty unfair to the Stanford lab (Vijay Pande). The Stanford team is using graph convolutions to process molecular structures. The article complains that the Stanford team is biased because they are not using sequences to represent the molecular structures. But this actually makes a lot of sense because molecular structures are graphs. The article then goes on to say that the Stanford team has an agenda to get everyone to use deepchem instead of TensorFlow or PyTorch. But TensorFlow is a dependency in deepchem, and deepchem looks like a bunch of useful tools wrapped around TensorFlow. I'm not buying the articles arguments
- blueblob 9y agoAgreed. Just because you don't compare your model to another model doesn't mean you're hiding something and it's irresponsible to pretend that's the case. Perhaps the author should have run char-cnn against data instead of making baseless accusations. On the whole, a lot of the arguments are realistic of ML in general. There is far too much focus on individual problems and the accuracy measure without enough focus on the interpretation of the accuracy, how well it will really generalize, etc.
- mostafab 9y agoIn general, you are right. But in this particular case, char-CNN is the standard used by many people, Stanford included. It is not sophisticated at all. Look at the big list of exotic models they tried: http://moleculenet.ai/models http://moleculenet.ai/models For me, this omission is a negligence. It is selective laziness. The author (me) does not have 20 PhD students, postdocs and startuppers under his hand, to do the job for Stanford. With my limited resources, it is more cost-effective to do my job against Stanford ;)
- dekhn 9y agoIt sounds like you have axe to grind against well-funded professors, not that you actually have a valid argument.
- Afforess 9y agoAI can't help much in drug discovery (i.e lead generation) because the pharamecutical industry is suffering from Eroom's law ( https://en.wikipedia.org/wiki/Eroom%27s_law https://en.wikipedia.org/wiki/Eroom%27s_law ). It's a play off Moore's law (spelled backwards), but the fundamental problem is the pharma/medical industries do not understand biology from first principles, unlike hard physical sciences, such as computer hardware engineering. First principles understanding of computing hardware allows chip manufacturers to create novel, new chips from scratch, as we fully understand the physics behind electricity, how logic gates work and signals propagate through physical mediums. Lock most college students in a room with resistors, diodes, wire, etc, and reference material, and they could recreate basic circuitry and very simple computers. You can not do the same with biology - no student or expert, given unlimited resources, equipment, or reference material can construct a new living cell from scratch (proteins / molecules). This is not a slander of those sciences and scientists, but there is simply a huge gap in how well we understand the basic principles of life and how well we understand the basic principles of computing. Throwing more resources / computing into "drug discovery" is like trying to build chips by wiring computer parts differently. Occasionally it might work and produce a "useful" result, but it's fundamentally a broken approach.
- efangs 9y agoYES. Exactly this. It's fine to make approximations to avoid exponential scaling, but applying function approximators essentially randomly won't get you anywhere. This is then compounded by the fact that the functional framework you're starting from is not a first-principles approach. Until there is QMC for drug discovery, it will all be hype.
- randcraw 9y agoMostly hype. Yes, automating drug discovery to any extent is utterly hopeless, and as likely to impact the pharma business any time as autonomous killer robots overrunning the battlefield -- Not In My Lifetime. But AI definitely has a near term future in addressing well formed questions like specific assays or searching for well-constrained targets, like ligand matches. The trick is for the AI contributor TO LEARN SOMETHING ABOUT THE DAMNED DOMAIN. Unless the chemist/biologist is intimately involved in the task, the AI provider is shooting blind. But with many wise eyes on the ball, even the hardest problems becomes a lot more assailable. [I say this as someone who processes images and analyzes data within a big pharma, and has seen several grand IT plans fail (like systems biology disease modeling) and many small & specific scientist-assistance tasks succeed.]
- lilleswing 9y agoDeepChem developer and MoleculeNet co-author here. We are NOT trying to get vendor lock in for users of DeepChem. If there are complaints about lock in using our tools please file a github issue and we can work on trying to create an open easy to use API for everyone.
- mostafab 9y agoFrom a user viewpoint, Deepchem would greatly benefit from being a better team player with lower-level (Tensorflow) or other (Pytorch) frameworks. The pace of research (in NLP in particular) is too fast to make it realistic to port everything in Deepchem without an unreasonable delay. Does it fit the Deepchem agenda? That's another question ;)
- lilleswing 9y agoThanks for the feedback (I love feedback). While that is a good high level goal the devil is in the details of how to make it happen. Here are some doable ideas in the medium term which might help. 1) Tutorial and documentation on how to access the raw Tensorflow graph when using DeepChem 2) Tutorial of combining raw Tensorflow with our existing chemistry specific layers 3) Different documentation quick-start sections for ML practitioners and application practitioners. 4) Better overall documentation of our Chemistry Specific layers. Would these ideas have made a better first user experience for you?
- mostafab 9y agoYes, especially 2)
- lurr 9y agoAI is overhyped is probably an equally valid title.
- frisco 9y agoResponding to his issues with Vijay Pande's work (I'm not affiliated), graph convolutions really are better than char-RNN for this. There's a good theoretical motivation for why, and people have spent a lot of time trying to find better embeddings of chemical space. (Admittedly, still not great - I wouldn't dispute the overall thesis that AI in drug discovery is still very early.) I did a project a year or so ago to reimplement one of Aspuru-Guzik's papers, a variational autoencoder, (https://github.com/maxhodak/keras-molecules https://github.com/maxhodak/keras-molecules) and when I did that I compared to char-RNN and the VAE did get much more interesting results. I also saw results from other people around that time showing that using graph convolutions on the front end instead of one-hot encoded SMILES strings was even better. Also, as for OP's objection about GANs not working because it's a "perfect discriminator," this is an obvious result that becomes apparent after spending 20 seconds with the problem. (Eg, this thread here: https://github.com/maxhodak/keras-molecules/issues/55 https://github.com/maxhodak/keras-molecules/issues/55) I haven't read the Harvard paper referenced but I'd be absolutely shocked if this was lost on them. There are definitely ways to work through it.
- mostafab 9y agoI am not sure to fully understand your remark, but if you have a benchmark graph convolutions vs. char-CNN, it would be great to write your result and post it on Arxiv. Pande will be interested ;) The problem is not with the theoretical motivation, but with the empirical confirmation. I also agree that there are many ways to work through the perfect discriminator problem for ORGAN. But it remains to be done (afaik).
- maxander 9y agoThe fundamental problem with AI (here meaning, neural-network based applications) applied to biology is that pattern recognition doesn't get you very far. When designing a small molecule drug, for example, the goal is to find a molecule that will slot into some protein (or other biomolecule) of interest in just the right way; whether it does so is dependent on shape in complex and impossible-to-predict ways. As an analogy, take the task of making a key (drug) to open the front door of an apartment across town (disease-relevant target); your "training data" (known drug-y molecules) are the keys for a bunch of other apartments sampled mostly randomly from around town. Obviously, you can make any number of plausible key-oid objects that would pass visual inspection as potential keys to the target, but even a godly-intelligent post-Bostromian superAI couldn't solve the problem in any principled way. You just need information about the actual lock you want to open, plain and simple. There are many, many, many other open problems in biology, of course, ranging all along the conceptual gamut, so I can't say there's nothing where AI would be a killer app. And there's also the sort of last-ditch null argument that, well, being smart helps humans do biology, so computers that are smart must be useful somehow. But typical progress in this field is from patiently acquiring new data. There's no point where you know "enough" of the picture to infer the rest with any degree of certainty- the degrees of freedom in the "design" of biological systems is simply too huge.
- steve_musk 9y agoI’m not sure this is a great analogy. If we didn’t know anything about the locks (target binding sites) I would agree with you, but there is no reason to enforce that restriction. A pipeline that given a target binding could predict a number of potential drug structures with high accuracy would be extremely useful. Going with your analogy, this would be like training a net using a database of lock structures and their corresponding keys, and then having it predict a key for a given lock. That seems fairly doable for current ML techniques. Obviously actual drug design is more complicated than this, and you have to consider side effects. But I don’t think it’s as hopeless as your analogy.
- maxander 9y ago> Going with your analogy, this would be like training a net using a database of lock structures and their corresponding keys, and then having it predict a key for a given lock. If we have that, then the problem is entirely trivial- just look at the lengths of the tumblers, and give the key teeth that match. And so it is, with the corresponding increase in complexity, in biology- given a complete and confident structure of the target, we have (non-AI, reasonably reliable) chemical modelling methods to see whether and how a given compound will bind; iterate molecules until you find one that works and you've found your "key" quite cheaply. So, there's another limitation to the practicality of AI- when it's not irreducibly complex and intrinsically unknown, biology is usually mechanistic and deterministic enough that conventional approaches can reason about it quite effectively without resorting to anything so fancy as a neural network. Biology is almost always more complicated than a given description, so yes, the real problem is more complicated. As you mention, the key we create must not only fit the target lock, but it must also not fit any other locks that might cause problems. Perhaps AI methods could try to make a working key that is un-key-like as possible, to make it unlikely to fit off-target locks; potentially useful. But, absent knowledge of the various perverse configurations that locks in the wider world can have, the confidence the AI method's inferences are very limited- roughly speaking, it's attempting to make a classifier for a population that is mostly unknown. We might hope for a severalfold increase in the number of successful drug candidates, in the best case, but not a fundamental disruption of the pharmaceutical research process.
- aaavl2821 9y agoThis comment is related to AI in healthcare services, not drug discovery specifically, but is illustrative. I recently went to a talk by a sr exec at one of the biggest health systems in the country. They were evaluating AI tools to predict which patients would die in 18 months so they could design the most patient friendly and cost effective end of life plan. None of the algorithms they evaluated outperformed simply asking a doctor which patients she thought would die in the next year
- deleted 9y ago[deleted]
- YeGoblynQueenne 9y agoOf course, "these days" (scine ~2014 by my reckoning) AI is synonymous with Deep Learning (in the lay press anyway) but there are actually other techniques still being researched, specifically theory- and knowledge-driven techniques, from the field of symbolic machine learning. Such techniques have long been used in medicine and biology with great success, very notably in the example of the Robot Scientist, a system that automates the scientific process end-to-end, in a biology context. This is from the abstract of the Nature paper, from January 2004 [1]: The question of whether it is possible to automate the scientific process is of both great theoretical interest1,2 and increasing practical importance because, in many scientific areas, data are being generated much faster than they can be effectively analysed. We describe a physically implemented robotic system that applies techniques from artificial intelligence3,4,5,6,7,8 to carry out cycles of scientific experimentation. The system automatically originates hypotheses to explain observations, devises experiments to test these hypotheses, physically runs the experiments using a laboratory robot, interprets the results to falsify hypotheses inconsistent with the data, and then repeats the cycle. Here we apply the system to the determination of gene function using deletion mutants of yeast (Saccharomyces cerevisiae) and auxotrophic growth experiments9. We built and tested a detailed logical model (involving genes, proteins and metabolites) of the aromatic amino acid synthesis pathway. In biological experiments that automatically reconstruct parts of this model, we show that an intelligent experiment selection strategy is competitive with human performance and significantly outperforms, with a cost decrease of 3-fold and 100-fold (respectively), both cheapest and random-experiment selection. There's more info and especially links in the Wikipedia article [2]. _____________ [1] Functional genomic hypothesis generation and experimentation by a robot scientist, https://www.nature.com/articles/nature02236 https://www.nature.com/articles/nature02236 [2] https://en.wikipedia.org/wiki/Robot_Scientist https://en.wikipedia.org/wiki/Robot_Scientist
- deleted 9y ago[deleted]
- isoprophlex 9y agoI'm not qualified to discuss drug discovery, but my eye was caught by the authors statement that char-RNN on SMILES strings might outperform the use of a coulomb matrix. IMO this is really misguided as a SMILES string does not contain specific geometrical data (bond angles, dihedrals etc) of a compound... Learning the grammar of smiles-representation is not the same as learning to use inter atomic distances.
- dekhn 9y agoSMILES implicitly stores the geometry data. It's technically true it doesn't encode a unique 3D structure (for a rigid molecule), but you could convert the implict bonds to standard-length ones, embed the molecule in 3D space, and minimize it (that's precisely what SMILES to 3D structure systems do).
- jaytyagi 9y agoCulprit? Journalists.