6 ms·

Alphafold is interesting, but not a particularly big deal. It's just better homology modeling at the end of the day, and the difference between 60% and 90% is a

Alphafold is interesting, but not a particularly big deal. It's just better homology modeling at the end of the day, and the difference between 60% and 90% is a lot less critical than the difference between 90% and 100%. Those I have talked to in pharmaceuticals are entirely unwilling to stake billion dollar projects on it.

sillysaurusx 5y ago
"billion dollar projects" is telling. Work like this helps reduce the cost. I think you're underestimating the impact, and that in a decade or so it'll become clear that research like this laid the foundation for breakthroughs we then enjoy.
- COGlory 5y ago
  I think my problem with the AlphaFold narrative is that it's being treated as if it's something new, or that (as I see frequently) "protein folding is solved". As far as I can tell, our knowledge of protein folding hasn't increased at all. It's a black box of homology modeling and its standing on the shoulders of decades of fundamental wet bench biochemistry and simple wouldn't be possible without it. Google came along and tweaked something to make it significantly better, but still not good enough.
Rochus 5y ago
And it is trained for a bias towards crystal structures.
KarlKemp 5y ago
That ‘just’ is doing a lot of work in the sentence, considering protein structure was/is often named as the “holy grail” of computational biology, and homology was always the method to get there. I’m equally skeptical of the claim that this follows the 80/20 (or 90/10) rule. If you use straightforward measurements (rmsd etc), reality may well be 10 % off considering these aren’t static structures, but molecules in perpetual chaotic motion on several scales.
- COGlory 5y ago
  >considering protein structure was/is often named as the “holy grail” of computational biology, and homology was always the method to get there Homology is never considered "solving the protein folding problem". It's 100% based off of past data which incorporates all sorts of biases, including as others have pointed out, experimental error. For instance, in my field, where there is terrible amino acid sequence conservation, AlphaFold is just as poor at prediction as existing solutions like Phyre.
lmeyerov 5y ago
That doesn't line up with what we're seeing. Especially around supply chain, conversations have included the chemistry/materials data teams at some of the biggest companies. Likewise, academic researchers have steadily been switching over from past methods. GNNs here feels very much like imagenet for NNs for perception. It's hard to call that not a big deal. Many aren't switching, but not because they disbelieve it. For example, in precision medicine, a Stanford professor was mostly skeptical of practical relevance because of the general gap between real-world data findings and the multi-year clinical trial process... but that's not about GNNs, but the well-known issue of institutional resistance to real-world data. (We're biased: We're working with enterprise to bring in GNNs more on the supply chain, finance, fraud, security, etc. sides.)
- COGlory 5y ago
  I too am using AlphaFold when I can, and it is immensely valuable as a tool. The same way Phyre and ITASSER were immensely valuable. But it still suffers from the same categorical limitations that all homology modeling suffers from (although those limitations aren't reaching the public narrative, likely because they're technical details). It's a generational improvement in homology modeling, which is nice, but not particularly groundbreaking, and doesn't get us a whole lot closer to solving the big problems of drug design. It's still lousy when there's inadequate folding, it's lousy at oligomeric state (although that can probably be improved), and it's got tons of baked in, innate bias. It's also lousy at conformational states (but so is most empirical data). It's good enough for something that doesn't need to be high resolution, but so were preexisting methods. It's not good enough to predict a fold de novo.
  - lmeyerov 5y ago
    Most scientists fail to change a field so quickly like this, and so greatly improve the quality of results, and I've worked with Turing award winners who can't say the same. It's hard not to call that ground breaking. Ex: https://www.icr.ac.uk/blogs/the-drug-discoverer/page-details/reflecting-on-deepmind-s-alphafold-artificial-intelligence-success-what-s-the-real-significance-for-protein-folding-research-and-drug-discovery https://www.icr.ac.uk/blogs/the-drug-discoverer/page-details... """ Not surprisingly Lupas was impressed that AlphaFold enabled him to determine in half an hour the structure of a protein that he’d failed to solve for 10 years! """ From a computational / biology perspective, this is also important as part of the broader push to get end-to-end simulation & neural networks for areas beyond individual protein structures. Imagenet is a really good analogy: what started with a jump in mnist tasks is now at the level of deepfakes and multi-modal audiovisual inference. The leaders of the field defined the bar for what it means to do well in this area, and with alphafold doing well, better and composite solutions can now happen. (And more subtlely, they showed ~GNNs work in this area in general, which was not viable till recently.)
deeviant 5y ago
> and the difference between 60% and 90% is a lot less critical than the difference between 90% and 100%. Hard disagree. It's common knowledge, and was even mentioned in this fluff article, 90% accuracy is within the experimental error of the "truth set". It's entirely possible alphafold is more accurate then the X-ray crystallography source for a large portion of the data set. I would like to know what are you basing your very bold claim that 90 to 100% is bigger than 60 to 90%? Talking to random people in the pharmaceutical industry also seems like a very poor way to find truth. It's a very big field/industry with entrenched interests and very narrow fields of view. Also, roughly 100% of the companies who are asked, "are you about to be disrupted by new technology that you didn't create and is totally different then the way you do things?" answer, "Nope!".
- COGlory 5y ago
  >Hard disagree. It's common knowledge, and was even mentioned in this fluff article, 90% accuracy is within the experimental error of the "truth set". It's entirely possible alphafold is more accurate then the X-ray crystallography source for a large portion of the data set. Even if it was, we don't have a way to distinguish and so I'm not certain what extra good that does. >I would like to know what are you basing your very bold claim that 90 to 100% is bigger than 60 to 90%? Because 5-10 angstrom resolution isn't terribly hard to do either with traditional biochemistry, or with homology modeling, with very few exceptions here. The problem I have is that the confidence level there isn't good enough for the actual work that matters. Exact ligand interactions, etc.
  - deeviant 5y ago
    > Because 5-10 angstrom resolution isn't terribly hard to do either with traditional biochemistry, or with homology modeling, with very few exceptions here. The problem I have is that the confidence level there isn't good enough for the actual work that matters. Exact ligand interactions, etc. It really doesn't seem like you are well versed on this topic. I don't know any other way to put it. Before alphafold, the best modeling methods would hit ~75% accuracy, and x-ray crystallography is both prohibitively expensive and also specimen specific in regards to it's efficacy or even possibility. So what alphafold can do, "traditional biochemistry, or with homology modeling" can not touch and I don't know why you are claiming they can nor what you are basing that claim off of. I feel like I'm repeating myself and I wish I didn't have to, but, alphafold's 90% is 90% because that is the highest possible score with the level of error within the source material, not because we know there is 10% error left to "fix". > The problem I have is that the confidence level there isn't good enough for the actual work that matters. Yeah, that is actually how the world works. Confidence starts low then builds upon successful results. It's really not worth bringing up unless we start to see a string of failures from alphafold's prediction ability, which is the opposite of what we've seen from alphafold. 90% accuracy has to be good enough to be useful as it already has been pointed out, that's within the error range of x-ray crystallography , the current "Gold standard", so let's be clear here, any "crisis of confidence" here isn't "this isn't precise enough to work" but "I don't trust dem neural networks, they are going to take our jobs".
oneoff786 5y ago
I’m not a bio expert but I had the chance to ask several professors in the space what they thought. They felt similarly. It is impressive work but it’s not clear that it will be easily actionable into new applications. I wouldn’t discount it as progress toward something grander though.
rafaelero 5y ago
>Those I have talked to in pharmaceuticals are entirely unwilling to stake billion dollar projects on it. They will get crushed by their pessimism. It is definitely a big deal and you throwing "muh 90% is not enough" just points that ignorance is still widespread.
- COGlory 5y ago
  Could you enlighten me of my ignorance on the topic?
  - rafaelero 5y ago
    Simply because we can't know if the 10% error is because of measurement error of the method used to produce the labels. It is therefore only speculation that ~it is not good enough. It could very well be even better. For us to be sure, it is necessary to repeat some crystallographies and/or find a better method to produce these labels.
    - COGlory 5y ago
      >For us to be sure, it is necessary to repeat some crystallographies and/or find a better method to produce these labels. We have this information. And we've had it for decades. There's hundreds of duplicate structures on the PDB. There are new cryo-em structures that are nearing sub angstrom. Also, the xray data we do have is never considered absolute, but the boundaries and margin of error have been quite well understood for some time.
      - rafaelero 5y ago
        So you are saying that it underperforms even taking into account measurement error? Can you point to an expert agreeing with your position?
        COGlory 5y ago
        Yes, click on my bio. I'm a PhD structural biologist.
Gatsky 5y ago
Derek Lowe draws a similar conclusion: https://www.science.org/content/blog-post/more-protein-folding-progress---what-s-it-mean https://www.science.org/content/blog-post/more-protein-foldi... The real ‘breakthrough’ IMHO is that deep learning is so generally useful, rather than any specific application per se.
41b696ef1113 5y ago
I work in pharma, though not related to modelling. Completely agree with the stance. Protein modelling is one part of the enormous jigsaw that is drug development. Structural biology is already a computational savvy discipline that is accustomed to using available tooling/predictions to enable better decisions with spare/minimal data. Sometimes the predictions are good, sometimes bad, but they are always being reviewed by a human expert. Maybe alpha fold is significantly better than the competition -then it is likely to make solving structures faster. Hypothetical: the protein structures of all human proteins were solved tomorrow -is human disease cured? No, far from it, there is so much development work that goes beyond knowing the accessible pockets on a protein.
- mrjangles 5y ago
  Could you elaborate on this a little please? Let's say a computer program was created that could model any protein and any disease perfectly and you could, in an hour or two, take any virus, or disease or whatever, and design a drug that was guaranteed to disrupt it -- or do whatever it is medicines do (I don't know how it all works honestly). Given this, wouldn't it make creating drugs a breeze? Wouldn't that cut down billions on research and testing into drugs that turn out to be failures? If not, what other obstacles does pharma face in creating these drugs?
  - dflock 5y ago
    Doing this would (probably) create a drug which would work well in theory - and possibly in a test tube - but there's no guarantee that it would continue to work when put into the incredibly, vastly more complex environment of the human body.
  - steve76 5y ago
    The recent porcine kidney transplant, in which a hog did all the computations for you, was done on a patient in a coma. Vascular endothelial growth factor is being tested with CABG procedures. Find someone who needs a hail Mary, and do your best. That will always be faster. Genes are not the end all be all to medicine let alone biology. There are places where medicine won't go, such as the delicate structures of the brain. Much too unethical. Neurotransmitters fire across synapses. You will always be uncertain of what temperature or charge you need. No way of telling what's really on the inside. And that's with an incomplete model. You also can't just zap or plunge in medicine. If you are going gene by gene or protein by protein, yet alone cell by cell, how long is that going to take?
jeffbee 5y ago
How does pharma decide to stake billions on something? I'm thinking of Biogen and Aduhelm, but in total tens of billions have been spent investigating Alzheimer's treatments that don't work. How did they decide to pursue those dead ends?
- sjg007 5y ago
  Total addressable market and people willing to pay. But a lot of what they do is outsource the initial R&D. First, however you have to have a promising pre-clinical studies, then likely pass a Phase-1 trial. Lots of bio startups take a pre-clinical product through a Phase-1 trial. Traditionally, they would IPO to fund this. That game has changed a bit, where now some VCs will sponsor the Phase 1. There's also the whole SPAC thing as well. If Phase-1 is a success they usually get bought by a big biotech or pharma.

sillysaurusx 5y ago

"billion dollar projects" is telling. Work like this helps reduce the cost. I think you're underestimating the impact, and that in a decade or so it'll become clear that research like this laid the foundation for breakthroughs we then enjoy.

COGlory 5y ago
I think my problem with the AlphaFold narrative is that it's being treated as if it's something new, or that (as I see frequently) "protein folding is solved". As far as I can tell, our knowledge of protein folding hasn't increased at all. It's a black box of homology modeling and its standing on the shoulders of decades of fundamental wet bench biochemistry and simple wouldn't be possible without it. Google came along and tweaked something to make it significantly better, but still not good enough.

Rochus 5y ago

And it is trained for a bias towards crystal structures.

KarlKemp 5y ago

That ‘just’ is doing a lot of work in the sentence, considering protein structure was/is often named as the “holy grail” of computational biology, and homology was always the method to get there. I’m equally skeptical of the claim that this follows the 80/20 (or 90/10) rule. If you use straightforward measurements (rmsd etc), reality may well be 10 % off considering these aren’t static structures, but molecules in perpetual chaotic motion on several scales.

COGlory 5y ago
>considering protein structure was/is often named as the “holy grail” of computational biology, and homology was always the method to get there Homology is never considered "solving the protein folding problem". It's 100% based off of past data which incorporates all sorts of biases, including as others have pointed out, experimental error. For instance, in my field, where there is terrible amino acid sequence conservation, AlphaFold is just as poor at prediction as existing solutions like Phyre.

lmeyerov 5y ago

That doesn't line up with what we're seeing. Especially around supply chain, conversations have included the chemistry/materials data teams at some of the biggest companies. Likewise, academic researchers have steadily been switching over from past methods. GNNs here feels very much like imagenet for NNs for perception. It's hard to call that not a big deal. Many aren't switching, but not because they disbelieve it. For example, in precision medicine, a Stanford professor was mostly skeptical of practical relevance because of the general gap between real-world data findings and the multi-year clinical trial process... but that's not about GNNs, but the well-known issue of institutional resistance to real-world data. (We're biased: We're working with enterprise to bring in GNNs more on the supply chain, finance, fraud, security, etc. sides.)

COGlory 5y ago
I too am using AlphaFold when I can, and it is immensely valuable as a tool. The same way Phyre and ITASSER were immensely valuable. But it still suffers from the same categorical limitations that all homology modeling suffers from (although those limitations aren't reaching the public narrative, likely because they're technical details). It's a generational improvement in homology modeling, which is nice, but not particularly groundbreaking, and doesn't get us a whole lot closer to solving the big problems of drug design. It's still lousy when there's inadequate folding, it's lousy at oligomeric state (although that can probably be improved), and it's got tons of baked in, innate bias. It's also lousy at conformational states (but so is most empirical data). It's good enough for something that doesn't need to be high resolution, but so were preexisting methods. It's not good enough to predict a fold de novo.
- lmeyerov 5y ago
  Most scientists fail to change a field so quickly like this, and so greatly improve the quality of results, and I've worked with Turing award winners who can't say the same. It's hard not to call that ground breaking. Ex: https://www.icr.ac.uk/blogs/the-drug-discoverer/page-details/reflecting-on-deepmind-s-alphafold-artificial-intelligence-success-what-s-the-real-significance-for-protein-folding-research-and-drug-discovery https://www.icr.ac.uk/blogs/the-drug-discoverer/page-details... """ Not surprisingly Lupas was impressed that AlphaFold enabled him to determine in half an hour the structure of a protein that he’d failed to solve for 10 years! """ From a computational / biology perspective, this is also important as part of the broader push to get end-to-end simulation & neural networks for areas beyond individual protein structures. Imagenet is a really good analogy: what started with a jump in mnist tasks is now at the level of deepfakes and multi-modal audiovisual inference. The leaders of the field defined the bar for what it means to do well in this area, and with alphafold doing well, better and composite solutions can now happen. (And more subtlely, they showed ~GNNs work in this area in general, which was not viable till recently.)

deeviant 5y ago

> and the difference between 60% and 90% is a lot less critical than the difference between 90% and 100%. Hard disagree. It's common knowledge, and was even mentioned in this fluff article, 90% accuracy is within the experimental error of the "truth set". It's entirely possible alphafold is more accurate then the X-ray crystallography source for a large portion of the data set. I would like to know what are you basing your very bold claim that 90 to 100% is bigger than 60 to 90%? Talking to random people in the pharmaceutical industry also seems like a very poor way to find truth. It's a very big field/industry with entrenched interests and very narrow fields of view. Also, roughly 100% of the companies who are asked, "are you about to be disrupted by new technology that you didn't create and is totally different then the way you do things?" answer, "Nope!".

COGlory 5y ago
>Hard disagree. It's common knowledge, and was even mentioned in this fluff article, 90% accuracy is within the experimental error of the "truth set". It's entirely possible alphafold is more accurate then the X-ray crystallography source for a large portion of the data set. Even if it was, we don't have a way to distinguish and so I'm not certain what extra good that does. >I would like to know what are you basing your very bold claim that 90 to 100% is bigger than 60 to 90%? Because 5-10 angstrom resolution isn't terribly hard to do either with traditional biochemistry, or with homology modeling, with very few exceptions here. The problem I have is that the confidence level there isn't good enough for the actual work that matters. Exact ligand interactions, etc.
- deeviant 5y ago
  > Because 5-10 angstrom resolution isn't terribly hard to do either with traditional biochemistry, or with homology modeling, with very few exceptions here. The problem I have is that the confidence level there isn't good enough for the actual work that matters. Exact ligand interactions, etc. It really doesn't seem like you are well versed on this topic. I don't know any other way to put it. Before alphafold, the best modeling methods would hit ~75% accuracy, and x-ray crystallography is both prohibitively expensive and also specimen specific in regards to it's efficacy or even possibility. So what alphafold can do, "traditional biochemistry, or with homology modeling" can not touch and I don't know why you are claiming they can nor what you are basing that claim off of. I feel like I'm repeating myself and I wish I didn't have to, but, alphafold's 90% is 90% because that is the highest possible score with the level of error within the source material, not because we know there is 10% error left to "fix". > The problem I have is that the confidence level there isn't good enough for the actual work that matters. Yeah, that is actually how the world works. Confidence starts low then builds upon successful results. It's really not worth bringing up unless we start to see a string of failures from alphafold's prediction ability, which is the opposite of what we've seen from alphafold. 90% accuracy has to be good enough to be useful as it already has been pointed out, that's within the error range of x-ray crystallography , the current "Gold standard", so let's be clear here, any "crisis of confidence" here isn't "this isn't precise enough to work" but "I don't trust dem neural networks, they are going to take our jobs".

oneoff786 5y ago

I’m not a bio expert but I had the chance to ask several professors in the space what they thought. They felt similarly. It is impressive work but it’s not clear that it will be easily actionable into new applications. I wouldn’t discount it as progress toward something grander though.

rafaelero 5y ago

>Those I have talked to in pharmaceuticals are entirely unwilling to stake billion dollar projects on it. They will get crushed by their pessimism. It is definitely a big deal and you throwing "muh 90% is not enough" just points that ignorance is still widespread.

COGlory 5y ago
Could you enlighten me of my ignorance on the topic?
- rafaelero 5y ago
  Simply because we can't know if the 10% error is because of measurement error of the method used to produce the labels. It is therefore only speculation that ~it is not good enough. It could very well be even better. For us to be sure, it is necessary to repeat some crystallographies and/or find a better method to produce these labels.
  - COGlory 5y ago
    >For us to be sure, it is necessary to repeat some crystallographies and/or find a better method to produce these labels. We have this information. And we've had it for decades. There's hundreds of duplicate structures on the PDB. There are new cryo-em structures that are nearing sub angstrom. Also, the xray data we do have is never considered absolute, but the boundaries and margin of error have been quite well understood for some time.
    - rafaelero 5y ago
      So you are saying that it underperforms even taking into account measurement error? Can you point to an expert agreeing with your position?
      - COGlory 5y ago
        Yes, click on my bio. I'm a PhD structural biologist.

Gatsky 5y ago

Derek Lowe draws a similar conclusion: https://www.science.org/content/blog-post/more-protein-folding-progress---what-s-it-mean https://www.science.org/content/blog-post/more-protein-foldi... The real ‘breakthrough’ IMHO is that deep learning is so generally useful, rather than any specific application per se.

41b696ef1113 5y ago

I work in pharma, though not related to modelling. Completely agree with the stance. Protein modelling is one part of the enormous jigsaw that is drug development. Structural biology is already a computational savvy discipline that is accustomed to using available tooling/predictions to enable better decisions with spare/minimal data. Sometimes the predictions are good, sometimes bad, but they are always being reviewed by a human expert. Maybe alpha fold is significantly better than the competition -then it is likely to make solving structures faster. Hypothetical: the protein structures of all human proteins were solved tomorrow -is human disease cured? No, far from it, there is so much development work that goes beyond knowing the accessible pockets on a protein.

mrjangles 5y ago
Could you elaborate on this a little please? Let's say a computer program was created that could model any protein and any disease perfectly and you could, in an hour or two, take any virus, or disease or whatever, and design a drug that was guaranteed to disrupt it -- or do whatever it is medicines do (I don't know how it all works honestly). Given this, wouldn't it make creating drugs a breeze? Wouldn't that cut down billions on research and testing into drugs that turn out to be failures? If not, what other obstacles does pharma face in creating these drugs?
- dflock 5y ago
  Doing this would (probably) create a drug which would work well in theory - and possibly in a test tube - but there's no guarantee that it would continue to work when put into the incredibly, vastly more complex environment of the human body.
- steve76 5y ago
  The recent porcine kidney transplant, in which a hog did all the computations for you, was done on a patient in a coma. Vascular endothelial growth factor is being tested with CABG procedures. Find someone who needs a hail Mary, and do your best. That will always be faster. Genes are not the end all be all to medicine let alone biology. There are places where medicine won't go, such as the delicate structures of the brain. Much too unethical. Neurotransmitters fire across synapses. You will always be uncertain of what temperature or charge you need. No way of telling what's really on the inside. And that's with an incomplete model. You also can't just zap or plunge in medicine. If you are going gene by gene or protein by protein, yet alone cell by cell, how long is that going to take?

jeffbee 5y ago

How does pharma decide to stake billions on something? I'm thinking of Biogen and Aduhelm, but in total tens of billions have been spent investigating Alzheimer's treatments that don't work. How did they decide to pursue those dead ends?

sjg007 5y ago
Total addressable market and people willing to pay. But a lot of what they do is outsource the initial R&D. First, however you have to have a promising pre-clinical studies, then likely pass a Phase-1 trial. Lots of bio startups take a pre-clinical product through a Phase-1 trial. Traditionally, they would IPO to fund this. That game has changed a bit, where now some VCs will sponsor the Phase 1. There's also the whole SPAC thing as well. If Phase-1 is a success they usually get bought by a big biotech or pharma.