3 ms·

Having solved nearly all protein structures known to biology, As a biologist, “hahaha……no”. The AI might have solved structures according to the rules it’s be

by refurb 4y ago

Having solved nearly all protein structures known to biology, As a biologist, “hahaha……no”. The AI might have solved structures according to the rules it’s been told, but are they accurate with what happens in reality? No.

ipsum2 4y ago
Can you expand on the inconsistencies between what the model predicts and reality? I'd love to learn more and see some examples.
gilleain 4y ago
Do you mean examples like trans-membrane proteins where there are fewer experimental models, or inaccuracies in models compared to the structure in solution?
teshier-A 4y ago
Oh come on. "Protein structures" almost exclusively means "experimentally determined protein structures". That sentence is correct.
- refurb 4y ago
  Huh? How do you “solve” experimentally determined protein structures? The AI models were basically trained on those.
  - jojobas 4y ago
    You train on a subset and verify on another subset, validating your model.
    - refurb 4y ago
      And the results are? I'd love to see that analysis because I'd bet the accuracy rate isn't that great.
  - kragen 4y ago
    maybe by evaluating it on a test set not used for training?
soco 4y ago
Stupid layman question here: from what I've seen so far the AI generators imagine things starting from their training data. As in, don't find real things. Is the difference relevant in this field? Is the extrapolation and mashup they do enough to help? Is it trustworthy?
- refurb 4y ago
  That's pretty much it. Protein structures are immensely complex. Think of stringing a 50 magnets on a piece of string, then trying to predict the structure it would take on if in zero gravity? We have a pretty big set of protein structures that are known. AI is takes those known structures, plus a set of rules about how amino acids might bind when a part of a protein chain, then tries to predict conformation. If you ask AI to predict a known structure, it will do well, because, that's the training data. Ask it to predict a structure that isn't known, and it will do a good job get it's mostly right, but the small differences make all the difference. It's not a useless tool by any means, but the predictive power is still quite limited.
  - le-mark 4y ago
    > Think of stringing a 50 magnets on a piece of string, then trying to predict the structure it would take on if in zero gravity? It’s actually even more complex than that due to interaction with the local environment at the time of adding the magnets onto the string. Both the aqueous and transcription machinery.
    - refurb 4y ago
      I was trying to keep it simple, but yes. It's not only the protein composition itself, but also the environment. It's like trying to solve an equation with 1000's of unknown variables and only 200 rules on variable interactions that are correct 90% of the time. Even just modeling a 100% pure aqueous environment and the interactions of water molecules is a massively complex problem.
- jinto36 4y ago
  Alphafold is a big improvement, but a structure of a single protein in isolation isn't representative of how these things exist in vivo. Binding substrates can modify protein shapes, and proteins often function in complexes, which can form some pretty complex arrangements, where positioning is critical to function. I think training set bias is an issue to some extent, even with single-protein prediction. For example, I've been looking at a family of transcription factors, and most of the resolved crystal structures are of just the DNA-binding domain, crystallized with the substrate (DNA) bound. Alphafold predictions for homologous proteins that haven't been experimentally resolved but share a decent amount of sequence similarity thus have high confidence for the DNA-binding domain, but lower confidence in other parts of the protein, even if they're "ordered" regions (e.g. helices and sheets rather than floppy loops), and all the predictions for the DNA-binding domain look like the bound-to-DNA conformation. So we don't have a good way yet to predict different "modes" of a protein that has interaction-dependent conformations. Technically with Alphafold if you were interested in modelling a protein that had similar experimentally resolved both with and without substrates bound, but were interested in sampling just one of those states, you could customize your sequence database to include one or the other, which would be mostly manual curation. I've been testing out the multimer (protein complex) mode of Alphafold recently, to see if could predict interactions for a family of proteins where some members in the family are known to form complexes, but others previously were found to not form complexes at least when expressed in vitro rather than in vivo. So far I've found that if you try to throw two completely unrelated proteins together, they won't be modeled with any contacts, but for the ones in the family I'm interested in, there's always at least one (of the five models per run) that has them interacting such that there's something that looks like a real DNA-binding domain. For the latter case, it's presently hard to know based just on Alphafold output if it's a structure that could actually form, or if it's just due to bias in the training data, with perhaps the rest of the structured regions of the protein being conformed in unrealistic ways due to less training information for those parts. TL;DR Alphafold results are biased by existing experimentally resolved structures, and not based on simulating physics, so proteins- or parts of proteins- that don't have good coverage in existing experimental data are not going to be predicted with high confidence.
adamsmith143 4y ago
>As a biologist, “hahaha……no”. Learn to use the AI tools or get replaced by Gen-Z interns that can.
Hizonner 4y ago
> The AI might have solved structures according to the rules it’s been told, Wanna know how I know you don't know how that stuff works? You don't tell these things rules. At all. Nothing even slightly resembling that happens at any point in the process. That's kind of the whole point of machine learning. > but are they accurate with what happens in reality? For any given protein, AlphaFold usually predicts a structure very close to what you get for the isolated protein from crystallography or cryo-EM or whatever. It massively outperforms computational chemistry or any other "rules-based" system. Does it predict every conformation a protein might adopt in vivo? No. But those are not "protein structures known to biology", because guess where biology gets its "known" structures? It does predict the known structures. Yes, including ones that weren't in its training data.

by refurb 4y ago

ipsum2 4y ago

Can you expand on the inconsistencies between what the model predicts and reality? I'd love to learn more and see some examples.

gilleain 4y ago

Do you mean examples like trans-membrane proteins where there are fewer experimental models, or inaccuracies in models compared to the structure in solution?

teshier-A 4y ago

Oh come on. "Protein structures" almost exclusively means "experimentally determined protein structures". That sentence is correct.

refurb 4y ago
Huh? How do you “solve” experimentally determined protein structures? The AI models were basically trained on those.
- jojobas 4y ago
  You train on a subset and verify on another subset, validating your model.
  - refurb 4y ago
    And the results are? I'd love to see that analysis because I'd bet the accuracy rate isn't that great.
- kragen 4y ago
  maybe by evaluating it on a test set not used for training?

soco 4y ago

Stupid layman question here: from what I've seen so far the AI generators imagine things starting from their training data. As in, don't find real things. Is the difference relevant in this field? Is the extrapolation and mashup they do enough to help? Is it trustworthy?

refurb 4y ago
That's pretty much it. Protein structures are immensely complex. Think of stringing a 50 magnets on a piece of string, then trying to predict the structure it would take on if in zero gravity? We have a pretty big set of protein structures that are known. AI is takes those known structures, plus a set of rules about how amino acids might bind when a part of a protein chain, then tries to predict conformation. If you ask AI to predict a known structure, it will do well, because, that's the training data. Ask it to predict a structure that isn't known, and it will do a good job get it's mostly right, but the small differences make all the difference. It's not a useless tool by any means, but the predictive power is still quite limited.
- le-mark 4y ago
  > Think of stringing a 50 magnets on a piece of string, then trying to predict the structure it would take on if in zero gravity? It’s actually even more complex than that due to interaction with the local environment at the time of adding the magnets onto the string. Both the aqueous and transcription machinery.
  - refurb 4y ago
    I was trying to keep it simple, but yes. It's not only the protein composition itself, but also the environment. It's like trying to solve an equation with 1000's of unknown variables and only 200 rules on variable interactions that are correct 90% of the time. Even just modeling a 100% pure aqueous environment and the interactions of water molecules is a massively complex problem.
jinto36 4y ago
Alphafold is a big improvement, but a structure of a single protein in isolation isn't representative of how these things exist in vivo. Binding substrates can modify protein shapes, and proteins often function in complexes, which can form some pretty complex arrangements, where positioning is critical to function. I think training set bias is an issue to some extent, even with single-protein prediction. For example, I've been looking at a family of transcription factors, and most of the resolved crystal structures are of just the DNA-binding domain, crystallized with the substrate (DNA) bound. Alphafold predictions for homologous proteins that haven't been experimentally resolved but share a decent amount of sequence similarity thus have high confidence for the DNA-binding domain, but lower confidence in other parts of the protein, even if they're "ordered" regions (e.g. helices and sheets rather than floppy loops), and all the predictions for the DNA-binding domain look like the bound-to-DNA conformation. So we don't have a good way yet to predict different "modes" of a protein that has interaction-dependent conformations. Technically with Alphafold if you were interested in modelling a protein that had similar experimentally resolved both with and without substrates bound, but were interested in sampling just one of those states, you could customize your sequence database to include one or the other, which would be mostly manual curation. I've been testing out the multimer (protein complex) mode of Alphafold recently, to see if could predict interactions for a family of proteins where some members in the family are known to form complexes, but others previously were found to not form complexes at least when expressed in vitro rather than in vivo. So far I've found that if you try to throw two completely unrelated proteins together, they won't be modeled with any contacts, but for the ones in the family I'm interested in, there's always at least one (of the five models per run) that has them interacting such that there's something that looks like a real DNA-binding domain. For the latter case, it's presently hard to know based just on Alphafold output if it's a structure that could actually form, or if it's just due to bias in the training data, with perhaps the rest of the structured regions of the protein being conformed in unrealistic ways due to less training information for those parts. TL;DR Alphafold results are biased by existing experimentally resolved structures, and not based on simulating physics, so proteins- or parts of proteins- that don't have good coverage in existing experimental data are not going to be predicted with high confidence.

adamsmith143 4y ago

>As a biologist, “hahaha……no”. Learn to use the AI tools or get replaced by Gen-Z interns that can.

Hizonner 4y ago

> The AI might have solved structures according to the rules it’s been told, Wanna know how I know you don't know how that stuff works? You don't tell these things rules. At all. Nothing even slightly resembling that happens at any point in the process. That's kind of the whole point of machine learning. > but are they accurate with what happens in reality? For any given protein, AlphaFold usually predicts a structure very close to what you get for the isolated protein from crystallography or cryo-EM or whatever. It massively outperforms computational chemistry or any other "rules-based" system. Does it predict every conformation a protein might adopt in vivo? No. But those are not "protein structures known to biology", because guess where biology gets its "known" structures? It does predict the known structures. Yes, including ones that weren't in its training data.