4 ms·

This is a big step forward, but the outstanding question as far as to whether or not this is useful for evaluating novel proteins, is going to be how good is th

This is a big step forward, but the outstanding question as far as to whether or not this is useful for evaluating novel proteins, is going to be how good is the confidence metric at telling the user to trust or not trust the results. You can see from their examples, that AlphaFold is very good but not perfect. I imagine for some proteins it will still give misleading or erroneous results and if you can’t tell when that happens without verifying the structure experimentally then this will likely not be that useful for new science.

asdfasgasdgasdg 6y ago
> the outstanding question as far as to whether or not this is useful for evaluating novel proteins That is not an outstanding question. The test on which DeepMind scored high marks is a test of how well the algorithm folds novel proteins -- proteins whose ground-truth structure has not yet been published.
- woeirua 6y ago
  We’d have to see the distribution of GDT scores evaluated on unknown proteins to say anything about how confident we can be. If the distribution is tightly distributed around the median then great, this works really well. If the variance is large though then you’re going to have a hard time using this for meaningful predictions.
  - foota 6y ago
    According to the article there's a confidence score as well. As long as this is sufficiently predictive of errors either a tight or wide distribution is likely acceptable.
    - woeirua 6y ago
      We need to see the relationship between confidence and GDT score. If you have a nice relationship then again everything is great. But... most confidence metrics from neural networks do not have a nice relationship to the primary metric.
- sundarurfriend 6y ago
  You missed the actual outstanding question in their comment: > the outstanding question ... is going to be how good is the confidence metric at telling the user to trust or not trust the results.
  - deeviant 6y ago
    You don't generally look at neural network output like that. There is generally a threshold, less than X, not the class, equal or more, is the class. Then you run the network with the same threshold on a known data set and compute a confusion matrix, which tells you about the error, I don't even want to know what a confusion matrix analogue for 3D geometry would look like but I'm sure they have something. This is literally the process that one does in taking part of the this. And the error rate (specifically the lack of errors) is what is everybody is talking about. 90 is just as accurate as we can get with experimental measurement. It's likely at this point the source of error is in the data set (we can only train on data we experimentally measure and these are not perfect measurements). It's also possible, at this point, the model generalized so well that when it deviates from experimental measurements it's actually correct and the experimental value was the one that was wrong. So no, the outstanding question is not "is going to be how good is the confidence metric at telling the user to trust or not trust the results.". Nobody is going to be looking confidence values when it model is giving an output, they are going to be looking at the overall error rate across a broad spectrum of proteins to get a sense of it's accuracy.
aardvarkr 6y ago
Every simulator is going to have error. In this case this biennial challenge represents the computational state of the art with scores of 30-40 over the last decade. The AlphaFold2 model sends that score up to 87 with errors about than the width of the atom. You can actually see the difference between their prediction and the actual result and it’s stunning. This is all on the blog site so I recommend reading before throwing shade.
- woeirua 6y ago
  I read the blog. But there’s a big difference between a mind blowing tech demo and something that can be used in a commercially viable process.
  - refulgentis 6y ago
    There's a difference between being a random commentator on HN and being one of the several experts in the field quoted in the article, among other things, predicting a mass exodus from the computational biology field as the major problem of that field is now solved.
  - blinding-streak 6y ago
    Are you always so dismissive of Nobel-level achievements?
mcshicks 6y ago
I was wondering the same thing. But I also wonder if having good guesses makes the x-ray crystallography and other experiments to verify a given protein easier/cheaper/quicker? I don't know enough about the actual techniques to have an informed opinion but I would think it would be helpful.
- sanxiyn 6y ago
  It does. https://www.nature.com/articles/d41586-020-03348-4 https://www.nature.com/articles/d41586-020-03348-4 reports a case of x-ray crystallography helped by AlphaFold prediction.
comicjk 6y ago
Scientists can verify that an AlphaFold-predicted structure is correct, or at least useful, without being able to get the structure experimentally. For instance, we could use the AlphaFold-predicted structure to do protein-ligand binding calculations for a bunch of known molecules. If these calculations agree with experimental protein-ligand binding (which they generally do for proteins with known structures), then we can say with high confidence that we've got a good structure.
- dontbeevil1992 6y ago
  does that mean that protein-folding is sort of in NP?
  - dekhn 6y ago
    The way computer scientists do it, yes, it is. In the CS situation you define an energy function (in this case representing the physical behavior of the protein in water) and find a heuristic to approximate the coordinates of the lowest energy configuration; done, problem solved. in reality, that's not how it works at all. The energy functions we have are crappy and require too much sampling before we can find the lowest energy configuration. And more importantly, it doesn't look like proteins typically fold to their lowest energy configuration (with the exception of some small fast two state folders), but rather explore a kinetically accessible region around there (or even somewhere else entirely, if the energy cost to transition is too high). Methods like AF depend heavily on large amount of information correlation from evolutionary data, which has historically been of the highest value for making decisions about protein structure.
  - hedora 6y ago
    It’s probably not in NP, in that there is not a polynomial time algorithm that checks solutions for correctness.
kxs 6y ago
There are other experimental methods that are much cheaper that can be used to assist validation. Also the models look damn impressive, even down to the sidechain packing.
theptip 6y ago
It’s a good question, and I’m not a domain expert here. The article did claim: > According to Professor Moult, a score of around 90 GDT is informally considered to be competitive with results obtained from experimental methods. So perhaps their score of 87 GDT is pretty significant. But “competitive with” is not the same as “always in agreement with”, as you point out. Could be the failure modes are problematic.

asdfasgasdgasdg 6y ago

> the outstanding question as far as to whether or not this is useful for evaluating novel proteins That is not an outstanding question. The test on which DeepMind scored high marks is a test of how well the algorithm folds novel proteins -- proteins whose ground-truth structure has not yet been published.

woeirua 6y ago
We’d have to see the distribution of GDT scores evaluated on unknown proteins to say anything about how confident we can be. If the distribution is tightly distributed around the median then great, this works really well. If the variance is large though then you’re going to have a hard time using this for meaningful predictions.
- foota 6y ago
  According to the article there's a confidence score as well. As long as this is sufficiently predictive of errors either a tight or wide distribution is likely acceptable.
  - woeirua 6y ago
    We need to see the relationship between confidence and GDT score. If you have a nice relationship then again everything is great. But... most confidence metrics from neural networks do not have a nice relationship to the primary metric.
sundarurfriend 6y ago
You missed the actual outstanding question in their comment: > the outstanding question ... is going to be how good is the confidence metric at telling the user to trust or not trust the results.
- deeviant 6y ago
  You don't generally look at neural network output like that. There is generally a threshold, less than X, not the class, equal or more, is the class. Then you run the network with the same threshold on a known data set and compute a confusion matrix, which tells you about the error, I don't even want to know what a confusion matrix analogue for 3D geometry would look like but I'm sure they have something. This is literally the process that one does in taking part of the this. And the error rate (specifically the lack of errors) is what is everybody is talking about. 90 is just as accurate as we can get with experimental measurement. It's likely at this point the source of error is in the data set (we can only train on data we experimentally measure and these are not perfect measurements). It's also possible, at this point, the model generalized so well that when it deviates from experimental measurements it's actually correct and the experimental value was the one that was wrong. So no, the outstanding question is not "is going to be how good is the confidence metric at telling the user to trust or not trust the results.". Nobody is going to be looking confidence values when it model is giving an output, they are going to be looking at the overall error rate across a broad spectrum of proteins to get a sense of it's accuracy.

aardvarkr 6y ago

Every simulator is going to have error. In this case this biennial challenge represents the computational state of the art with scores of 30-40 over the last decade. The AlphaFold2 model sends that score up to 87 with errors about than the width of the atom. You can actually see the difference between their prediction and the actual result and it’s stunning. This is all on the blog site so I recommend reading before throwing shade.

woeirua 6y ago
I read the blog. But there’s a big difference between a mind blowing tech demo and something that can be used in a commercially viable process.
- refulgentis 6y ago
  There's a difference between being a random commentator on HN and being one of the several experts in the field quoted in the article, among other things, predicting a mass exodus from the computational biology field as the major problem of that field is now solved.
- blinding-streak 6y ago
  Are you always so dismissive of Nobel-level achievements?

mcshicks 6y ago

I was wondering the same thing. But I also wonder if having good guesses makes the x-ray crystallography and other experiments to verify a given protein easier/cheaper/quicker? I don't know enough about the actual techniques to have an informed opinion but I would think it would be helpful.

sanxiyn 6y ago
It does. https://www.nature.com/articles/d41586-020-03348-4 https://www.nature.com/articles/d41586-020-03348-4 reports a case of x-ray crystallography helped by AlphaFold prediction.

comicjk 6y ago

Scientists can verify that an AlphaFold-predicted structure is correct, or at least useful, without being able to get the structure experimentally. For instance, we could use the AlphaFold-predicted structure to do protein-ligand binding calculations for a bunch of known molecules. If these calculations agree with experimental protein-ligand binding (which they generally do for proteins with known structures), then we can say with high confidence that we've got a good structure.

dontbeevil1992 6y ago
does that mean that protein-folding is sort of in NP?
- dekhn 6y ago
  The way computer scientists do it, yes, it is. In the CS situation you define an energy function (in this case representing the physical behavior of the protein in water) and find a heuristic to approximate the coordinates of the lowest energy configuration; done, problem solved. in reality, that's not how it works at all. The energy functions we have are crappy and require too much sampling before we can find the lowest energy configuration. And more importantly, it doesn't look like proteins typically fold to their lowest energy configuration (with the exception of some small fast two state folders), but rather explore a kinetically accessible region around there (or even somewhere else entirely, if the energy cost to transition is too high). Methods like AF depend heavily on large amount of information correlation from evolutionary data, which has historically been of the highest value for making decisions about protein structure.
- hedora 6y ago
  It’s probably not in NP, in that there is not a polynomial time algorithm that checks solutions for correctness.

kxs 6y ago

There are other experimental methods that are much cheaper that can be used to assist validation. Also the models look damn impressive, even down to the sidechain packing.

theptip 6y ago

It’s a good question, and I’m not a domain expert here. The article did claim: > According to Professor Moult, a score of around 90 GDT is informally considered to be competitive with results obtained from experimental methods. So perhaps their score of 87 GDT is pretty significant. But “competitive with” is not the same as “always in agreement with”, as you point out. Could be the failure modes are problematic.