5 ms·

I’m not a specialist in the biology domain, but more generally, for some models 66% accuracy is amazing. For example, a model predicting the next word with 66%

by willj 6y ago

I’m not a specialist in the biology domain, but more generally, for some models 66% accuracy is amazing. For example, a model predicting the next word with 66% accuracy.

kylegill 6y ago
My machine learning professor would always remind us that 50% is analogous to random chance in a binary decision, but in a model where the prediction is not so black and white, 66% doesn't sound half bad!
- LeifCarrotson 6y ago
  It seems like this protein folding problem is something like the binary decision "guesses the winning lotto numbers" vs. "does not guess the winning lotto numbers". If it got answer right 1 time in 100, that would be amazing and you'd be foolish not to use it!
  - gph 6y ago
    >If it got answer right 1 time in 100, that would be amazing and you'd be foolish not to use it! Except you have no way of knowing if the answer it gives you is one of the 66 right predictions vs one of the 33 wrong predictions. You could say it's likely correct, but not to a high enough degree of confidence that you could really trust it without verifying using the old established techniques.
    - LeifCarrotson 6y ago
      You missed my point - there are not 33 but billions (an infinite number, really) of wrong protein structures. There is only one right structure, not 66. Also important - verifying that a given model has a signature that matches the established techniques is far easier than using those techniques to generate the complete model from scratch.
      - gph 6y ago
        The 33 vs 66 is in reference to the percentage chance that a prediction is correct. But if you have no way to tell if a given prediction is correct or not without doing the tests that you were trying to avoid in the first place then it's not really worthwhile except for perhaps some exploratory research. I'm not really sure what your point is.
        nl 6y ago
        > But if you have no way to tell if a given prediction is correct or not without doing the tests that you were trying to avoid in the first place then it's not really worthwhile except for perhaps some exploratory research. This isn't really how it works. To quote the CASP competition organisers: The organizers even worried DeepMind may have been cheating somehow. So Lupas set a special challenge: a membrane protein from a species of archaea, an ancient group of microbes. For 10 years, his research team tried every trick in the book to get an x-ray crystal structure of the protein. “We couldn’t solve it.” But AlphaFold had no trouble. It returned a detailed image of a three-part protein with two long helical arms in the middle. The model enabled Lupas and his colleagues to make sense of their x-ray data; within half an hour, they had fit their experimental results to AlphaFold’s predicted structure. “It’s almost perfect,” Lupas says. “They could not possibly have cheated on this. I don’t know how they do it.”[1] So you have experimental results, but still don't know how it folds. You aren't trying to avoid the all the experiments, just understand them. [1] https://www.sciencemag.org/news/2020/11/game-has-changed-ai-triumphs-solving-protein-structures https://www.sciencemag.org/news/2020/11/game-has-changed-ai-...
- stjohnswarts 6y ago
  I guess but wouldn't that only be useful if there were only two possiblities. If the next "thing" in a sequence is from thousands, millions, or quintillions of possibilities then 67% is hellatiously better than 1 in $really_huge_number
vannevar 6y ago
Exactly. I might not buy a satnav that was 66% accurate, but I would definitely buy a lottery number predictor with that accuracy. I don't know whether protein folding is closer to the former or the latter, but you can't make a blanket statement without considering the context.
visarga 6y ago
It's not strictly 60% or 92%, it depends on the threshold of what is considered "good enough", 0.3Å or 1.6Å, which depends on what you want to use the prediction for. When you set a threshold you improve precision at the detriment of recall. It's a tradeoff you can play with, but the score depends on it.

by willj 6y ago

I’m not a specialist in the biology domain, but more generally, for some models 66% accuracy is amazing. For example, a model predicting the next word with 66% accuracy.

kylegill 6y ago

My machine learning professor would always remind us that 50% is analogous to random chance in a binary decision, but in a model where the prediction is not so black and white, 66% doesn't sound half bad!

LeifCarrotson 6y ago
It seems like this protein folding problem is something like the binary decision "guesses the winning lotto numbers" vs. "does not guess the winning lotto numbers". If it got answer right 1 time in 100, that would be amazing and you'd be foolish not to use it!
- gph 6y ago
  >If it got answer right 1 time in 100, that would be amazing and you'd be foolish not to use it! Except you have no way of knowing if the answer it gives you is one of the 66 right predictions vs one of the 33 wrong predictions. You could say it's likely correct, but not to a high enough degree of confidence that you could really trust it without verifying using the old established techniques.
  - LeifCarrotson 6y ago
    You missed my point - there are not 33 but billions (an infinite number, really) of wrong protein structures. There is only one right structure, not 66. Also important - verifying that a given model has a signature that matches the established techniques is far easier than using those techniques to generate the complete model from scratch.
    - gph 6y ago
      The 33 vs 66 is in reference to the percentage chance that a prediction is correct. But if you have no way to tell if a given prediction is correct or not without doing the tests that you were trying to avoid in the first place then it's not really worthwhile except for perhaps some exploratory research. I'm not really sure what your point is.
      - nl 6y ago
        > But if you have no way to tell if a given prediction is correct or not without doing the tests that you were trying to avoid in the first place then it's not really worthwhile except for perhaps some exploratory research. This isn't really how it works. To quote the CASP competition organisers: The organizers even worried DeepMind may have been cheating somehow. So Lupas set a special challenge: a membrane protein from a species of archaea, an ancient group of microbes. For 10 years, his research team tried every trick in the book to get an x-ray crystal structure of the protein. “We couldn’t solve it.” But AlphaFold had no trouble. It returned a detailed image of a three-part protein with two long helical arms in the middle. The model enabled Lupas and his colleagues to make sense of their x-ray data; within half an hour, they had fit their experimental results to AlphaFold’s predicted structure. “It’s almost perfect,” Lupas says. “They could not possibly have cheated on this. I don’t know how they do it.”[1] So you have experimental results, but still don't know how it folds. You aren't trying to avoid the all the experiments, just understand them. [1] https://www.sciencemag.org/news/2020/11/game-has-changed-ai-triumphs-solving-protein-structures https://www.sciencemag.org/news/2020/11/game-has-changed-ai-...
stjohnswarts 6y ago
I guess but wouldn't that only be useful if there were only two possiblities. If the next "thing" in a sequence is from thousands, millions, or quintillions of possibilities then 67% is hellatiously better than 1 in $really_huge_number

vannevar 6y ago

Exactly. I might not buy a satnav that was 66% accurate, but I would definitely buy a lottery number predictor with that accuracy. I don't know whether protein folding is closer to the former or the latter, but you can't make a blanket statement without considering the context.

visarga 6y ago

It's not strictly 60% or 92%, it depends on the threshold of what is considered "good enough", 0.3Å or 1.6Å, which depends on what you want to use the prediction for. When you set a threshold you improve precision at the detriment of recall. It's a tradeoff you can play with, but the score depends on it.