4 ms·

You should check out Lane et al. 2013 Fig. 1: http://www.sciencedirect.com/science/article/pii/S0959440X12001820 http://www.sciencedirect.com/science/article/pi

by cing 13y ago

You should check out Lane et al. 2013 Fig. 1: http://www.sciencedirect.com/science/article/pii/S0959440X12001820 http://www.sciencedirect.com/science/article/pii/S0959440X12... All computational protein folding times match experimental results within 1 order of magnitude. If folding time isn't a good measure of forcefield accuracy then I don't know what is. Sampling error is a worse offender than forcefield accuracy in my opinion.

varelse 13y ago
To be fair, the Pande Lab's data has involved a degree of human intervention during the generation of the multi-state trajectories upon which they're based. And that's a failing that has only recently been addressed. IMO it's indeterminate whether that human intervention could have unwittingly influenced those trajectories. Similarly, a lot of the folded proteins were used to develop the same force fields now used to simulate their folding. I'm not dismissing this data, but I am saying I think the jury is still out on the models until we have a true test set/training set dichotomy, a separation that was an absolute revelation for ab initio structure prediction algorithms in the 1990s. That said, I think we both agree that undersampling is the biggest offender. And it only gets worse with results published for larger systems simulated at the same timescale as much smaller ones.
timr 13y ago
" If folding time isn't a good measure of forcefield accuracy then I don't know what is." Folding kinetics are surprisingly insensitive to detail. For example, it's been understood for a while that even ridiculously simple models can predict transition state energies for simple proteins: http://www.ncbi.nlm.nih.gov/pubmed/10322214?dopt=Abstract http://www.ncbi.nlm.nih.gov/pubmed/10322214?dopt=Abstract http://www.ncbi.nlm.nih.gov/pubmed/10500172?dopt=Abstract http://www.ncbi.nlm.nih.gov/pubmed/10500172?dopt=Abstract But to answer your question, I'd say that "predicting the correct structure of a protein" is the gold-standard benchmark of forcefield accuracy, and MD forcefields are really bad at it. (You could reasonably add other great measures, like: "does the simulation tend to fly apart without hacked-up pseudo-physical constraints?", but that feels like piling on.)
- cing 13y ago
  I'm biased since I do simulations but you're being too harsh on force fields. All the proteins in the paper I mentioned had their structure correctly predicted starting from an extended state. Correct structure and folding time! That's a serious accomplishment compared to the decade-old papers you're citing.
  - timr 13y ago
    "That's a serious accomplishment compared to the decade-old papers you're citing." What? It's not like the "decade-old" paper has become less correct over time. I mean, it's great that MD is finally close to predicting something that could be predicted a decade ago with much simpler methods, but that isn't saying a lot. Plus, as the other commenter pointed out, there's a complicated cross-validation problem that biases the results you're reporting. In truly blind tests (i.e. CASP), MD force fields just aren't that good at predicting structure from sequence. The usual counter-argument is that they aren't meant to work with non-MD methods (fair enough, I guess), but you don't have to try very hard to find reasons not to trust them. MD simulations have always been very finicky things, requiring lots of manual intervention to get "right".

by cing 13y ago

varelse 13y ago

To be fair, the Pande Lab's data has involved a degree of human intervention during the generation of the multi-state trajectories upon which they're based. And that's a failing that has only recently been addressed. IMO it's indeterminate whether that human intervention could have unwittingly influenced those trajectories. Similarly, a lot of the folded proteins were used to develop the same force fields now used to simulate their folding. I'm not dismissing this data, but I am saying I think the jury is still out on the models until we have a true test set/training set dichotomy, a separation that was an absolute revelation for ab initio structure prediction algorithms in the 1990s. That said, I think we both agree that undersampling is the biggest offender. And it only gets worse with results published for larger systems simulated at the same timescale as much smaller ones.

timr 13y ago

" If folding time isn't a good measure of forcefield accuracy then I don't know what is." Folding kinetics are surprisingly insensitive to detail. For example, it's been understood for a while that even ridiculously simple models can predict transition state energies for simple proteins: http://www.ncbi.nlm.nih.gov/pubmed/10322214?dopt=Abstract http://www.ncbi.nlm.nih.gov/pubmed/10322214?dopt=Abstract http://www.ncbi.nlm.nih.gov/pubmed/10500172?dopt=Abstract http://www.ncbi.nlm.nih.gov/pubmed/10500172?dopt=Abstract But to answer your question, I'd say that "predicting the correct structure of a protein" is the gold-standard benchmark of forcefield accuracy, and MD forcefields are really bad at it. (You could reasonably add other great measures, like: "does the simulation tend to fly apart without hacked-up pseudo-physical constraints?", but that feels like piling on.)

cing 13y ago
I'm biased since I do simulations but you're being too harsh on force fields. All the proteins in the paper I mentioned had their structure correctly predicted starting from an extended state. Correct structure and folding time! That's a serious accomplishment compared to the decade-old papers you're citing.
- timr 13y ago
  "That's a serious accomplishment compared to the decade-old papers you're citing." What? It's not like the "decade-old" paper has become less correct over time. I mean, it's great that MD is finally close to predicting something that could be predicted a decade ago with much simpler methods, but that isn't saying a lot. Plus, as the other commenter pointed out, there's a complicated cross-validation problem that biases the results you're reporting. In truly blind tests (i.e. CASP), MD force fields just aren't that good at predicting structure from sequence. The usual counter-argument is that they aren't meant to work with non-MD methods (fair enough, I guess), but you don't have to try very hard to find reasons not to trust them. MD simulations have always been very finicky things, requiring lots of manual intervention to get "right".