5 ms·

Reading the description of the submission process is pretty interesting: http://predictioncenter.org/casp13/doc/CASP13_Abstracts.pdf http://predictioncenter.org

Reading the description of the submission process is pretty interesting: http://predictioncenter.org/casp13/doc/CASP13_Abstracts.pdf http://predictioncenter.org/casp13/doc/CASP13_Abstracts.pdf. There were multiple submissions and manual synthesis of different model results in some cases. I’m curious how this competition works exactly — it seems like a set of label predictions are submitted and some form of accuracy result feedback is provided (a single accuracy score for the whole prediction set?). And that there are a certain number of allowed submissions ...? How much of the ultimate strategy for playing this game at a high-level ends up being around optimizing for receiving as much leaked information from the test set as possible — is best guess at this point that this result is likely to be a good indicator of a true increase in prediction capability ...?

kxs 8y ago
The targets are completely unknown. They were experimentally solved, but not submitted to the Protein Data Bank. You basically get a target every day (meaning sequence of amino acids) and then depending on the category, you have three days to predict it (I think upto two weeks for "human" servers). In the modeling category they participated in, you can submit 5 models. A model is a fully predicted 3d structure of the protein. The targets are mostly independent. Some targets were split over a few days. But in principle, a target on day X has no connection to a target on day Y. The targets have varying difficulty. There are two categories: TBM (templated based) and FM (free modeling). You don't know which protein corresponds to which category, you can just guess by looking at the available template data. They focused on FM targets. Meaning there are no homologous available. It's hard to say how good of an indicator the results are. Looking at the contact prediction results, many methods are getting very good at constructing MSAs (gathering similar sequences). We already saw this at CASP12 - I think the FM targets are getting "easier" in that sense. There is basically zero feedback throughout the whole competition. Some targets are released after the deadline (because of publications), but in general, you don't know anything until the CASP meeting, which currently takes place. The competition ended in August.
YetAnotherNick 8y ago
There are multiple neural network models which looks like they are aiming to do the same thing. I wonder deepmind won just because of computation power and expertise in training the neural networks or they come up with much better and different model than other participants.

kxs 8y ago

The targets are completely unknown. They were experimentally solved, but not submitted to the Protein Data Bank. You basically get a target every day (meaning sequence of amino acids) and then depending on the category, you have three days to predict it (I think upto two weeks for "human" servers). In the modeling category they participated in, you can submit 5 models. A model is a fully predicted 3d structure of the protein. The targets are mostly independent. Some targets were split over a few days. But in principle, a target on day X has no connection to a target on day Y. The targets have varying difficulty. There are two categories: TBM (templated based) and FM (free modeling). You don't know which protein corresponds to which category, you can just guess by looking at the available template data. They focused on FM targets. Meaning there are no homologous available. It's hard to say how good of an indicator the results are. Looking at the contact prediction results, many methods are getting very good at constructing MSAs (gathering similar sequences). We already saw this at CASP12 - I think the FM targets are getting "easier" in that sense. There is basically zero feedback throughout the whole competition. Some targets are released after the deadline (because of publications), but in general, you don't know anything until the CASP meeting, which currently takes place. The competition ended in August.

YetAnotherNick 8y ago

There are multiple neural network models which looks like they are aiming to do the same thing. I wonder deepmind won just because of computation power and expertise in training the neural networks or they come up with much better and different model than other participants.