6 ms·
Sometimes announcements like this are a bit over-the-top. But what really, to me, cements the 'big-deal' of this is the "Median Free-Modelling Accuracy" graph h
by mabbo 6y ago
Sometimes announcements like this are a bit over-the-top. But what really, to me, cements the 'big-deal' of this is the "Median Free-Modelling Accuracy" graph half way down the page.
Scores of 30-45 for 15 years. Now scores of 87-92.
This isn't a minor improvement, it's a leap forward.
- entropicdrifter 6y agoNot to mention the fact that two years ago they took it from 45% to >60%. If they can continue improving, even with an exponential decay in rate of improvement, this is certainly a stunning example of technological disruption.
- Zenst 6y agoEven without any improvement, the amount of grunt-work the AI can pre-do and get down to a short-list - that in itself will see changes in progress speeding research up.
- kordlessagain 6y ago> and get down to a short-list There's no reason to believe the list will contain all solutions, however.
- patagurbon 6y agoNo but it will hopefully contain some. Which for many if not most problems is all that matters
- treis 6y agoThat is an impressive improvement, but I think you've missed the most important point: >a score of around 90 GDT is informally considered to be competitive with results obtained from experimental methods So DeepMind is to the point where it's a question of whether their generated model or the experimentally determined structure is closest to the actual physical structure.
- asdfasgasdgasdg 6y agoI have a related question about this. If experimental methods produce results around a score of 90, what is the baseline we are comparing the DeepMind results against? If the experimental error is equal to the observed DeepMind error, how can we say which one is actually more erroneous?
- 0-_-0 6y agoThat's a damn good question, it looks like we don't know how much above 90 AlphaFold is.
- deleted 6y ago[deleted]
- mrDmrTmrJ 6y agoExcellent question. At somepoint, I think the only answer is, "have a bunch of different people run a bunch of experiments on the same protein." The threshold for "real" in particle physics is +5 sigma. Which takes a lot of data.
- dnautics 6y agoyou really can't compare stats like that. Those are independent, uncorrelated measurements. When you take RMSD measurements on a molecule they are not independent (for example, atoms near the core are less likely to be "inaccurate").
- cpeterso 6y agoAnd is it even meaningful for DeepMind to score better than experimental results? How are DeepMind’s results scored then?
- marcosdumay 6y agoFinding the energy of each configuration should be much easier than finding the lowest-energy configuration. Can that be calculated ab-initio or it is still too expensive?
- WhompingWindows 6y agoThis reminds me of AlphaGo and AlphaZero. DeepMind was able to produce a very solid model on their first attempt, at both protein folding and at Go (and Starcraft2 as well). Their second models, however, seemed to blow their first out of the water. This bodes extremely well for the future of computational biology, I'm very excited thinking about the prospects. If we know how a protein folds, we know its shape, meaning we know which shaped/charged molecules are needed to act as suppressors/enhancers of those proteins.
- layer8 6y agoOne difference to AlphaZero though, if my understanding is correct, is that AlphaFold is trained on a predetermined data set and hence didn’t learn how “arbitrary” proteins fold in general, but just how the kinds of proteins fold for which we already know how they fold. To work more like AlphaZero, AlphaFold would have to be able to synthesize arbitrary proteins and run the experiments on them to verify and correct its predictions. Therefore it’s conceivable that AlphaFold is biased by the existing training data and doesn’t fully generalize to all proteins we would want to apply it to. Maybe that won’t be a problem in practice, but nevertheless it makes for a significant difference from what AlphaZero was about, being solely self-trained.
- the8472 6y ago> AlphaFold would have to be able to synthesize arbitrary proteins and run the experiments on them to verify and correct its predictions. Could this lead to a virtuous cycle where AlphaFold is used generate a ton of random sequences where it has low confidence, those are then screened for ease of synthesis, measured and the results used to improve the model? Edit: nevermind, according to another comment[0] there are still plenty of real proteins without experimental data left to explore. [0] https://news.ycombinator.com/item?id=25255601 https://news.ycombinator.com/item?id=25255601
- cma 6y ago> AlphaFold would have to be able to synthesize arbitrary proteins and run the experiments on them to verify and correct its predictions. It can verify how much it minimizes the potential energy, which may not always line up with how it would fold in the real world but is a strong indicator.
- martinpw 6y agoWhy is the graph not monotonically increasing? Does the complexity of the problem to be solved increase each time? If so, does that make the relative improvement from the previous result even more impressive?
- breatheoften 6y agoThat's quite interesting ... I believe the test set size is not constant year to year but rather a function of how many new structures have been experimentally discovered since the last contest? Does seem like the contest structure could include quite a bit of risk for hiding the effect of overfitting ... I wonder if there is anything inherent about the problem that reduces that risk ...?
- FrojoS 6y agoMy understanding is, that it's always 100 new structures, which is a small fraction of the total structures identified in that year. The reason why the top score in one year, can be lower than in the previous year, is that the test (the 100 structures to guess) is always new and different, so it can end up being 'harder' than the year before. Luck will also play a small role. Another explanation for a reduction in the top score would be, that previous winners are not re-submitted unchanged. For instance AlphaFold v1 seems to not have been submitted to the latest competition.
- breatheoften 6y agoOnly 100 new structures each test cycle? That seems a very small test set size ... Is it really possible to select 100 new structures which together are likely to represent a meaningful increase in the sample generalization versus the prior years test set ...?
- MauranKilom 6y agoGiven that we only know the structure of on the order of 100k proteins, we might only get another 10k new ones per year. I guess. Using 1% of those (presumably from the more-often-reproduced subset) for this challenge seems reasonable? Note that the structures have to remain secret up until the challenge, and presumably all those teams uncovering the structures don't want to have to wait up to 2 years every time to actually make their results public.