4 ms·
I wonder if a transformer with 1 recurrent layer (or a transformer Deep Equilibrium Model) could work well. Transformers are almost like a physics simulation i
by eutectic 5y ago
I wonder if a transformer with 1 recurrent layer (or a transformer Deep Equilibrium Model) could work well. Transformers are almost like a physics simulation in that they sum vector-valued interactions which depend on distance in some space, and then add the result to the state of each particle / element.
- dekhn 5y agoReasonably speaking, I would expect that within 2 years 10 groups will be as proficient as AF2 at predicting static structures. I don't think anybody who is trying to emulate physics simulations will be in that group, just folks who have learned enough tricks to quickly incorporate all the evidence during training and choose how to apply it during prediction. I expect to see a "multiple feature embedding heads on top of 2 fullly connected layers" (as used in modern ads training) will end up being the simplest architecture capable of folding proteins well.