5 ms·

Something that I'm fascinated with is the question, would other models show equivalent results if given the same amount of compute? I looked through the list o

Something that I'm fascinated with is the question, would other models show equivalent results if given the same amount of compute? I looked through the list of solvers for the protein folding challenges and there were other deep learning, neural network, and classical machine learning approaches on there. Even some hybrid ones! But none of the participants had even a fraction of the compute power that AlphaFold had behind it. Some of the entries were small university teams. Others were powered by the computers some professor had in their closet (!). Most of the teams were dramatically under-powered as compared to AlphaFold. How much did this influence the final result? What would the other results look like if they'd been on equal footing? Would they have been closer? It's a genuine question.

WithinReason 5y ago
The advantage of DeepMind from more compute resources is cumulative, you can develop better models if you have more compute. So the question really is: If academia and other research institutions had DeepMind's resources, would their output match DeepMind's?
gradys 5y ago
Probably not. It's not clear how to scale other methods to make use of so much compute. They top out sooner, with fewer parameters and less compute. One way to look at why deep learning is having the impact it does is that unlike other ML methods, it's actually capable of making use of so much compute. It gives us modular ways to add more and more parameters and still fit them effectively.
- WanderPanda 5y ago
  What always astonishes me that deep learning seems to work on human timescales. For other problems (even if they are polynomial in complexity) we get into infeasible runtimes when we increase the problems complexity by 10x. With deep learning the fuzzy, approximative nature seems to help to grasp the gist of the 10x problem and somehow allows us to reach 95% of the solution in just e.g. 2x the runtime. Heuristics might play in the same league, but the development time kind of scales with the problem, while in deep learning I would put it in the linear or log basket.
dekhn 5y ago
actually, companies can buy the amount of TPU time that AlphaFold used. Note that the price breaks from "here is a listed price" to "contact sales" at the largest size (TPU v3-2048). I assume that during initial training and experimentation that DM kept 1 or two TPU v3-2048 busy 24/7 for 6 months to a year. That's entirely within the budget of well-heeled companies
vasco 5y ago
Presumably the team started with existing methods on their fleet of compute and was forced to develop new ones. While development happens your fleet also grows and your ability to utilize it expands and you end up not going back to ever try methods that are generations behind on your current fleet because the overhead of making it work doesn't justify it. It's like asking if using the google homepage from 10k iterations ago would perform better than the current version on the present user cohorts. There's just too much invested to justify testing things like that when you can use the time to improve what is showing promise.
sdenton4 5y ago
A couple thoughts... First, having 'infinite compute' is a way for researchers to be sure that compute isn't the thing holding back their method. So, DeepMind made a protein folder using all the compute they had available; later others managed to greatly reduce the amount of compute needed to get equivalent results, by re-implementing and innovating on DeepMind's initial write-up. Second, I think there's a lot of interesting ground to explore in hybridizing ML with more classical algorithms. The end-to-end deep learning approach gives us models that get the best scores on imagenet classification, but are extremely prone to domain shift problems. An alternative approach is to take a 'classical' algorithm and swap out certain parts of the algorithm with a deep network. Then you should often get something more explainable: The network is explicitly filling in a specific parameter estimation, and the rest of the algorithm gives you some quality guarantees based on the quality of that parameter estimator. I saw some nice work along these lines in sparse coding a couple years ago, for example...
randcraw 5y ago
No, some DL architectures are unquestionably superior to others especially for some tasks. In the first few years of deep CNN development, several limits quickly became apparent (sensitivity to scale, vanishing gradients, gradient signal loss with some activation functions, and many engineering constraint tradeoffs like poor RAM use). This was addressed by making changes to a simple dense net which begat architectures that suffer less from those limits. Without question, limits exist in every DL architecture; we just haven't taken the time to diagnose or quantify all the limits yet. Now that attention has shifted to transformers, analysis of DNNs' inherent limits is made substantially more difficult, given transformers' huge size. That, and their multi-million dollar training cost likely will make if infeasible to diagnose or cure each design's inherent limits.
- machiaweliczny 5y ago
  Anyone has some more info about geometric deep learning? It seems to offer some theory why certain architectures work and others dont.
neatze 5y ago
I don't think computational resources will have greater effect, because you can see this by simple plot running time x axis and model performance y axis.
Iv 5y ago
You can't just throw more compute at an algorithm and have it spit better results! Well, unless your algorithm uses deep learning that is. And that's why teams with a lot of compute use DL, and why team that use DL use a lot of compute
kragen 5y ago
ANNs are among the few kinds of programs that perform significantly better on small problems with exaflops than with teraflops. Gurobi, LS-Dyna, or American Fuzzy Lop will produce only marginally better results as you scale from a closet full of computers to a warehouse-scale computer. GCC or linear regression won't even work marginally better.
- jedbrown 5y ago
  "The easiest way to make software scalable is to make it sequentially inefficient." -- Bill Gropp (1999)
  - kragen 5y ago
    That's true! But ANNs are sometimes able to do things we haven't found a way to do more efficiently, as in this case.
Epokhe 5y ago
Alphafold vs Alphafold 2 success: https://lh3.googleusercontent.com/KKbgSsS1qIoesiy2Ws_WDsDSyGhTZgP9W3qZr-xS5ElnafEu80joptKmc2hgz01a6j6yIj5cvCnqz8bBfXG8BND44ZKJ_kv7tTHQAA=w2048-rw-v1 https://lh3.googleusercontent.com/KKbgSsS1qIoesiy2Ws_WDsDSyG... Papers say that both are trained on PDB dataset. And still, we see a dramatic gap between old and new Alphafold models. Both were trained by Deepmind, probably with a similar computer power. I think it's obvious that it's not just compute power, method matters a lot.

WithinReason 5y ago

The advantage of DeepMind from more compute resources is cumulative, you can develop better models if you have more compute. So the question really is: If academia and other research institutions had DeepMind's resources, would their output match DeepMind's?

gradys 5y ago

Probably not. It's not clear how to scale other methods to make use of so much compute. They top out sooner, with fewer parameters and less compute. One way to look at why deep learning is having the impact it does is that unlike other ML methods, it's actually capable of making use of so much compute. It gives us modular ways to add more and more parameters and still fit them effectively.

WanderPanda 5y ago
What always astonishes me that deep learning seems to work on human timescales. For other problems (even if they are polynomial in complexity) we get into infeasible runtimes when we increase the problems complexity by 10x. With deep learning the fuzzy, approximative nature seems to help to grasp the gist of the 10x problem and somehow allows us to reach 95% of the solution in just e.g. 2x the runtime. Heuristics might play in the same league, but the development time kind of scales with the problem, while in deep learning I would put it in the linear or log basket.

dekhn 5y ago

actually, companies can buy the amount of TPU time that AlphaFold used. Note that the price breaks from "here is a listed price" to "contact sales" at the largest size (TPU v3-2048). I assume that during initial training and experimentation that DM kept 1 or two TPU v3-2048 busy 24/7 for 6 months to a year. That's entirely within the budget of well-heeled companies

vasco 5y ago

Presumably the team started with existing methods on their fleet of compute and was forced to develop new ones. While development happens your fleet also grows and your ability to utilize it expands and you end up not going back to ever try methods that are generations behind on your current fleet because the overhead of making it work doesn't justify it. It's like asking if using the google homepage from 10k iterations ago would perform better than the current version on the present user cohorts. There's just too much invested to justify testing things like that when you can use the time to improve what is showing promise.

sdenton4 5y ago

A couple thoughts... First, having 'infinite compute' is a way for researchers to be sure that compute isn't the thing holding back their method. So, DeepMind made a protein folder using all the compute they had available; later others managed to greatly reduce the amount of compute needed to get equivalent results, by re-implementing and innovating on DeepMind's initial write-up. Second, I think there's a lot of interesting ground to explore in hybridizing ML with more classical algorithms. The end-to-end deep learning approach gives us models that get the best scores on imagenet classification, but are extremely prone to domain shift problems. An alternative approach is to take a 'classical' algorithm and swap out certain parts of the algorithm with a deep network. Then you should often get something more explainable: The network is explicitly filling in a specific parameter estimation, and the rest of the algorithm gives you some quality guarantees based on the quality of that parameter estimator. I saw some nice work along these lines in sparse coding a couple years ago, for example...

randcraw 5y ago

No, some DL architectures are unquestionably superior to others especially for some tasks. In the first few years of deep CNN development, several limits quickly became apparent (sensitivity to scale, vanishing gradients, gradient signal loss with some activation functions, and many engineering constraint tradeoffs like poor RAM use). This was addressed by making changes to a simple dense net which begat architectures that suffer less from those limits. Without question, limits exist in every DL architecture; we just haven't taken the time to diagnose or quantify all the limits yet. Now that attention has shifted to transformers, analysis of DNNs' inherent limits is made substantially more difficult, given transformers' huge size. That, and their multi-million dollar training cost likely will make if infeasible to diagnose or cure each design's inherent limits.

machiaweliczny 5y ago
Anyone has some more info about geometric deep learning? It seems to offer some theory why certain architectures work and others dont.

neatze 5y ago

I don't think computational resources will have greater effect, because you can see this by simple plot running time x axis and model performance y axis.

Iv 5y ago

You can't just throw more compute at an algorithm and have it spit better results! Well, unless your algorithm uses deep learning that is. And that's why teams with a lot of compute use DL, and why team that use DL use a lot of compute

kragen 5y ago

ANNs are among the few kinds of programs that perform significantly better on small problems with exaflops than with teraflops. Gurobi, LS-Dyna, or American Fuzzy Lop will produce only marginally better results as you scale from a closet full of computers to a warehouse-scale computer. GCC or linear regression won't even work marginally better.

jedbrown 5y ago
"The easiest way to make software scalable is to make it sequentially inefficient." -- Bill Gropp (1999)
- kragen 5y ago
  That's true! But ANNs are sometimes able to do things we haven't found a way to do more efficiently, as in this case.

Epokhe 5y ago

Alphafold vs Alphafold 2 success: https://lh3.googleusercontent.com/KKbgSsS1qIoesiy2Ws_WDsDSyGhTZgP9W3qZr-xS5ElnafEu80joptKmc2hgz01a6j6yIj5cvCnqz8bBfXG8BND44ZKJ_kv7tTHQAA=w2048-rw-v1 https://lh3.googleusercontent.com/KKbgSsS1qIoesiy2Ws_WDsDSyG... Papers say that both are trained on PDB dataset. And still, we see a dramatic gap between old and new Alphafold models. Both were trained by Deepmind, probably with a similar computer power. I think it's obvious that it's not just compute power, method matters a lot.