11 ms·

DeepMind achieves SOTA image recognition with 8.7x faster training

291 points by highfrequency 6y ago

dheera 6y ago
> 8.7x faster to train This is an achievement but it would be helpful to have put "to train" in the title as this is quite different from efficiency at inference time, which is what often actually matters in deployed applications. From Table 3 on Page 7 it appears to me that NFNet is significantly heavier in the number of parameters than EfficientNet for similar accuracies. For example EffNet-B5 achieves 83.7% with 30M params and 9.9B FLOPs on Top-1 while NFNet-F0 achieves 83.6% with 71.5M params and 12.38B FLOPs on Top-1. It appears to me at first glance that NFNet has not achieved SOTA at inference.
- trott 6y ago
  > It appears to me at first glance that NFNet has not achieved SOTA at inference. It has, for larger models (F1 vs B7). See Fig 4 in the Appendix.
  - The_rationalist 6y ago
    No it hasn't https://paperswithcode.com/sota/image-classification-on-imagenet https://paperswithcode.com/sota/image-classification-on-imag...
    - trott 6y ago
      > No it hasn't https://paperswithcode.com/sota/image-classification-on-imag https://paperswithcode.com/sota/image-classification-on-imag... We were talking about models trained on ImageNet, specifically about the trade-off between accuracy and FLOPs. But the higher-accuracy models listed in your link use extra data. So it's not quite the same benchmark we were talking about.
      - The_rationalist 6y ago
        The deepmind paper NFNet-F4+ you were talking about also has external training data. The number one in accuracy (Meta pseudo labels) is also faster for inference (390M vs 570M parameters) vs the deepmind one. So what are you disagreeing with?
        trott 6y ago
        > The deepmind paper NFNet-F4+ you were talking about also has external training data. @dheera and I did not mention NFNet-F4+. All models, tables, figures and numbers that we did mention resulted from training on ImageNet alone.
sillysaurusx 6y ago
I was skeptical of the 83%-whatever top-1 accuracy on Imagenet. But someone pointed out that the accuracy increases when the model is pretrained on JFT, google's proprietary 100-million image dataset, the model's accuracy increases to 87%-whatever. That's pretty interesting. It implies that the original accuracy rating might be legit. The concern is that we're chasing the imagenet benchmark as if it's the holy grail, when in fact it's a very narrow slice of what we normally care about as ML researchers. However, the fact that pretraining on JFT increases the accuracy means that the model is generalizing, which is very interesting; it implies that models might be "just that good now." Or more succinctly, if the result was bogus, you'd expect JFT pretraining to have no effect whatsoever (or a negative effect). But it has a positive result. The other thing worth mentioning is that AJMooch seems to have killed batch normalization dead, which is very strange to think about. BN has had a long reign of some ~4 years, but the drawbacks are significant: you have to maintain counters yourself, for example, which was quite annoying. It always seemed like a neural net ought to be able to learn what BN forces you to keep track of. And AJMooch et al seem to prove this is true. I recommend giving evonorm-s a try; it worked perfectly for us the first time, with no loss in generality, and it's basically a copy-paste replacement. (Our BigGAN-Deep model is so good that I doubt you can tell the difference vs the official model. It uses AJMooch's evonorm-s rather than batchnorm: [1] https://i.imgur.com/sfGVbuq.png https://i.imgur.com/sfGVbuq.png [2] https://i.imgur.com/JMJ1Ll0.png https://i.imgur.com/JMJ1Ll0.png and lol at the fake speedometer.)
- trhway 6y ago
  > pretraining on JFT increases the accuracy means that the model is generalizing not necessarily, it may be mostly a bonus of the transfer, especially considering that JFT is that much larger - getting for example the first conv layers kernels to converge to Gabor-like takes time, yet those layers kernels are very similar across the well trained image nets (and there were some works showing that it is optimal in a sense, and that it is one of the reasons it is in our visual cortex) and thus transferrable, and can practically be treated as fixed in the new model (especially if those layers were pretrained on very large model and reached the state of generic feature extraction). I suspect the similar is applicable for the low level feature aggregating layers too.
- 6y ago
Willson50 6y ago
*11.5% as much compute.
- The_rationalist 6y ago
  But what was the baseline hardware for a reasonable training time?
  - nl 6y ago
    Page 7 has a table of one training step on TPUv3 and V100 GPUs. I don't completely understand this: NFNet is slower than its competitors on this benchmark, but they claim higher efficiency. This isn't obvious to me.
    - trott 6y ago
      > I don't completely understand this: NFNet is slower than its competitors on this benchmark, but they claim higher efficiency. Take a look at F1 and B7: They have the same accuracy, but F1 is smaller and much faster.
longtom 6y ago
tl;dr: Don't use batch norm for preventing exploding gradients but adaptive gradient thresholds. For this they compute the Frobenius norm (square root of the sum of squares) of the weight layer and its gradient and take the ratio of these as clipping threshold. That saves the meta search for the optimal threshold but also is better than a fixed threshold could ever be. Very simple idea.
- sillysaurusx 6y ago
  Thanks for macroexpanding frobnorm. I'm skeptical that these hand-coded thresholds can ever match what a model can learn automatically. But it's hard to argue with results.
- highfrequency 6y ago
  Is this the first time that the gradient clip threshold has been chosen relative to the size of the weight matrix?
- longtom 6y ago
  Lol, why is my comment down here with 7 upvotes.
smeeth 6y ago
The speed improvements are certainly interesting, the performance improvements seem decidedly not. This method has more than 2x the parameters of all but one of the models it was compared against. If I’m off-base here can someone explain?
- modeless 6y ago
  I don't care how many parameters my model has per se. What I care about is how expensive it is to train in time and dollars. If this makes it cheaper to train better models despite more parameters, that's still a win.
  - sillysaurusx 6y ago
    There's one important caveat, though I agree with your thrust: at GPT-3 scale, cutting params in half is a nontrivial optimization. So it's worth keeping an eye out for that concern. (Yeah, none of us are anywhere near GPT-3 scale. But I spend most of my time thinking about scaling issues, and it was interesting to see your comment pop up; I would've agreed entirely with it myself, if not for seeing all the anguish caused by attempting to train and deploy billions of parameters.)
  - 6gvONxR4sf7o 6y ago
    Some models are still memory limited. Fewer parameters are very useful in those settings.
  - dontreact 6y ago
    In cases where you have to deploy the model and you are limited in terms of flops, this paper does not help much, unless it’s removal of batchnorm somehow allows a future network that is actually faster at inference time.
    - modeless 6y ago
      There are a lot of techniques for sparsifying or pruning or distilling models to reduce inference FLOPS, and they almost always produce better results when starting with a better model. Also, if your model is 8x faster to train at the same size then you can do 8x as much hyperparameter tuning and get a better result.
nullifidian 6y ago
Do they disclose any important techniques/ideas on how to achieve these results in the paper, or it's more of a technical press release?
- belgian_guy 6y ago
  Yes they do (arxiv is a place for scientific papers not press releases). I've only skimmed it, but the paper introduce an adaptive way to clip gradients. Meaning that if the ratio of the gradient norm to weight norm surpasses a certain threshold, they clip it. This stabilizes learning and seems to avoid the need for batch normalization. Seems quite promising imo and something that could stick (I'm quite happy if we could finally do away with batchnorm).
  - eggie5 6y ago
    You missed a big part: they did a big NAS run to make it work.
    - rsfern 6y ago
      Where did you see that they used NAS? Their preliminary results show it works even for the baseline model they did a lot of manual hyperparameter optimization, and spend a fair amount of time unpacking the rationale for their choices, including a negative results section (!) in the appendix
modeless 6y ago
The big deal here is the removal of BatchNorm. People never really liked BatchNorm for various theoretical and practical reasons, and yet it was required for all the top performing models. If this allows us to get rid of it forever that will be really nice.
highfrequency 6y ago
They compare the training latency for different models with a fixed batch size of 32. But if the DeepMind models are several times larger than the comparison models in each latency class, it seems that the comparison models could use larger batch sizes for faster overall training time.
- trott 6y ago
  For ConvNets, the memory use of the models themselves is pretty modest. For example, even with 0.5B parameters, with FP32, weights+gradients+momentum should use just 6GB (unless your framework sucks, or you have extra overhead from distributed training) So, if your model is twice smaller, you'll only save 3GB. If your VRAM is 32GB, saving 3GB won't let you use a much bigger batch size. On the other hand, the absence of batch norm can actually lead to memory savings proportional to batch size.
f430 6y ago
eli5 what SOTA image recognition is?
- jphoward 6y ago
  SOTA is 'state of the art'. Image recognition is a task classically appraised by calculating the accuracy on the ImageNet dataset, which requires a system to classify images each as one of 1,000 pre-determined classes.
  - f430 6y ago
    so how many images does the current SOTA take to train a classifier? Trying to gauge how much of an improvement Deepmind has made here.
    - jphoward 6y ago
      The results that should be used to compare their results to others involves training on just under 1.3 million images across the 1,000 classes. Their best results involve 'pretraining' on a dataset of 300 million examples, before 'tuning' it on the actual ImageNet training dataset as above.
- deleted 6y ago
  [deleted]
- marviel 6y ago
  State of The Art == SOTA You might enjoy paperswithcode.com
throwaway189262 6y ago
I have a feeling the ML community is going to pivot focus to faster and smaller training before larger advancements are made. It's simply too expensive for much AI research to happen when state of the art models take 500k of hardware to train. For all the mathematician hype around ML research, much of the work is closer to alchemy than science. We simply don't understand a great deal of why these neural nets work. The people doing math above algebra are few and the scene is dominated by "guess and check" style model tinkering. Many "state of the art models" are simply a bunch of common strategies glued together in a way researchers found worked the best (by trying a bunch of different ones). An average Joe could probably write influential ML papers by gluing RNN/GAN layers to existing models and fiddling with the parameters until they beat current state of the art. In fact, in NLP models, this is essentially what has happened with roBERTa, XLNET, ELECTRA, etc. They're all somewhat trivial variations on Google's BERT, which is more creative but yet again built on existing models. Anyways, my point is, none of this required math or genius or particularly demanding thought. It was basically let's tinker with this until we find a way that's better, using guess and check. No equations needed. We are a long way from the type of simulations done for protein folding and materials strength and basically every other scientific field. It's still the wild west
- The_rationalist 6y ago
  While I generally agree to some extent: roBERTa, XLNET, ELECTRA, etc. They're all somewhat trivial variations on Google's BERT, which is more creative Researchers take inspirations from existing models of course and some BERT derivatives are trivial. However, XLnet is in it's own league, while the author (a genius chinese student) was inspired by BERT it is one of the few SOTA pré trained models to be not based on BERT and is actually an auto regressive one! Such difference allow it to be better at many things as it doesn't has to corrupt the tokens (from my shallow understanding). This model is two years old but is still sadly the one that ranks the most SOTA in key tasks e.g dependency parsing. And after all those time nobody cared enough to even test it on other foundational tasks (which is extremely sad and pathetic) like e.g coreference resolution. Sadly because of conformism effects almost zero researcher has created XLnet derivatives. Almost all researchers continue to search in the local minima that is BERT, which I find, immensely ironic. While ad hoc empirical fine tuning is a big part of improving sota, mathematical genius can still enable revolutions e.g this recent alternative to classical backpropagation that is 300X faster with low accuracy loss https://paperswithcode.com/paper/zorb-a-derivative-free-backpropagation https://paperswithcode.com/paper/zorb-a-derivative-free-back...
The_rationalist 6y ago
This isn't the real SOTA, "Meta pseudo labels" has ~10% less errors, while having less parameters. https://paperswithcode.com/sota/image-classification-on-imagenet https://paperswithcode.com/sota/image-classification-on-imag... However the fast training is an interesting property. It would be interesting to test thoses efficientNets with zeroth order backpropagation as it allows a 300X speedup (vs 8.7x) while not regressing accuracy that much https://paperswithcode.com/paper/zorb-a-derivative-free-backpropagation https://paperswithcode.com/paper/zorb-a-derivative-free-back...
nl 6y ago
Make it work then make it fast and efficient. People complaining about how slow and expensive brand new models are to train are ignorant of the history of machine learning and of engineering in general.
eggie5 6y ago
Do what do attribute to the gains? Adaptive clipping? Or $$$ spent on NAS??
- rsfern 6y ago
  They do a little study in section 4.1 comparing batchnorm to adaptive gradient clipping for resnets over a range of hyperparameters, and they also compare perf to batchnorm versions in table 6. The results indicate AGC does give a real boost over batchnorm They do a bunch of manual hyperparameter tuning that seems necessary to get the state of the art results, from my reading it doesn’t seem like they actually used NAS. Just that the baseline they compare to was found with NAS
FL33TW00D 6y ago
As distributed training becomes the norm, the lack of BN will become more and more desirable.
jjssmith 6y ago
Rediscovering NLMS now, good for ML folks. https://en.wikipedia.org/wiki/Least_mean_squares_filter#Normalized_least_mean_squares_filter_(NLMS) https://en.wikipedia.org/wiki/Least_mean_squares_filter#Norm...
Yenrabbit 6y ago
Yannic Kilcher has a great video on this out already, including some thoughtful critique. https://youtu.be/rNkHjZtH0RQ https://youtu.be/rNkHjZtH0RQ One of the things he mentioned was that they introduce this fancy alternative to BatchNorm AND come up with a fancy new architecture. The combination does really well, but it isn't clear how much of the gain is due to the new improved architecture Vs the 'adaptive gradient clipping' they introduce.
diptanu 6y ago
Is there an implementation for what they are describing in PyTorch?
rxsel 6y ago
Is there any video work being done in the realm of image processing? I mean a video is just essentially an array/string of images/frames no?

291 points by highfrequency 6y ago

> 8.7x faster to train This is an achievement but it would be helpful to have put "to train" in the title as this is quite different from efficiency at inference time, which is what often actually matters in deployed applications. From Table 3 on Page 7 it appears to me that NFNet is significantly heavier in the number of parameters than EfficientNet for similar accuracies. For example EffNet-B5 achieves 83.7% with 30M params and 9.9B FLOPs on Top-1 while NFNet-F0 achieves 83.6% with 71.5M params and 12.38B FLOPs on Top-1. It appears to me at first glance that NFNet has not achieved SOTA at inference.

trott 6y ago
> It appears to me at first glance that NFNet has not achieved SOTA at inference. It has, for larger models (F1 vs B7). See Fig 4 in the Appendix.
- The_rationalist 6y ago
  No it hasn't https://paperswithcode.com/sota/image-classification-on-imagenet https://paperswithcode.com/sota/image-classification-on-imag...
  - trott 6y ago
    > No it hasn't https://paperswithcode.com/sota/image-classification-on-imag https://paperswithcode.com/sota/image-classification-on-imag... We were talking about models trained on ImageNet, specifically about the trade-off between accuracy and FLOPs. But the higher-accuracy models listed in your link use extra data. So it's not quite the same benchmark we were talking about.
    - The_rationalist 6y ago
      The deepmind paper NFNet-F4+ you were talking about also has external training data. The number one in accuracy (Meta pseudo labels) is also faster for inference (390M vs 570M parameters) vs the deepmind one. So what are you disagreeing with?
      - trott 6y ago
        > The deepmind paper NFNet-F4+ you were talking about also has external training data. @dheera and I did not mention NFNet-F4+. All models, tables, figures and numbers that we did mention resulted from training on ImageNet alone.

sillysaurusx 6y ago

I was skeptical of the 83%-whatever top-1 accuracy on Imagenet. But someone pointed out that the accuracy increases when the model is pretrained on JFT, google's proprietary 100-million image dataset, the model's accuracy increases to 87%-whatever. That's pretty interesting. It implies that the original accuracy rating might be legit. The concern is that we're chasing the imagenet benchmark as if it's the holy grail, when in fact it's a very narrow slice of what we normally care about as ML researchers. However, the fact that pretraining on JFT increases the accuracy means that the model is generalizing, which is very interesting; it implies that models might be "just that good now." Or more succinctly, if the result was bogus, you'd expect JFT pretraining to have no effect whatsoever (or a negative effect). But it has a positive result. The other thing worth mentioning is that AJMooch seems to have killed batch normalization dead, which is very strange to think about. BN has had a long reign of some ~4 years, but the drawbacks are significant: you have to maintain counters yourself, for example, which was quite annoying. It always seemed like a neural net ought to be able to learn what BN forces you to keep track of. And AJMooch et al seem to prove this is true. I recommend giving evonorm-s a try; it worked perfectly for us the first time, with no loss in generality, and it's basically a copy-paste replacement. (Our BigGAN-Deep model is so good that I doubt you can tell the difference vs the official model. It uses AJMooch's evonorm-s rather than batchnorm: [1] https://i.imgur.com/sfGVbuq.png https://i.imgur.com/sfGVbuq.png [2] https://i.imgur.com/JMJ1Ll0.png https://i.imgur.com/JMJ1Ll0.png and lol at the fake speedometer.)

trhway 6y ago
> pretraining on JFT increases the accuracy means that the model is generalizing not necessarily, it may be mostly a bonus of the transfer, especially considering that JFT is that much larger - getting for example the first conv layers kernels to converge to Gabor-like takes time, yet those layers kernels are very similar across the well trained image nets (and there were some works showing that it is optimal in a sense, and that it is one of the reasons it is in our visual cortex) and thus transferrable, and can practically be treated as fixed in the new model (especially if those layers were pretrained on very large model and reached the state of generic feature extraction). I suspect the similar is applicable for the low level feature aggregating layers too.
6y ago

Willson50 6y ago

*11.5% as much compute.

The_rationalist 6y ago
But what was the baseline hardware for a reasonable training time?
- nl 6y ago
  Page 7 has a table of one training step on TPUv3 and V100 GPUs. I don't completely understand this: NFNet is slower than its competitors on this benchmark, but they claim higher efficiency. This isn't obvious to me.
  - trott 6y ago
    > I don't completely understand this: NFNet is slower than its competitors on this benchmark, but they claim higher efficiency. Take a look at F1 and B7: They have the same accuracy, but F1 is smaller and much faster.

longtom 6y ago

tl;dr: Don't use batch norm for preventing exploding gradients but adaptive gradient thresholds. For this they compute the Frobenius norm (square root of the sum of squares) of the weight layer and its gradient and take the ratio of these as clipping threshold. That saves the meta search for the optimal threshold but also is better than a fixed threshold could ever be. Very simple idea.

sillysaurusx 6y ago
Thanks for macroexpanding frobnorm. I'm skeptical that these hand-coded thresholds can ever match what a model can learn automatically. But it's hard to argue with results.
highfrequency 6y ago
Is this the first time that the gradient clip threshold has been chosen relative to the size of the weight matrix?
longtom 6y ago
Lol, why is my comment down here with 7 upvotes.

smeeth 6y ago

The speed improvements are certainly interesting, the performance improvements seem decidedly not. This method has more than 2x the parameters of all but one of the models it was compared against. If I’m off-base here can someone explain?

modeless 6y ago
I don't care how many parameters my model has per se. What I care about is how expensive it is to train in time and dollars. If this makes it cheaper to train better models despite more parameters, that's still a win.
- sillysaurusx 6y ago
  There's one important caveat, though I agree with your thrust: at GPT-3 scale, cutting params in half is a nontrivial optimization. So it's worth keeping an eye out for that concern. (Yeah, none of us are anywhere near GPT-3 scale. But I spend most of my time thinking about scaling issues, and it was interesting to see your comment pop up; I would've agreed entirely with it myself, if not for seeing all the anguish caused by attempting to train and deploy billions of parameters.)
- 6gvONxR4sf7o 6y ago
  Some models are still memory limited. Fewer parameters are very useful in those settings.
- dontreact 6y ago
  In cases where you have to deploy the model and you are limited in terms of flops, this paper does not help much, unless it’s removal of batchnorm somehow allows a future network that is actually faster at inference time.
  - modeless 6y ago
    There are a lot of techniques for sparsifying or pruning or distilling models to reduce inference FLOPS, and they almost always produce better results when starting with a better model. Also, if your model is 8x faster to train at the same size then you can do 8x as much hyperparameter tuning and get a better result.

nullifidian 6y ago

Do they disclose any important techniques/ideas on how to achieve these results in the paper, or it's more of a technical press release?

belgian_guy 6y ago
Yes they do (arxiv is a place for scientific papers not press releases). I've only skimmed it, but the paper introduce an adaptive way to clip gradients. Meaning that if the ratio of the gradient norm to weight norm surpasses a certain threshold, they clip it. This stabilizes learning and seems to avoid the need for batch normalization. Seems quite promising imo and something that could stick (I'm quite happy if we could finally do away with batchnorm).
- eggie5 6y ago
  You missed a big part: they did a big NAS run to make it work.
  - rsfern 6y ago
    Where did you see that they used NAS? Their preliminary results show it works even for the baseline model they did a lot of manual hyperparameter optimization, and spend a fair amount of time unpacking the rationale for their choices, including a negative results section (!) in the appendix

modeless 6y ago

The big deal here is the removal of BatchNorm. People never really liked BatchNorm for various theoretical and practical reasons, and yet it was required for all the top performing models. If this allows us to get rid of it forever that will be really nice.

highfrequency 6y ago

They compare the training latency for different models with a fixed batch size of 32. But if the DeepMind models are several times larger than the comparison models in each latency class, it seems that the comparison models could use larger batch sizes for faster overall training time.

trott 6y ago
For ConvNets, the memory use of the models themselves is pretty modest. For example, even with 0.5B parameters, with FP32, weights+gradients+momentum should use just 6GB (unless your framework sucks, or you have extra overhead from distributed training) So, if your model is twice smaller, you'll only save 3GB. If your VRAM is 32GB, saving 3GB won't let you use a much bigger batch size. On the other hand, the absence of batch norm can actually lead to memory savings proportional to batch size.

f430 6y ago

eli5 what SOTA image recognition is?

jphoward 6y ago
SOTA is 'state of the art'. Image recognition is a task classically appraised by calculating the accuracy on the ImageNet dataset, which requires a system to classify images each as one of 1,000 pre-determined classes.
- f430 6y ago
  so how many images does the current SOTA take to train a classifier? Trying to gauge how much of an improvement Deepmind has made here.
  - jphoward 6y ago
    The results that should be used to compare their results to others involves training on just under 1.3 million images across the 1,000 classes. Their best results involve 'pretraining' on a dataset of 300 million examples, before 'tuning' it on the actual ImageNet training dataset as above.
deleted 6y ago
[deleted]
marviel 6y ago
State of The Art == SOTA You might enjoy paperswithcode.com

throwaway189262 6y ago

I have a feeling the ML community is going to pivot focus to faster and smaller training before larger advancements are made. It's simply too expensive for much AI research to happen when state of the art models take 500k of hardware to train. For all the mathematician hype around ML research, much of the work is closer to alchemy than science. We simply don't understand a great deal of why these neural nets work. The people doing math above algebra are few and the scene is dominated by "guess and check" style model tinkering. Many "state of the art models" are simply a bunch of common strategies glued together in a way researchers found worked the best (by trying a bunch of different ones). An average Joe could probably write influential ML papers by gluing RNN/GAN layers to existing models and fiddling with the parameters until they beat current state of the art. In fact, in NLP models, this is essentially what has happened with roBERTa, XLNET, ELECTRA, etc. They're all somewhat trivial variations on Google's BERT, which is more creative but yet again built on existing models. Anyways, my point is, none of this required math or genius or particularly demanding thought. It was basically let's tinker with this until we find a way that's better, using guess and check. No equations needed. We are a long way from the type of simulations done for protein folding and materials strength and basically every other scientific field. It's still the wild west

The_rationalist 6y ago
While I generally agree to some extent: roBERTa, XLNET, ELECTRA, etc. They're all somewhat trivial variations on Google's BERT, which is more creative Researchers take inspirations from existing models of course and some BERT derivatives are trivial. However, XLnet is in it's own league, while the author (a genius chinese student) was inspired by BERT it is one of the few SOTA pré trained models to be not based on BERT and is actually an auto regressive one! Such difference allow it to be better at many things as it doesn't has to corrupt the tokens (from my shallow understanding). This model is two years old but is still sadly the one that ranks the most SOTA in key tasks e.g dependency parsing. And after all those time nobody cared enough to even test it on other foundational tasks (which is extremely sad and pathetic) like e.g coreference resolution. Sadly because of conformism effects almost zero researcher has created XLnet derivatives. Almost all researchers continue to search in the local minima that is BERT, which I find, immensely ironic. While ad hoc empirical fine tuning is a big part of improving sota, mathematical genius can still enable revolutions e.g this recent alternative to classical backpropagation that is 300X faster with low accuracy loss https://paperswithcode.com/paper/zorb-a-derivative-free-backpropagation https://paperswithcode.com/paper/zorb-a-derivative-free-back...

The_rationalist 6y ago

This isn't the real SOTA, "Meta pseudo labels" has ~10% less errors, while having less parameters. https://paperswithcode.com/sota/image-classification-on-imagenet https://paperswithcode.com/sota/image-classification-on-imag... However the fast training is an interesting property. It would be interesting to test thoses efficientNets with zeroth order backpropagation as it allows a 300X speedup (vs 8.7x) while not regressing accuracy that much https://paperswithcode.com/paper/zorb-a-derivative-free-backpropagation https://paperswithcode.com/paper/zorb-a-derivative-free-back...

nl 6y ago

Make it work then make it fast and efficient. People complaining about how slow and expensive brand new models are to train are ignorant of the history of machine learning and of engineering in general.

eggie5 6y ago

Do what do attribute to the gains? Adaptive clipping? Or $$$ spent on NAS??

rsfern 6y ago
They do a little study in section 4.1 comparing batchnorm to adaptive gradient clipping for resnets over a range of hyperparameters, and they also compare perf to batchnorm versions in table 6. The results indicate AGC does give a real boost over batchnorm They do a bunch of manual hyperparameter tuning that seems necessary to get the state of the art results, from my reading it doesn’t seem like they actually used NAS. Just that the baseline they compare to was found with NAS

FL33TW00D 6y ago

As distributed training becomes the norm, the lack of BN will become more and more desirable.

jjssmith 6y ago

Rediscovering NLMS now, good for ML folks. https://en.wikipedia.org/wiki/Least_mean_squares_filter#Normalized_least_mean_squares_filter_(NLMS) https://en.wikipedia.org/wiki/Least_mean_squares_filter#Norm...

Yenrabbit 6y ago

Yannic Kilcher has a great video on this out already, including some thoughtful critique. https://youtu.be/rNkHjZtH0RQ https://youtu.be/rNkHjZtH0RQ One of the things he mentioned was that they introduce this fancy alternative to BatchNorm AND come up with a fancy new architecture. The combination does really well, but it isn't clear how much of the gain is due to the new improved architecture Vs the 'adaptive gradient clipping' they introduce.

diptanu 6y ago

Is there an implementation for what they are describing in PyTorch?

rxsel 6y ago

Is there any video work being done in the realm of image processing? I mean a video is just essentially an array/string of images/frames no?