4 ms·
They compare the training latency for different models with a fixed batch size of 32. But if the DeepMind models are several times larger than the comparison mo
by highfrequency 6y ago
They compare the training latency for different models with a fixed batch size of 32. But if the DeepMind models are several times larger than the comparison models in each latency class, it seems that the comparison models could use larger batch sizes for faster overall training time.
- trott 6y agoFor ConvNets, the memory use of the models themselves is pretty modest. For example, even with 0.5B parameters, with FP32, weights+gradients+momentum should use just 6GB (unless your framework sucks, or you have extra overhead from distributed training) So, if your model is twice smaller, you'll only save 3GB. If your VRAM is 32GB, saving 3GB won't let you use a much bigger batch size. On the other hand, the absence of batch norm can actually lead to memory savings proportional to batch size.