3 ms·
It's not really all that useful for AI training. The reason why energy is a problem for AI training is that you need a lot of compute at one place, distributing
by macrolime 2y ago
It's not really all that useful for AI training. The reason why energy is a problem for AI training is that you need a lot of compute at one place, distributing backpropagation doesn't work that well. When the energy problems hit, it's when the datacenter need exceeds the capacity of nearby power plants.
If we want an alternative I think we'd need to try alternative training algorithms that are easier to distribute. Predictive coding for example can approximate backprop, but requires more compute. Something in that direction might work https://arxiv.org/abs/2006.04182 https://arxiv.org/abs/2006.04182