3 ms·
ML training is a latency sensitive problem that benefits from locality: most of the learning algorithms wait for all parallel workers to send gradients back bef
by krasin 5y ago
ML training is a latency sensitive problem that benefits from locality: most of the learning algorithms wait for all parallel workers to send gradients back before updating weights. While workarounds exist, life is easier without them.
- Mizza 5y agoWhy is this a necessary constraint? (Not trolling, just curious if this has been investigated.) Our neurons don't have a global clock, it doesn't seem to be a problem for us. My intuition is that as long as the input is changing continuously and not through random presentation, and at a rate where big changes happen at a lower order of magnitude than the average compute rate, it wouldn't matter all that much in terms of accuracy.