4 ms·
Dang only forward passes. The real secret was in the backward pass! I was also curious to learn how they implemented the dualpipe scheduler
by imranq 2y ago
Dang only forward passes. The real secret was in the backward pass! I was also curious to learn how they implemented the dualpipe scheduler
- rfoo 2y agoDo they even have an optimized backward? It looks like optimizations like this aren't needed during training. Their V2 paper also suggests so.