2 ms·
data bandwidth limits distributed training under current architectures. really interesting implications if we can make progress on that
by DesaiAshu 8mo ago
data bandwidth limits distributed training under current architectures. really interesting implications if we can make progress on that
- dogcomplex 8mo agoLimits but doesn't prohibit. See https://www.primeintellect.ai/blog/intellect-3 https://www.primeintellect.ai/blog/intellect-3 - still useful and can scale enormously. Takes a particular shape and relies heavily on RL, but still big.
- andoando 8mo agoWhat bandwith limits? Im assuming the forward and backward passes have to be done sequentially?
- DesaiAshu 7mo agoYes also passing data within each layer