4 ms·
data bandwidth limits distributed training under current architectures. really interesting implications if we can make progress on that
by DesaiAshu 6mo ago
data bandwidth limits distributed training under current architectures. really interesting implications if we can make progress on that
- dogcomplex 6mo agoLimits but doesn't prohibit. See https://www.primeintellect.ai/blog/intellect-3 https://www.primeintellect.ai/blog/intellect-3 - still useful and can scale enormously. Takes a particular shape and relies heavily on RL, but still big.
- andoando 6mo agoWhat bandwith limits? Im assuming the forward and backward passes have to be done sequentially?
- DesaiAshu 6mo agoYes also passing data within each layer