4 ms·
The sharding done for turbo models has almost certainly affected the attention mechanisms. It makes me think the sharding was done at the layer level.
by spdustin 3y ago
The sharding done for turbo models has almost certainly affected the attention mechanisms. It makes me think the sharding was done at the layer level.