3 ms·
The compare against “DeepSpeed ZeRO-3” apparently.
by bee_rider 7mo ago
The compare against “DeepSpeed ZeRO-3” apparently.
- jazzpush2 7mo agoFWIW Zero-3 refers to a common strategy for sharding model components across GPUs (commonly called FSDP-2, Full Sharded Data Parallel). The "3" is the level of sharding (how much stuff to distribute across GPUs, e.g. just weights, versus optimizer state as well, etc.)