3 ms·
Zero-Offload: Democratizing Billion-Scale Model Training
- coolspot 6y ago> ZeRO-Offload changes the large model training landscape by making large model training accessible to nearly everyone. It can train models with over 13 billion parameters on a single GPU, a 10x increase in size compared to popular framework such as PyTorch, and it does so without requiring any model change from the data scientists or sacrificing computational efficiency. Official open-source implementation: https://www.deepspeed.ai/ https://www.deepspeed.ai/