4 ms·
Does it leverage deepspeed/zero 3?
by The_rationalist 6y ago
Does it leverage deepspeed/zero 3?
- minimaxir 6y agoThat’s PyTorch only; the current models are TensorFlow.
- The_rationalist 6y agoOh that's unfortunate, can't the models be exported to pytorch through e.g onnx?
- machinevision 6y agoEven small models can be a headache to export if they have even they use anything custom. I can't even imagine something the size of GPT-3.
- Voloskaya 6y agoLarger models aren't really more complicated than smaller ones though. GPT-2 is already supported, I believe only difference with GPT-3 is sparse attention.
- minimaxir 6y agoThe GPT-2 models included with Transformers can export to ONNX fine w/ a helper included with the Python onnxruntime.
- minimaxir 6y agoTo run in PyTorch the model architecture must be ported. ONNX is slightly different; you could export the model to run in the onnxruntime but that has tradeoffs.
- stellaathena 6y agoThere's a PyTorch + DeepSpeed repository here: https://github.com/EleutherAI/gpt-neox https://github.com/EleutherAI/gpt-neox
- MasterScrat 6y agoGPT-NeoX, which is a model from the same group but using GPUs instead of TPUs, uses techniques from DeepSpeed: https://github.com/EleutherAI/gpt-neox/ https://github.com/EleutherAI/gpt-neox/