4 ms·

GPT-NeoX is an example project that is using deepspeed and Zero-3 offloading. The wider project intend to train a GPT-3 sized model and release it freely to the

by joshlk 6y ago

GPT-NeoX is an example project that is using deepspeed and Zero-3 offloading. The wider project intend to train a GPT-3 sized model and release it freely to the world. https://github.com/EleutherAI/gpt-neox https://github.com/EleutherAI/gpt-neox

ma2rten 6y ago
It seems like Zero-3 doesn't work for them: https://github.com/EleutherAI/gpt-neox/issues/171 https://github.com/EleutherAI/gpt-neox/issues/171
- dqpb 6y ago
  Did you even read through the issue? I don't see anything that indicates it won't work.
  - ma2rten 6y ago
    Yes, I did. The last comment is a traceback and an explanation what would have to be done to fix it.
    - minimaxir 6y ago
      Your comment implied it's not possible at all for them to use it, not that it's currently not working.
      - ma2rten 6y ago
        I guess an ambiguity in the English language is at fault here. I meant it literally doesn't work for them, and not the idiom.
- joshlk 6y ago
  Looks like they got it working recently https://github.com/EleutherAI/gpt-neox/pull/178 https://github.com/EleutherAI/gpt-neox/pull/178
- stellaathena 6y ago
  Hi! I’m the one who wrote this code. My ZeRO-3 implementation is currently not working, but I’ve spoken with DeepSpeed devs and they’ve explained to me what I’ve been doing wrong. I haven’t had time to implement the fix but I don’t see any reason to assume it won’t work. https://github.com/microsoft/DeepSpeed/issues/846 https://github.com/microsoft/DeepSpeed/issues/846 Also, the specific problem described in that Issue was due to a bug I found in DeepSpeed that has since been corrected.