3 ms·

No. We weren't sure if that'd be a good idea since it wasn't trained with low-precision, and 345M thankfully didn't require going that far. 744M might, though.

by gwern 7y ago

No. We weren't sure if that'd be a good idea since it wasn't trained with low-precision, and 345M thankfully didn't require going that far. 744M might, though. (Another option is model parallelism since I have 2 GPUs and that might be enough, perhaps freezing more layers and training incrementally, or reducing the 1024 token window to smaller ones like 700.)

slashcom 7y ago
fp16 saves a lot of memory and is worth doing. I've not had trouble fine tuning all these models with fp16.
- sdan 7y ago
  Have you fine tuned 774 successfully using a single GPU?
p1esk 7y ago
I recommend Nvidia Apex, it offers several ways to mix precision.
The_rationalist 7y ago
You maybe should try tensorflow automatic mixed precision! https://github.com/zihangdai/xlnet/pull/200 https://github.com/zihangdai/xlnet/pull/200
JonathanFly 7y ago
On the NVIDIA GPT- 2 implementation: >What would be the largest model one could train across 2x 2080Ti? >~800M gpt2. this is largely due to the memory required to House parameters + optimizer states. If one uses a smaller optimizer than Adam training something larger should be possible. Make sure to turn on activation checkpointing with —checkpoint-activations https://twitter.com/TheRealRPuri/status/1161322580126470145 https://twitter.com/TheRealRPuri/status/1161322580126470145
- gwern 7y ago
  They haven't released such models, though, and I don't know if it would be drop-in compatible with the OA GPT-2-774M checkpoint (they're training their own GPT-2s using their own webtext corpus).
  - JonathanFly 7y ago
    I haven't look into at all myself, but he also said: >We do provide training code that should work out of the box for gpt2 117M/345M https://twitter.com/TheRealRPuri/status/1161319745259393024 https://twitter.com/TheRealRPuri/status/1161319745259393024
    - p1esk 7y ago
      It would take forever (or $$$) to train even 117M model from scratch.
      - JonathanFly 7y ago
        I read that meaning you can start with the actual pre-trained GPT-2 models but I never got an answer when I specifically asked if that was the case.

by gwern 7y ago

slashcom 7y ago

fp16 saves a lot of memory and is worth doing. I've not had trouble fine tuning all these models with fp16.

sdan 7y ago
Have you fine tuned 774 successfully using a single GPU?

p1esk 7y ago

I recommend Nvidia Apex, it offers several ways to mix precision.

The_rationalist 7y ago

You maybe should try tensorflow automatic mixed precision! https://github.com/zihangdai/xlnet/pull/200 https://github.com/zihangdai/xlnet/pull/200

JonathanFly 7y ago

On the NVIDIA GPT- 2 implementation: >What would be the largest model one could train across 2x 2080Ti? >~800M gpt2. this is largely due to the memory required to House parameters + optimizer states. If one uses a smaller optimizer than Adam training something larger should be possible. Make sure to turn on activation checkpointing with —checkpoint-activations https://twitter.com/TheRealRPuri/status/1161322580126470145 https://twitter.com/TheRealRPuri/status/1161322580126470145

gwern 7y ago
They haven't released such models, though, and I don't know if it would be drop-in compatible with the OA GPT-2-774M checkpoint (they're training their own GPT-2s using their own webtext corpus).
- JonathanFly 7y ago
  I haven't look into at all myself, but he also said: >We do provide training code that should work out of the box for gpt2 117M/345M https://twitter.com/TheRealRPuri/status/1161319745259393024 https://twitter.com/TheRealRPuri/status/1161319745259393024
  - p1esk 7y ago
    It would take forever (or $$$) to train even 117M model from scratch.
    - JonathanFly 7y ago
      I read that meaning you can start with the actual pre-trained GPT-2 models but I never got an answer when I specifically asked if that was the case.