3 ms·

Huge fan of your gpt2-simple library, which I used to train a satirical news generator in a Colab notebook: https://colab.research.google.com/drive/1buF7Tju3DkZ

Huge fan of your gpt2-simple library, which I used to train a satirical news generator in a Colab notebook: https://colab.research.google.com/drive/1buF7Tju3DkZeL-EV4FtSu2KEJ4yepbL0?usp=sharing https://colab.research.google.com/drive/1buF7Tju3DkZeL-EV4Ft... > Generates text faster than gpt-2-simple and with better memory efficiency! (even from the 1.5B GPT-2 model!) This is exciting news. One of very few drawbacks of gpt2-simple is the inability to fine-tune a model of more than ~355M parameters. Do these memory management improvements make it possible to fine-tune a larger one?

minimaxir 6y ago
> Do these memory management improvements make it possible to fine-tune a larger one? Unfortunately not yet; I need to implement gradient checkpointing first. Memory-wise, the results for finetuning 124M are promising (<8 GB VRAM when it used to take about 12 GB VRAM with gpt-2-simple)
- britmob 6y ago
  Have I been using gpt-2-simple wrong..? I’ve been fine-tuning 355M on a 8GB 1080 for months..
  - minimaxir 6y ago
    gpt-2-simple has gradient checkpointing; aitextgen does not (yet).