13 ms·
Lit-GPT is what I have been using to pretrain models at home: https://github.com/Lightning-AI/litgpt https://github.com/Lightning-AI/litgpt Using the openwebtex
by buildbot 3y ago
Lit-GPT is what I have been using to pretrain models at home: https://github.com/Lightning-AI/litgpt https://github.com/Lightning-AI/litgpt
Using the openwebtext example, I can train a 700M param model to 2.6 loss in a few days on dual 4090s. Pretty awesome!