3 ms·
Just fine. GPT-Neo-125m can generate ~200 tokens in 10 seconds. Larger models like 1.3b use more memory (~6gb) and take longer (~20-30 seconds), but produces so
by smoldesu 4y ago
Just fine. GPT-Neo-125m can generate ~200 tokens in 10 seconds. Larger models like 1.3b use more memory (~6gb) and take longer (~20-30 seconds), but produces somewhat more coherant answers.
Seriously though, the answer is "better than you think". Go boot up PyTorch and see for yourself!
- ricopags 4y agoChatGPT is built on GPT-3 which is 175 billion parameters or roughly 300GiB VRAM. Significant difference between smaller models.
- smoldesu 4y agoYep, I'm merely pointing out that pruned GPT models do exist, and work on laptops. The technology does scale.
- verdverm 4y agoChatGPT is GPT-3++, the one people are thinking of is 175B parameters, post-trained on a very curated dataset to get the chat like behavior At the local speeds, my time is worth more than what OpenAI charges
- smoldesu 4y agoThat's fine, but it's also a far-cry from "GPT doesn't run on your laptop". They're asking about how it scales, I'm giving them an answer. Yes, you can also use API requests to get the same results, but that's not running locally.
- verdverm 4y agoOP asked about ChatGPT, which is not GPT-2, it is GPT-3.5 based with a secondary training regime