4 ms·

How fast would ChatGPT run on a laptop locally?

3 points by owenpalmer 4y ago

Obviously it depends on the laptop, but that's not the point. Would it be extremely fast? Extremely slow?

deleted 4y ago
[deleted]
smoldesu 4y ago
Just fine. GPT-Neo-125m can generate ~200 tokens in 10 seconds. Larger models like 1.3b use more memory (~6gb) and take longer (~20-30 seconds), but produces somewhat more coherant answers. Seriously though, the answer is "better than you think". Go boot up PyTorch and see for yourself!
- ricopags 4y ago
  ChatGPT is built on GPT-3 which is 175 billion parameters or roughly 300GiB VRAM. Significant difference between smaller models.
  - smoldesu 4y ago
    Yep, I'm merely pointing out that pruned GPT models do exist, and work on laptops. The technology does scale.
- verdverm 4y ago
  ChatGPT is GPT-3++, the one people are thinking of is 175B parameters, post-trained on a very curated dataset to get the chat like behavior At the local speeds, my time is worth more than what OpenAI charges
  - smoldesu 4y ago
    That's fine, but it's also a far-cry from "GPT doesn't run on your laptop". They're asking about how it scales, I'm giving them an answer. Yes, you can also use API requests to get the same results, but that's not running locally.
    - verdverm 4y ago
      OP asked about ChatGPT, which is not GPT-2, it is GPT-3.5 based with a secondary training regime
verdverm 4y ago
It's quite unlikely that you would even be able to run it at all. The model is quite large and probably won't fit in memory. If you could get it to run, it would be extremely slow Worth checking out: https://github.com/karpathy/nanoGPT https://github.com/karpathy/nanoGPT The associated videos will go into this more, iirc, at the end he said the real GPT-2 would take 5-20 seconds to generate a small amount of text using CPU only.
ricopags 4y ago
It wouldn't[0], at all[1] [0] https://bdtechtalks.com/2020/09/21/gpt-3-economy-business-model/#:~:text=The%20cost%20of%20running%20GPT%2D3 https://bdtechtalks.com/2020/09/21/gpt-3-economy-business-mo... [1] https://lifearchitect.ai/chatgpt/#local https://lifearchitect.ai/chatgpt/#local
guluarte 4y ago
the question depends on how much data are you going to use it with

3 points by owenpalmer 4y ago

Obviously it depends on the laptop, but that's not the point. Would it be extremely fast? Extremely slow?

deleted 4y ago

[deleted]

smoldesu 4y ago

Just fine. GPT-Neo-125m can generate ~200 tokens in 10 seconds. Larger models like 1.3b use more memory (~6gb) and take longer (~20-30 seconds), but produces somewhat more coherant answers. Seriously though, the answer is "better than you think". Go boot up PyTorch and see for yourself!

ricopags 4y ago
ChatGPT is built on GPT-3 which is 175 billion parameters or roughly 300GiB VRAM. Significant difference between smaller models.
- smoldesu 4y ago
  Yep, I'm merely pointing out that pruned GPT models do exist, and work on laptops. The technology does scale.
verdverm 4y ago
ChatGPT is GPT-3++, the one people are thinking of is 175B parameters, post-trained on a very curated dataset to get the chat like behavior At the local speeds, my time is worth more than what OpenAI charges
- smoldesu 4y ago
  That's fine, but it's also a far-cry from "GPT doesn't run on your laptop". They're asking about how it scales, I'm giving them an answer. Yes, you can also use API requests to get the same results, but that's not running locally.
  - verdverm 4y ago
    OP asked about ChatGPT, which is not GPT-2, it is GPT-3.5 based with a secondary training regime

verdverm 4y ago

It's quite unlikely that you would even be able to run it at all. The model is quite large and probably won't fit in memory. If you could get it to run, it would be extremely slow Worth checking out: https://github.com/karpathy/nanoGPT https://github.com/karpathy/nanoGPT The associated videos will go into this more, iirc, at the end he said the real GPT-2 would take 5-20 seconds to generate a small amount of text using CPU only.

ricopags 4y ago

It wouldn't[0], at all[1] [0] https://bdtechtalks.com/2020/09/21/gpt-3-economy-business-model/#:~:text=The%20cost%20of%20running%20GPT%2D3 https://bdtechtalks.com/2020/09/21/gpt-3-economy-business-mo... [1] https://lifearchitect.ai/chatgpt/#local https://lifearchitect.ai/chatgpt/#local

guluarte 4y ago

the question depends on how much data are you going to use it with