3 ms·

ChatGPT can run on our laptops? How? The only thing I could find was that it can't because it requires 128 GB VRAM.

recuter 4y ago
https://github.com/ggerganov/llama.cpp/issues/39 https://github.com/ggerganov/llama.cpp/issues/39 40GB for the big llama.
- FooBarWidget 4y ago
  But that's not ChatGPT?
  - ShamelessC 4y ago
    No one said ChatGPT. It was a vague reference to the recent LLaMA developments from the community.
  - arlcode 4y ago
    Not it isn't but what the OP is saying that with the llama models (and others) being runable on consumer hardware it's only a matter of time until we get something that's "good enough" for many use cases. Maybe not first of the class but there is a lot value to be found in the middle ground and it's good for the market as a whole.
  - deleted 4y ago
    [deleted]
- karmasimida 4y ago
  Even biggest Llama is like 1/3 size of ChatGPT
  - recuter 4y ago
    Might as well compare it to AlpacaGPT, they are trained different. Carefully prompted the output shows flashes of excellence and really whips ass and it is still early days. We have almost closed the gap in no time whereas on ChatGPT day 1 it seemed they were years ahead. Hopefully there will be more models and developments.
ly3xqhl8g9 4y ago
LLaMa [1] works on Apple Mx processors due to unified memory. The 7B model works on a 8GB Apple MacBook Pro M1, ~40 seconds/token. [1] https://dev.l1x.be/posts/2023/03/12/using-llama-with-m1-mac https://dev.l1x.be/posts/2023/03/12/using-llama-with-m1-mac
- recuter 4y ago
  I get 220ms/token on an M1 8GB Macbook Air, so ChatGPT speeds. on a 32GB M1 Max it babbles faster than my ex-gf.
  - ly3xqhl8g9 4y ago
    Weird, I ran it alongside quite a few other processes, didn't expect to even be able to run and didn't bother to optimize it in any meaningful way.
mach1ne 4y ago
Not ChatGPT, but LLaMa. Not as good as ChatGPT, but seeing the pace of progress when performance has been demonstrated, we'll have ChatGPT running on our laptops by summer. End of the year at the latest.

https://github.com/ggerganov/llama.cpp/issues/39 https://github.com/ggerganov/llama.cpp/issues/39 40GB for the big llama.

FooBarWidget 4y ago
But that's not ChatGPT?
- ShamelessC 4y ago
  No one said ChatGPT. It was a vague reference to the recent LLaMA developments from the community.
- arlcode 4y ago
  Not it isn't but what the OP is saying that with the llama models (and others) being runable on consumer hardware it's only a matter of time until we get something that's "good enough" for many use cases. Maybe not first of the class but there is a lot value to be found in the middle ground and it's good for the market as a whole.
- deleted 4y ago
  [deleted]
karmasimida 4y ago
Even biggest Llama is like 1/3 size of ChatGPT
- recuter 4y ago
  Might as well compare it to AlpacaGPT, they are trained different. Carefully prompted the output shows flashes of excellence and really whips ass and it is still early days. We have almost closed the gap in no time whereas on ChatGPT day 1 it seemed they were years ahead. Hopefully there will be more models and developments.

ly3xqhl8g9 4y ago

LLaMa [1] works on Apple Mx processors due to unified memory. The 7B model works on a 8GB Apple MacBook Pro M1, ~40 seconds/token. [1] https://dev.l1x.be/posts/2023/03/12/using-llama-with-m1-mac https://dev.l1x.be/posts/2023/03/12/using-llama-with-m1-mac

recuter 4y ago
I get 220ms/token on an M1 8GB Macbook Air, so ChatGPT speeds. on a 32GB M1 Max it babbles faster than my ex-gf.
- ly3xqhl8g9 4y ago
  Weird, I ran it alongside quite a few other processes, didn't expect to even be able to run and didn't bother to optimize it in any meaningful way.

mach1ne 4y ago

Not ChatGPT, but LLaMa. Not as good as ChatGPT, but seeing the pace of progress when performance has been demonstrated, we'll have ChatGPT running on our laptops by summer. End of the year at the latest.