3 ms·

Ignoring the licensing issues, there are a few other constraints that would make the model harder to go viral outside of developers who spend a lot of time in t

Ignoring the licensing issues, there are a few other constraints that would make the model harder to go viral outside of developers who spend a lot of time in this space already: 1) Model weights are heavy for just experimentation, although quantizing it down to 4-bit might make them on par with SD FP16. 2) Requires extreme CLI shenanigans (and likely configuration since you have to run make) compared to just running a Colab Notebook or a .bat Windows Installer for the A1111 UI. 3) Again hardware: a M1 Pro or a RTX 4090 is not super common among people who are just curious about text generation. 4) It is possible the extreme quantization could be affecting text output quality; although the examples are coherent for simple queries, more complex GPT-3-esque queries might become relatively incoherent. Particularly with ChatGPT and its cheap API (timely!) out now such that even nontechies have a strong baseline on good output already. The viral moment for SD was that it was easy to use and it was a significant quality leap over VQGAN + CLIP. I was going to say inference speed since that's usually another constraint for new LLMs but given the 61.41 ms/token cited for the 7B model in the repo/your GIF, that seems on par with the inference speed from OPT-6.7B FP16 in transformers on a T4. Some of these caveats are fixable, but even then I don't think LLaMA will have its Stable Diffusion moment.

simonw 4y ago
The 4-bit quantized models are 4GB for 7B and 8GB for 13B. I'm not too worried about CLI shenanigans, because of what happened with whisper.cpp - it resulted in apps like https://goodsnooze.gumroad.com/l/macwhisper https://goodsnooze.gumroad.com/l/macwhisper - wouldn't be at all surprised to see the same happen with llama.cpp A regular M1 with 8GB of RAM appears to be good enough to run that 7B model. I wonder at what point it will run on an iPhone... the Stable Diffusion model was 4GB when they first released it, and that runs on iOS now after some more optimization tricks. For me though, the "Stable Diffusion" moment isn't necessarily about the LLaMA model itself. It's not licensed for commercial use, so it won't see nearly the same level of things built on top of it. The key moment for me is that I've now personally seen a GPT-3 scale model running on my own personal laptop. I know it can be done! Now I just need to wait for the inevitable openly-licensed, instruction-tuned model that runs on the same hardware. It's that, but also the forthcoming explosion of developer innovation that a local model will unleash. llama.cpp is just the first hint of that.
- smoldesu 4y ago
  > The key moment for me is that I've now personally seen a GPT-3 scale model running on my own personal laptop. I hate to pooh-pooh it for everyone, but this was possible before LLaMa. GPT-J-125m/6b have been around for a while, and are frankly easier to install and get results out of. The smaller pruned model even fits on an iPhone. The problem is more that these smaller models won't ever compete with GPT-scale APIs. Tomorrow's local LLaMa might beat yesterday's ChatGPT, but I think those optimistic for the democratization of chatbot intelligence are setting their hopes a bit high. LLaMa really isn't breaking new ground.
  - stonebraker 4y ago
    Has anyone compared performance between LLaMa.cpp vs Alpaca.cpp?
  - simonw 4y ago
    I'm not particularly interested in beating ChatGPT: I'm looking for a "calculator for words" which I can use for things like summarization, term extraction, text rephrasing etc - maybe translation between languages too. There are all kinds of things I want to be able to do with a LLM that are a lot tighter than general chatbots. I'd love to see a demo of GPT-J on an iPhone!
    - sroussey 4y ago
      In a decade, these models will be baked into a SOC.
  - Vetch 4y ago
    There was only ever a 6B GPT-J, you must be thinking of GPT-neo's for smaller sizes. GPT-J was the best of its kind for a long while but even just the 7b version of LLaMa soundly surpasses it in how well it follows examples to solve problems and do useful work like summarization, keyword extraction and a mini search agent. Being able to run something on your laptop that matches the original davinci GPT3 and can be tuned to go beyond is nothing to sniff at. The best models will be by API but there is a threshold of utility for LLMs, below which they're just bad story telling toys, that hasn't been available for home users until recently, with FlanT5s and LLaMas.

simonw 4y ago

The 4-bit quantized models are 4GB for 7B and 8GB for 13B. I'm not too worried about CLI shenanigans, because of what happened with whisper.cpp - it resulted in apps like https://goodsnooze.gumroad.com/l/macwhisper https://goodsnooze.gumroad.com/l/macwhisper - wouldn't be at all surprised to see the same happen with llama.cpp A regular M1 with 8GB of RAM appears to be good enough to run that 7B model. I wonder at what point it will run on an iPhone... the Stable Diffusion model was 4GB when they first released it, and that runs on iOS now after some more optimization tricks. For me though, the "Stable Diffusion" moment isn't necessarily about the LLaMA model itself. It's not licensed for commercial use, so it won't see nearly the same level of things built on top of it. The key moment for me is that I've now personally seen a GPT-3 scale model running on my own personal laptop. I know it can be done! Now I just need to wait for the inevitable openly-licensed, instruction-tuned model that runs on the same hardware. It's that, but also the forthcoming explosion of developer innovation that a local model will unleash. llama.cpp is just the first hint of that.

smoldesu 4y ago
> The key moment for me is that I've now personally seen a GPT-3 scale model running on my own personal laptop. I hate to pooh-pooh it for everyone, but this was possible before LLaMa. GPT-J-125m/6b have been around for a while, and are frankly easier to install and get results out of. The smaller pruned model even fits on an iPhone. The problem is more that these smaller models won't ever compete with GPT-scale APIs. Tomorrow's local LLaMa might beat yesterday's ChatGPT, but I think those optimistic for the democratization of chatbot intelligence are setting their hopes a bit high. LLaMa really isn't breaking new ground.
- stonebraker 4y ago
  Has anyone compared performance between LLaMa.cpp vs Alpaca.cpp?
- simonw 4y ago
  I'm not particularly interested in beating ChatGPT: I'm looking for a "calculator for words" which I can use for things like summarization, term extraction, text rephrasing etc - maybe translation between languages too. There are all kinds of things I want to be able to do with a LLM that are a lot tighter than general chatbots. I'd love to see a demo of GPT-J on an iPhone!
  - sroussey 4y ago
    In a decade, these models will be baked into a SOC.
- Vetch 4y ago
  There was only ever a 6B GPT-J, you must be thinking of GPT-neo's for smaller sizes. GPT-J was the best of its kind for a long while but even just the 7b version of LLaMa soundly surpasses it in how well it follows examples to solve problems and do useful work like summarization, keyword extraction and a mini search agent. Being able to run something on your laptop that matches the original davinci GPT3 and can be tuned to go beyond is nothing to sniff at. The best models will be by API but there is a threshold of utility for LLMs, below which they're just bad story telling toys, that hasn't been available for home users until recently, with FlanT5s and LLaMas.