12 ms·

Currently right now there's too many caveats to run even the 7B model per the workflows mentioned in the article. The big difference between it and Stable Diff

Currently right now there's too many caveats to run even the 7B model per the workflows mentioned in the article. The big difference between it and Stable Diffusion which caused the latter to go megaviral is a) it can run on a typical GPU that gamers likely already have without hitting a perf ceiling and b) it can run easily in a free Colab GPU. Although Hugging Face transformers can run a 7B model on a T4 GPU w/ 8-bit loading, but with its own caveats too. There's a big difference between "can run" and "can run well". VQGAN + CLIP had a lot of friction too and that's partially why AI image generation didn't go megaviral then.

simonw 4y ago
By caveats do you mean the licensing terms or the difficulty of prompting the model? Unless it's relicensed I don't expect LLaMA to be a long-term foundation model. But it's shown that yes, you can run a GPT-3 class model on an M1 Mac with 8GB of RAM (or maybe 16GB for the 13B one?) I fully expect other models to follow, from other organizations, with better capabilities and more friendly licensing terms.
- zamnos 4y ago
  But is anyone actually making money off of StableDiffusion? Maybe the shovel-sellers (runpod.io et al), but afaik no one using it as the foundation for a revenue generating company. I ask, because yes, technically, you can't get LLaMA legally unless you're a researcher and get it directly from Facebook. But that's not going to stop the faithful from finding a copy and working on it.
  - logifail 4y ago
    > is anyone actually making money off of StableDiffusion? We're all still waiting to hear about (non-shovel-selling) successes in this space.
    - pmoriarty 4y ago
      I don't know about Stable Diffusion in particular, but three examples of AI-generated art making money immediately spring to mind: 1 - some guy won hundreds of dollars in an art contest from AI generated art (and this made big news, so it should be easy to find) 2 - one person reported using midjourney's images as a starting point for images that wound up being used in a physical magazine 3 - another artist has used midjourney images that they modify to sell in all sorts of contexts (like background images on stock illustration sites) You'd probably find many other examples in midjourney's #in-the-world discord channel. I'd also be shocked if stock image sites, clipart sites and freelance design/illustration sites weren't already flooded with AI generated images that have been sold for money. That being said, because high questly AI-generated images are so easy to make, the value of images of all types is likely to plummet soon if it hasn't already.
      - dragonwriter 4y ago
        > the value of images of all types is likely to plummet soon if it hasn't already. The price might, but the marginal value having them or not has to other products which incorporate them likely won’t as much. So, say, if you are a TTRPG publisher, the value of art to your business doesn't change, but the cost of moving farther along the no art -> scattered sketches & tangentially relevant public domain works -> ideal quantity of art, specific to the product, goes down. So, you are, in effect, making money on SD.
    - dragonwriter 4y ago
      Ultimately, its an art tool — a shovel — ecosystem that is democratized enough that it is dominated by high-quality, end-user usable components. Making money with it in the non-shovel-selling sense is going to be commercial artists (or other people with visual image needs, perhaps avoiding the need for commercial artists) incorporating it in production workflows. But except when they are also selling shovels (e.g., Pixar with Renderman) there’s not a whole lot of incentive to trumpet workflow components rather than product in that space.
      - logifail 4y ago
  - simonw 4y ago
    I believe Midjourney may have used bits of Stable Diffusion in their product, which is definitely profitable.
  - nomdep 4y ago
    > But is anyone actually making money off of StableDiffusion? How about: - https://www.deepagency.com/ https://www.deepagency.com/ - https://avatarai.me/ https://avatarai.me/
  - blueblimp 4y ago
    NovelAI's image generation uses a fine-tune of Stable Diffusion.
  - vintermann 4y ago
    If anyone is making net money on Stable Diffusion already, they're either small scale or not going to be around for long, probably both. Anything in it for the longer term is going to be in the buildup phase.
- minimaxir 4y ago
  Ignoring the licensing issues, there are a few other constraints that would make the model harder to go viral outside of developers who spend a lot of time in this space already: 1) Model weights are heavy for just experimentation, although quantizing it down to 4-bit might make them on par with SD FP16. 2) Requires extreme CLI shenanigans (and likely configuration since you have to run make) compared to just running a Colab Notebook or a .bat Windows Installer for the A1111 UI. 3) Again hardware: a M1 Pro or a RTX 4090 is not super common among people who are just curious about text generation. 4) It is possible the extreme quantization could be affecting text output quality; although the examples are coherent for simple queries, more complex GPT-3-esque queries might become relatively incoherent. Particularly with ChatGPT and its cheap API (timely!) out now such that even nontechies have a strong baseline on good output already. The viral moment for SD was that it was easy to use and it was a significant quality leap over VQGAN + CLIP. I was going to say inference speed since that's usually another constraint for new LLMs but given the 61.41 ms/token cited for the 7B model in the repo/your GIF, that seems on par with the inference speed from OPT-6.7B FP16 in transformers on a T4. Some of these caveats are fixable, but even then I don't think LLaMA will have its Stable Diffusion moment.
  - simonw 4y ago
    The 4-bit quantized models are 4GB for 7B and 8GB for 13B. I'm not too worried about CLI shenanigans, because of what happened with whisper.cpp - it resulted in apps like https://goodsnooze.gumroad.com/l/macwhisper https://goodsnooze.gumroad.com/l/macwhisper - wouldn't be at all surprised to see the same happen with llama.cpp A regular M1 with 8GB of RAM appears to be good enough to run that 7B model. I wonder at what point it will run on an iPhone... the Stable Diffusion model was 4GB when they first released it, and that runs on iOS now after some more optimization tricks. For me though, the "Stable Diffusion" moment isn't necessarily about the LLaMA model itself. It's not licensed for commercial use, so it won't see nearly the same level of things built on top of it. The key moment for me is that I've now personally seen a GPT-3 scale model running on my own personal laptop. I know it can be done! Now I just need to wait for the inevitable openly-licensed, instruction-tuned model that runs on the same hardware. It's that, but also the forthcoming explosion of developer innovation that a local model will unleash. llama.cpp is just the first hint of that.
tracyhenry 4y ago
Another big difference is quality of the results. Haven't tried myself but seen many complaints that it's nowhere near GPT-3 (at least for the 7B version). Correct me if I'm wrong!
- bestcoder69 4y ago
  13B feels on-par with the base non-instruction davinci. People might not realize how it was a bit trickier to prompt gpt3 when it first released.
- simonw 4y ago
  That doesn't bother me so much. GPT-3 had instruction tuning, which makes it MUCH easier to use. Now that I've seen that LLaMA can work I'm confident someone will release an openly licensed instruction-tuned model that works on the same hardware at some point soon. I also expect that there are prompt engineering tricks which can be used to get really great results out of LLaMA. I'm hoping someone will come up with a good prompt to get it to summarization, for example.
  - sp332 4y ago
    ChatGPT had an estimated 20,000 hours of human feedback. That’s not going to be easy to replicate in an open source way.
    - simonw 4y ago
      That's the next level up from instruction tuning though: that was the RLHF stuff, which was essential to make ChatGPT useful and safe enough to expose to a wide audience. For a model running on my own laptop I'm OK taking more risks. I'd like it to be able to obey simple instructions like "Summarize this text" or "Extract the names of everyone mentioned in this article" - I don't care as much about the stuff ChatGPT has to get right.
    - renewiltord 4y ago
      Perhaps we will provide feedback to open source Llama using ChatGPT. The cost to adjust the model is presumably what's hard?
bestcoder69 4y ago
Then this is SD for Apple silicon users. 13B runs on my m1 air at 200-300ms/token using llama.cpp. Outputs feel like original GPT-3, unlike any of the competitors I’ve tried. Granted- non-scientific first impressions.
- j45 4y ago
  Agreed. For those who have been quietly sitting with a base Mac Studio, or a reasonably capable Mac Mini. The possibilities changed on some fronts, but GPT's extremely low price on their API remains a good option.
  - aaomidi 4y ago
    Difference is chatgpt is not privacy friendly.
    - staticautomatic 4y ago
      Is it still not privacy friendly on Azure?
      - aaomidi 4y ago
        Azure has access to your queries. Running locally really is the only way of having a privacy friendly LLM.
        fastball 4y ago
        I wonder if there is some way you could do an E2EE LLM SaSS? Guess more work probably needs to be done into homomorphic encryption for that.
        jamiek88 4y ago
        This would actually be a great use case for homomorphic encryption. I’m behind on what the state of the art is in that though, but my mind immediately went there.
BulgarianIdiot 4y ago
The 65B model runs on a MacBook. So "currently" changes really fast, these days. https://twitter.com/ItIsFinch/status/1634643547189137409 https://twitter.com/ItIsFinch/status/1634643547189137409
- yieldcrv 4y ago
  wow, this makes me wonder if human intelligence, or just biological intelligence, is based on how many parameters an individual can float and access. I wonder if there are learnings for how to measure intelligence, just from proliferation of these AI models. Maybe even learnings for how to more efficiently access biological intelligence.
  - BulgarianIdiot 4y ago
    I used to think that to achieve human-level intelligence with AI, we need to have human-sized neural network. It was a conservative guess, but seeing where things are going, I now strongly believe we need a lot LESS parameters with AI, because AI is trained from literally nothing (from noise) on our current culture, from the Internet. It has no baggage, it's built from the ground up to be a modern 21st century human. We're not, we train decades to become ones. And struggle with it. Our genes are mostly tuned for our environment from thousands of years ago. A human brain carries a lot more baggage with it. Long story short, our biological brain hardware is much superior to current AI hardware, but our biological brain software (where we learn the culture that surrounds us) is much inferior to current AI machine learning. GPT-3 trained 175B on basically the whole internet in 15 days. That's insane. And now there are rumors that ChatGPT is executed on an optimized subset of this 175B model, which may be a bit more than 1/10 that size. I think going forward we'll see 20-30B parameter models perform much better than an average human in typical tasks. Especially after we evolve the topology of an AI to not be a straight directed acyclic graph, as it is now, but allow structures to loop internally, to invoke other structures with input (as subroutines) and so on. And also there's new hybrid analog/digital hardware in the pipeline, which makes AI execution 100 times more energy efficient and compact. So above-human level AI on your phone is on the horizon, I'd say. We're only getting started.
throwaway290 4y ago
You can run ChatGPT-J on a Colab or local: https://news.ycombinator.com/item?id=35101630 https://news.ycombinator.com/item?id=35101630
jerpint 4y ago
This is probably a few weeks of iteration away of being able to run on regular consumer hardware
Dalewyn 4y ago
>it can run on a typical GPU that gamers likely already have So a GTX 1060[1] or GTX 1650[1]? [1]: https://store.steampowered.com/hwsurvey/videocard/ https://store.steampowered.com/hwsurvey/videocard/
- chii 4y ago
  someone who only has a 1060 is what you'd call a casual gamer. A proper gamer would have a 4080, or at least a 3070, with the requisite 8gig of video ram.
  - Dalewyn 4y ago
    A casual gamer would be running with Intel or AMD integrated graphics on whatever laptop they can get for a reasonable sum. They might have a mobile RTX 3060 if they splurged a bit. More likely they'll just have a console to game on. Gamers with RTX 3070 or above are a small count, at least according to Steam. RTX 4080 gamers are a staggering 0.20% of those surveyed.
pilooch 4y ago
The big difference is there's no full training code for llama, yet.
EricLeer 4y ago
Next to being easier to run, I also have the feeling that images by them selves are also more interesting then generating text. It is much easier for us to look past the defaults, whereas for a text it has to be of a really high quality to be of notice (and even more if you want it to be interesting).

simonw 4y ago

By caveats do you mean the licensing terms or the difficulty of prompting the model? Unless it's relicensed I don't expect LLaMA to be a long-term foundation model. But it's shown that yes, you can run a GPT-3 class model on an M1 Mac with 8GB of RAM (or maybe 16GB for the 13B one?) I fully expect other models to follow, from other organizations, with better capabilities and more friendly licensing terms.

zamnos 4y ago
But is anyone actually making money off of StableDiffusion? Maybe the shovel-sellers (runpod.io et al), but afaik no one using it as the foundation for a revenue generating company. I ask, because yes, technically, you can't get LLaMA legally unless you're a researcher and get it directly from Facebook. But that's not going to stop the faithful from finding a copy and working on it.
- logifail 4y ago
  > is anyone actually making money off of StableDiffusion? We're all still waiting to hear about (non-shovel-selling) successes in this space.
  - pmoriarty 4y ago
    I don't know about Stable Diffusion in particular, but three examples of AI-generated art making money immediately spring to mind: 1 - some guy won hundreds of dollars in an art contest from AI generated art (and this made big news, so it should be easy to find) 2 - one person reported using midjourney's images as a starting point for images that wound up being used in a physical magazine 3 - another artist has used midjourney images that they modify to sell in all sorts of contexts (like background images on stock illustration sites) You'd probably find many other examples in midjourney's #in-the-world discord channel. I'd also be shocked if stock image sites, clipart sites and freelance design/illustration sites weren't already flooded with AI generated images that have been sold for money. That being said, because high questly AI-generated images are so easy to make, the value of images of all types is likely to plummet soon if it hasn't already.
    - dragonwriter 4y ago
      > the value of images of all types is likely to plummet soon if it hasn't already. The price might, but the marginal value having them or not has to other products which incorporate them likely won’t as much. So, say, if you are a TTRPG publisher, the value of art to your business doesn't change, but the cost of moving farther along the no art -> scattered sketches & tangentially relevant public domain works -> ideal quantity of art, specific to the product, goes down. So, you are, in effect, making money on SD.
  - dragonwriter 4y ago
    Ultimately, its an art tool — a shovel — ecosystem that is democratized enough that it is dominated by high-quality, end-user usable components. Making money with it in the non-shovel-selling sense is going to be commercial artists (or other people with visual image needs, perhaps avoiding the need for commercial artists) incorporating it in production workflows. But except when they are also selling shovels (e.g., Pixar with Renderman) there’s not a whole lot of incentive to trumpet workflow components rather than product in that space.
    - logifail 4y ago
- simonw 4y ago
  I believe Midjourney may have used bits of Stable Diffusion in their product, which is definitely profitable.
- nomdep 4y ago
  > But is anyone actually making money off of StableDiffusion? How about: - https://www.deepagency.com/ https://www.deepagency.com/ - https://avatarai.me/ https://avatarai.me/
- blueblimp 4y ago
  NovelAI's image generation uses a fine-tune of Stable Diffusion.
- vintermann 4y ago
  If anyone is making net money on Stable Diffusion already, they're either small scale or not going to be around for long, probably both. Anything in it for the longer term is going to be in the buildup phase.
minimaxir 4y ago
Ignoring the licensing issues, there are a few other constraints that would make the model harder to go viral outside of developers who spend a lot of time in this space already: 1) Model weights are heavy for just experimentation, although quantizing it down to 4-bit might make them on par with SD FP16. 2) Requires extreme CLI shenanigans (and likely configuration since you have to run make) compared to just running a Colab Notebook or a .bat Windows Installer for the A1111 UI. 3) Again hardware: a M1 Pro or a RTX 4090 is not super common among people who are just curious about text generation. 4) It is possible the extreme quantization could be affecting text output quality; although the examples are coherent for simple queries, more complex GPT-3-esque queries might become relatively incoherent. Particularly with ChatGPT and its cheap API (timely!) out now such that even nontechies have a strong baseline on good output already. The viral moment for SD was that it was easy to use and it was a significant quality leap over VQGAN + CLIP. I was going to say inference speed since that's usually another constraint for new LLMs but given the 61.41 ms/token cited for the 7B model in the repo/your GIF, that seems on par with the inference speed from OPT-6.7B FP16 in transformers on a T4. Some of these caveats are fixable, but even then I don't think LLaMA will have its Stable Diffusion moment.
- simonw 4y ago
  The 4-bit quantized models are 4GB for 7B and 8GB for 13B. I'm not too worried about CLI shenanigans, because of what happened with whisper.cpp - it resulted in apps like https://goodsnooze.gumroad.com/l/macwhisper https://goodsnooze.gumroad.com/l/macwhisper - wouldn't be at all surprised to see the same happen with llama.cpp A regular M1 with 8GB of RAM appears to be good enough to run that 7B model. I wonder at what point it will run on an iPhone... the Stable Diffusion model was 4GB when they first released it, and that runs on iOS now after some more optimization tricks. For me though, the "Stable Diffusion" moment isn't necessarily about the LLaMA model itself. It's not licensed for commercial use, so it won't see nearly the same level of things built on top of it. The key moment for me is that I've now personally seen a GPT-3 scale model running on my own personal laptop. I know it can be done! Now I just need to wait for the inevitable openly-licensed, instruction-tuned model that runs on the same hardware. It's that, but also the forthcoming explosion of developer innovation that a local model will unleash. llama.cpp is just the first hint of that.

tracyhenry 4y ago

Another big difference is quality of the results. Haven't tried myself but seen many complaints that it's nowhere near GPT-3 (at least for the 7B version). Correct me if I'm wrong!

bestcoder69 4y ago
13B feels on-par with the base non-instruction davinci. People might not realize how it was a bit trickier to prompt gpt3 when it first released.
simonw 4y ago
That doesn't bother me so much. GPT-3 had instruction tuning, which makes it MUCH easier to use. Now that I've seen that LLaMA can work I'm confident someone will release an openly licensed instruction-tuned model that works on the same hardware at some point soon. I also expect that there are prompt engineering tricks which can be used to get really great results out of LLaMA. I'm hoping someone will come up with a good prompt to get it to summarization, for example.
- sp332 4y ago
  ChatGPT had an estimated 20,000 hours of human feedback. That’s not going to be easy to replicate in an open source way.
  - simonw 4y ago
    That's the next level up from instruction tuning though: that was the RLHF stuff, which was essential to make ChatGPT useful and safe enough to expose to a wide audience. For a model running on my own laptop I'm OK taking more risks. I'd like it to be able to obey simple instructions like "Summarize this text" or "Extract the names of everyone mentioned in this article" - I don't care as much about the stuff ChatGPT has to get right.
  - renewiltord 4y ago
    Perhaps we will provide feedback to open source Llama using ChatGPT. The cost to adjust the model is presumably what's hard?

bestcoder69 4y ago

Then this is SD for Apple silicon users. 13B runs on my m1 air at 200-300ms/token using llama.cpp. Outputs feel like original GPT-3, unlike any of the competitors I’ve tried. Granted- non-scientific first impressions.

j45 4y ago
Agreed. For those who have been quietly sitting with a base Mac Studio, or a reasonably capable Mac Mini. The possibilities changed on some fronts, but GPT's extremely low price on their API remains a good option.
- aaomidi 4y ago
  Difference is chatgpt is not privacy friendly.
  - staticautomatic 4y ago
    Is it still not privacy friendly on Azure?
    - aaomidi 4y ago
      Azure has access to your queries. Running locally really is the only way of having a privacy friendly LLM.
      - fastball 4y ago
        I wonder if there is some way you could do an E2EE LLM SaSS? Guess more work probably needs to be done into homomorphic encryption for that.
        jamiek88 4y ago
        This would actually be a great use case for homomorphic encryption. I’m behind on what the state of the art is in that though, but my mind immediately went there.

BulgarianIdiot 4y ago

The 65B model runs on a MacBook. So "currently" changes really fast, these days. https://twitter.com/ItIsFinch/status/1634643547189137409 https://twitter.com/ItIsFinch/status/1634643547189137409

yieldcrv 4y ago
wow, this makes me wonder if human intelligence, or just biological intelligence, is based on how many parameters an individual can float and access. I wonder if there are learnings for how to measure intelligence, just from proliferation of these AI models. Maybe even learnings for how to more efficiently access biological intelligence.
- BulgarianIdiot 4y ago
  I used to think that to achieve human-level intelligence with AI, we need to have human-sized neural network. It was a conservative guess, but seeing where things are going, I now strongly believe we need a lot LESS parameters with AI, because AI is trained from literally nothing (from noise) on our current culture, from the Internet. It has no baggage, it's built from the ground up to be a modern 21st century human. We're not, we train decades to become ones. And struggle with it. Our genes are mostly tuned for our environment from thousands of years ago. A human brain carries a lot more baggage with it. Long story short, our biological brain hardware is much superior to current AI hardware, but our biological brain software (where we learn the culture that surrounds us) is much inferior to current AI machine learning. GPT-3 trained 175B on basically the whole internet in 15 days. That's insane. And now there are rumors that ChatGPT is executed on an optimized subset of this 175B model, which may be a bit more than 1/10 that size. I think going forward we'll see 20-30B parameter models perform much better than an average human in typical tasks. Especially after we evolve the topology of an AI to not be a straight directed acyclic graph, as it is now, but allow structures to loop internally, to invoke other structures with input (as subroutines) and so on. And also there's new hybrid analog/digital hardware in the pipeline, which makes AI execution 100 times more energy efficient and compact. So above-human level AI on your phone is on the horizon, I'd say. We're only getting started.

throwaway290 4y ago

You can run ChatGPT-J on a Colab or local: https://news.ycombinator.com/item?id=35101630 https://news.ycombinator.com/item?id=35101630

jerpint 4y ago

This is probably a few weeks of iteration away of being able to run on regular consumer hardware

Dalewyn 4y ago

>it can run on a typical GPU that gamers likely already have So a GTX 1060[1] or GTX 1650[1]? [1]: https://store.steampowered.com/hwsurvey/videocard/ https://store.steampowered.com/hwsurvey/videocard/

chii 4y ago
someone who only has a 1060 is what you'd call a casual gamer. A proper gamer would have a 4080, or at least a 3070, with the requisite 8gig of video ram.
- Dalewyn 4y ago
  A casual gamer would be running with Intel or AMD integrated graphics on whatever laptop they can get for a reasonable sum. They might have a mobile RTX 3060 if they splurged a bit. More likely they'll just have a console to game on. Gamers with RTX 3070 or above are a small count, at least according to Steam. RTX 4080 gamers are a staggering 0.20% of those surveyed.

pilooch 4y ago

The big difference is there's no full training code for llama, yet.

EricLeer 4y ago

Next to being easier to run, I also have the feeling that images by them selves are also more interesting then generating text. It is much easier for us to look past the defaults, whereas for a text it has to be of a really high quality to be of notice (and even more if you want it to be interesting).