6 ms·
This is why I think we're seeing a Stable Diffusion moment for LLMs: https://simonwillison.net/2023/Mar/11/llama/ https://simonwillison.net/2023/Mar/11/llama/
by simonw 4y ago
This is why I think we're seeing a Stable Diffusion moment for LLMs: https://simonwillison.net/2023/Mar/11/llama/ https://simonwillison.net/2023/Mar/11/llama/
Look at the timeline:
24th February 2023: LLaMA is announced, starts being shared with academic partners: https://research.facebook.com/publications/llama-open-and-efficient-foundation-language-models/ https://research.facebook.com/publications/llama-open-and-ef...
2nd March: Someone posts a PR with a BitTorrent link to the models: https://github.com/facebookresearch/llama/pull/73 https://github.com/facebookresearch/llama/pull/73
10th March: First commit to llama.cpp by Georgi Gerganov: https://github.com/ggerganov/llama.cpp/commit/26c084662903ddaca19bef982831bfb0856e8257 https://github.com/ggerganov/llama.cpp/commit/26c084662903dd...
11th March: llama.cpp now runs the 7B model on a 4GB RaspberryPi: https://twitter.com/miolini/status/1634982361757790209 https://twitter.com/miolini/status/1634982361757790209
12th March: npx dalai llama: https://cocktailpeanut.github.io/dalai/ https://cocktailpeanut.github.io/dalai/
13th March (today): llama.cpp on a Pixel 6 phone: https://twitter.com/thiteanish/status/1635188333705043969 https://twitter.com/thiteanish/status/1635188333705043969
And now, Alpaca. It's not even lunchtime yet!
Turned this into a blog post: https://simonwillison.net/2023/Mar/13/alpaca/ https://simonwillison.net/2023/Mar/13/alpaca/
- nwoli 4y agoA lot of them aren’t very good though at the same stable diffusion vram level unfortunately (and we’ve had large non consumer level gpu llms open sourced for a while eg gpt-j)
- chaxor 4y agoThat is likely because "good" is a higher bar in language than images, because people don't mind or notice the longer range artifacts in image models as much.
- CuriouslyC 4y agoA lot of people are running Llama using the CPU/system memory.
- loufe 4y agoMight I suggest looking the story between the 2nd and 10th of march? I've noticed Hacker News hasn't been following certain areas of the effort. A lot of great work had happened and continues to be happen in close conjunction with the text-generation-webui (seriously, most of the cutting edge with 4-bit GPTQ etc. has been closely tied to the project). >https://github.com/oobabooga/text-generation-webui/ https://github.com/oobabooga/text-generation-webui/
- simonw 4y agoWow, yeah that's a VERY active project: https://github.com/oobabooga/text-generation-webui/graphs/contributors https://github.com/oobabooga/text-generation-webui/graphs/co... - only started Dec 18, 2022 and already 22 contributors and 806 commits!
- 2bitencryption 4y agoHere is one question I have not seen answered yet: All the magic of "7B LLaMA running on a potato" seems to involve lowering precision down to f16 and then further quantizing to int4. Clearly this quantized model still outputs something resembling human language, at the very least. But I haven't seen anyone show what effect this quantizing has on the quality of the output. If the quality of the output is bad, it's unclear if it's because the model needs to be finetuned (as Stanford did here) or if it's because the quanitizing reduced the quality, or both. If this fine-tuned Stanford model still has excellent output after quantizing it to run on a Raspberry Pi 4GB, that would be awesome!
- gamegoblin 4y agoDecrease in accuracy is negligible and decreases as model size increases. That is, larger models quantize even better than smaller models. https://arxiv.org/abs/2210.17323 https://arxiv.org/abs/2210.17323
- byteknight 4y agoIs this because averages are weighed less (less sensitive) as the total sample size increases?
- londons_explore 4y agoYes. In a dense everything to everything neural network layer, the number of 'inputs' to a node is proportional to the square root of the number of weights. Therefore, assuming quantization noise is uncorrelated, as the number of weights doubles, the number of inputs goes up by sqrt(2), and the (normalized) noise goes down by a factor of 2*(sqrt(2)). So, as a rule of thumb, you can remove 1 bit of precision of the weights for every 4x increase in the number of weights. All this assumes weights and activations are uncorrelated random variables - which may not hold true.
- londons_explore 4y agoSomething is wrong with this math... by your logic I could scale the network up big enough that I could quantize the weights down to zero bits...
- wkat4242 4y agoI know, this is crazy!! I can't fathom how development has suddenly seemed to accelerate.
- all2 4y agoThe timing of the Facebook leak seems suspect.
- meghan_rain 4y agoWhat do you mean?
- all2 4y agoI mean ChatGPT had a lot of attention, so a leak of a competing architecture would shift the attention away from ChatGPT. Which Meta's LLaMA did. And we see it swinging in the other direction with OpenAI announcing GPT-4.
- theWreckluse 4y agoDo you mean Meta’s publishing of Llama?
- ulnarkressty 4y agoI'm excited to see what the OpenAssistant crowd does with these models, they seem to have gathered the dataset to finetune them. Lots of people use these models as talk therapy. We really need 1) standalone options, 2) reproducible weights with crowd sourced datasets to reduce biases (or at least know who you're talking to).
- atleastoptimal 4y agoQuestion: what percentage of the hype and momentum for this is so people can run sex chatbots on their local machine?
- sebzim4500 4y agoA lower portion than the equivalent number for Stable Diffusion, but still significant.
- yellowapple 4y agoFeature-length AI-generated pornos don't seem that far off the horizon.
- jhbadger 4y agoOr really just any text generation that chatGPT dislikes. It's nice not to be judged by a program (and perhaps logged somewhere that you asked for something "inappropriate").
- zitterbewegung 4y agoLLAMA.cpp with 65B parameters runs on a MacBook M1 Max with 64GB of RAM. See https://gist.github.com/zitterbewegung/4787e42617aa0be6019c3db526d5d6fd https://gist.github.com/zitterbewegung/4787e42617aa0be6019c3...
- notpushkin 4y agoWhat's the correlation between parameter count and RAM usage? Will LLaMA-13B fit on my MacBook Air with 8 GB of RAM or am I stuck with 7B?
- maxxk 4y ago13B uses about 9GB on my MacBook Air. If you have another machine (x86) with enough RAM to convert the original LLaMA representation to GGML, you can give it a try. But quantization step must be done on MacBook. Maybe it is more feasible for you to use 7B with larger context. For some "autocompletion" experiments with Python code I had to extend context to 2048 tokens (+1-1.5GB).
- cfn 4y agoI have also seen it working on a Mac Studio with 64Gb of RAM. It is quite slow, not unbearably so, but slow.
- aortega 4y agoThat is still a 4000 usd computer. You can get 2 RTX3900 used for ~1000 usd and run 65B much faster. I have a discord server up serving almost 500 users with 65B. https://twitter.com/ortegaalfredo/status/1635402627327590400 https://twitter.com/ortegaalfredo/status/1635402627327590400 For some things is better than GPT3, for other even Alpaca is better.
- chesneyc 4y agoThanks for taking the time to set this up. I will definitely give it a go later today. I don't have access to hardware that I can run LLaMA on and I'm really curious to see what the 65B model has to offer.
- r721 4y ago>Turned this into a blog post: https://simonwillison.net/2023/Mar/13/alpaca/ https://simonwillison.net/2023/Mar/13/alpaca/ HN discussion: https://news.ycombinator.com/item?id=35141531 https://news.ycombinator.com/item?id=35141531
- tvink 4y agoI think the Stable Diffusion moment is very dependant on someone creating a commercially licensable version of this somehow. I think the prospect of never being able to put your creations in a product is too inhibitive for the hypergrowth stable diffusion saw.
- zh217 4y agoAlso today: ChatGLM released by Tsinghua University. I've made a separate submission for it: https://news.ycombinator.com/item?id=35150190 https://news.ycombinator.com/item?id=35150190 The GitHub page is https://github.com/THUDM/ChatGLM-6B https://github.com/THUDM/ChatGLM-6B. The GitHub description is all in Chinese, but the model itself can handle English queries on a single consumer GPU well. Considering its size, I'd say the quality of its responses are outstanding.