3 ms·
It's possible to run a full GPT-3 style language model on any device with 4GB of RAM now, so running models on consumer devices is getting more and more feasibl
by simonw 4y ago
It's possible to run a full GPT-3 style language model on any device with 4GB of RAM now, so running models on consumer devices is getting more and more feasible by the day. https://simonwillison.net/2023/Mar/11/llama/ https://simonwillison.net/2023/Mar/11/llama/
- whimsicalism 4y agoIts possible to run a RLHF tuned Llama 7b model. Whether this is "full GPT-3 style" is up for debate.
- freedomben 4y agoI'm mostly a layman with ML stuff, so I might be doing something wrong, but I've not been impressed with Llama even at higher levels. I've run the 35B model in my home lab and it gave some pretty nonsensical responses. The 13B did better though, so could very well be user error.
- refulgentis 4y agoThere’s this gold rush going on, you’re right, any B without RLHF is meh. The things getting published as “on device LLM” focus on bitcrushing the lowest B model with minimal RLHF and then pronouncing we have on device LLMs. We’ll definitely get there but signal >>> noise currently. First person to admit this and write their blog post with A / B tests vs. a Markov chain deserves the gold.
- simonw 4y agoHave you tried Alpaca yet? It's a massive improvement on base LLaMA.