5 ms·
I don't get the point. There is just no way you'll be able to run llama2 70B. And llama2 13B, although cool, is much much dumber than GPT3.5. I don't think it's
by redox99 3y ago
I don't get the point. There is just no way you'll be able to run llama2 70B. And llama2 13B, although cool, is much much dumber than GPT3.5. I don't think it's useful as an ChatGPT style assistant.
Maybe in the future we'll get very advanced models with that number of parameters. But running current Llama2 13B on a mobile device doesn't seem too useful IMO.
- YetAnotherNick 3y agoLlama 7B is infinite times smarter than current Siri though. And with 3 bit quantization it is like 3GB of RAM which is completely achievable.
- _c8fm 3y agoHow much does 3 bit quantization affect performance?
- YetAnotherNick 3y agoAccording to few papers and https://github.com/ggerganov/llama.cpp/pull/1684 https://github.com/ggerganov/llama.cpp/pull/1684, ~2.5GB 7B parameters model size has the same performance as baseline 7B model(with 14GB size).
- Grimblewald 3y agoYou don't need a lot to run things that can do simple sentiment analysis, or translating natural language statements to some API call. You can do a lot of the things that were initially promised with things like google assistant or siri through fairly basic models. They don't need to be able to write code and translate between 100's of languages to be useful. More importantly, they could significantly reduce the amount of server-side strain when it comes to collecting personal info from peoples conversations etc, which I imagine is the real incentive.