3 ms·
It's 2.7B, not 1.1. In my experience it goes off the rails and starts generating nonsense after a few paragraphs, but I haven't dug too much into tweaking the k
by regularfry 3y ago
It's 2.7B, not 1.1. In my experience it goes off the rails and starts generating nonsense after a few paragraphs, but I haven't dug too much into tweaking the kv cache params to see if that's controllable. It also needs a fair bit of prompt massaging to get it to do what you want. So no, not GPT3.5, but it's comfortably better than anything else in its size class.
- eurekin 3y agoWhat are kv cache params?
- regularfry 3y agoKey-value cache in the attention layers. There was a paper a little while back about how maintaining the first N tokens across an extended context helped an LLM keep sane for longer, and it turns out you can replicate it with the right CLI arguments to llama.cpp.
- stavros 3y agoThis sounds much more realistic, thanks!
- realPtolemy 3y agoHow is it compared to 7B LLaMA quantized to run on a raspberry pi?
- regularfry 3y agoProbably similar token rates out of the box, although I havent done a straight comparison. Where they'll differ is in the sorts of questions they're good at. Llama2 was trained (broadly speaking) for knowledge, Phi-2 for reasoning. And bear in mind that you can quantise phi-2 down too. The starting point is f16.
- jasonjmcghee 3y agoIf you can run quantized 7B, nothing beats mistral and its fine tunes- like openhermes2.5