2 ms·
Exciting, I think 12B is the sweet spot for running locally - large enough to be useful, fast enough to run on a decent laptop.
by pants2 2y ago
Exciting, I think 12B is the sweet spot for running locally - large enough to be useful, fast enough to run on a decent laptop.
- _flux 2y agoHow much memory does employing the complete 128k window take, though? I've sadly noticed that it can take a significant amount of VRAM to use a larger context window. edit: e.g. I wouldn't know the correct parameters for this calculator, but going from 8k window to 128k window goes from 1.5 GB to 23 GB: https://huggingface.co/spaces/NyxKrage/LLM-Model-VRAM-Calculator https://huggingface.co/spaces/NyxKrage/LLM-Model-VRAM-Calcul...
- azeirah 2y agoIn practice, it's fine to stick with "just" 8k or 16k or 32k. If you're working with data of over 128k tokens I'd personally not recommend using an open model anyway unless you know what you're doing. The models are kinda there, but the hardware mostly isn't. This is only realistic right now for people with those unified memory MacBook or for enthusiasts with Epyc servers or a very high end workstation built for inference. Anything above that I don't consider "consumer" inference
- mythz 2y agoIMO Google's Gemma2 27B [1] is the sweet spot for running locally on commodity 16GB VRAM cards. [1] https://ollama.com/library/gemma2:27b https://ollama.com/library/gemma2:27b
- Raed667 2y agoIf I "only" have 16GB of ram on a macbook pro, would that still work ?
- sofixa 2y agoIf it's an M-series one with "unified memory" (shared RAM between the CPU, GPU and NPU on the same chip), yes.
- mysteria 2y agoKeep in mind that Gemma is a larger model but it only has 8k context. The Mistral 12B will need less VRAM to store the weights but you'll need a much larger KV cache if you intend to use the full 128k context, especially if the KV is unquantized. Note sure if this new model has GQA but those without it absolutely eat memory when you increase the context size (looking at you Command R).