4 ms·
On an M1 Max 64GB laptop running gemma2:27b same prompt and settings from blog post total duration: 24.919887458s load duration: 39.315083
by fkyoureadthedoc 2y ago
On an M1 Max 64GB laptop running gemma2:27b same prompt and settings from blog post
total duration: 24.919887458s
load duration: 39.315083ms
prompt eval count: 37 token(s)
prompt eval duration: 963.071ms
prompt eval rate: 38.42 tokens/s
eval count: 441 token(s)
eval duration: 23.916616s
eval rate: 18.44 tokens/s
I have a gaming PC with a 4090 I could try, but I don't think this model would fit
- diggan 2y ago> gemma2:27b What quantization are you using? What's the runtime+version you run this with? And the rest of the settings? Edit: Turns out parent is using Q4 for their test. Doing the same test with LM Studio and a 3090ti + Ryzen 5950X (with 44 layers on GPU, 2 on CPU) I get ~15 tokens/second.
- fkyoureadthedoc 2y agoFresh install from brew, ollama version is 0.5.7 Only settings I did were the ones shown in the blog post OLLAMA_FLASH_ATTENTION=1 OLLAMA_KV_CACHE_TYPE=q8_0 Ran the model like ollama run gemma2:27b --verbose With the same prompt, "Can you write me a story about a tortoise and a hare, but one that involves a race to get the most tokens per second?"
- diggan 2y agoWhen you run that, what quantization do you get? The library website of Ollama (https://ollama.com/library/gemma2:27b https://ollama.com/library/gemma2:27b) isn't exactly a good use case in surfacing useful information like what the default quantization is.
- fkyoureadthedoc 2y agonot sure how to tell, but here's the full output from ollama serve https://pastes.io/ollama-run-gemma2-27b https://pastes.io/ollama-run-gemma2-27b
- diggan 2y agoThanks, that seems to indicate Q4 for the quantization, you're probably able to run that on the 4090 as well FWIW, the size of the model is just 14.55 GiB.
- navbaker 2y agoIf you hit the drop-down menu for the size of the model, then tap “view all”, you will see the size and hash of the model you have selected and can compare it to the full list below it that has the quantization specs in the name.
- diggan 2y agoStill, I don't see a way (from the web library) to see the default quantization (from Ollama's POV) at all, is that possible somehow?
- navbaker 2y agoThe model displayed in the drop-down when you access the web library is the default that will be pulled. Compare the size and hash to the more detailed model listing below it and you will see what quantization you have. Example: the default model weights for Llama 3.3 70b, after hitting the “view all” have this hash and size listed next to it - a6eb4748fd29 • 43GB Now scroll down through the list and you will find the one that matches that hash and size is “70b-instruct-q4_K_M”. That tells you that the default weights for Llama 3.3 70B from Ollama are 4-bit quantized (q4) while the “K_M” tells you a bit about what techniques were used during quantization to balance size and performance.
- mkesper 2y agoIf you leave the :27b off from that URL you'll see the default size which is 9b. Ollama seems to always use Q4_0 even if other quants are better.
- rahimnathwani 2y agogemma2:27b-instruct-q4_0 (checksum 53261bc9c192)
- condiment 2y agoOn a 3090 (24gb vram), same prompt & quant, I can report more than double the tokens per second, and significantly faster prompt eval. total_duration: 10530451000 load_duration: 54350253 prompt_eval_count: 36 prompt_eval_duration: 29000000 prompt_token/s: 1241.38 eval_count: 460 eval_duration: 10445000000 response_token/s: 44.04 Fast prompt eval is important when feeding larger contexts into these models, which is required for almost anything useful. GPUs have other advantages for traditional ML, whisper models, vision, and image generation. There's a lot of flexibility that doesn't really get discussed when folks trot out the 'just buy a mac' line. Anecdotally I can share my revealed preference. I have both an M3 (36gb) as well as a GPU machine, and I went through the trouble of putting my GPU box online because it was so much faster than the mac. And doubling up the GPUs allows me to run models like the deepseek-tuned llama 3.3, with which I have completely replaced my use of chatgpt 4o.
- svachalek 2y agoThanks for numbers! People should include their LLM runner as well I think, as there are differences in hardware optimization support. Like I haven't tested it but I've heard MLX is noticeably faster than Ollama on Macs.
- fkyoureadthedoc 2y ago7800X3D, 32GB DDR5, 4090: total duration: 10.5922028s load duration: 21.1739ms prompt eval count: 36 token(s) prompt eval duration: 546ms prompt eval rate: 65.93 tokens/s eval count: 467 token(s) eval duration: 10.023s eval rate: 46.59 tokens/s