3 ms·

Fresh install from brew, ollama version is 0.5.7 Only settings I did were the ones shown in the blog post OLLAMA_FLASH_ATTENTION=1 OLLAMA_KV_CACHE_TYP

Fresh install from brew, ollama version is 0.5.7 Only settings I did were the ones shown in the blog post OLLAMA_FLASH_ATTENTION=1 OLLAMA_KV_CACHE_TYPE=q8_0 Ran the model like ollama run gemma2:27b --verbose With the same prompt, "Can you write me a story about a tortoise and a hare, but one that involves a race to get the most tokens per second?"

diggan 2y ago
When you run that, what quantization do you get? The library website of Ollama (https://ollama.com/library/gemma2:27b https://ollama.com/library/gemma2:27b) isn't exactly a good use case in surfacing useful information like what the default quantization is.
- fkyoureadthedoc 2y ago
  not sure how to tell, but here's the full output from ollama serve https://pastes.io/ollama-run-gemma2-27b https://pastes.io/ollama-run-gemma2-27b
  - diggan 2y ago
    Thanks, that seems to indicate Q4 for the quantization, you're probably able to run that on the 4090 as well FWIW, the size of the model is just 14.55 GiB.
  - navbaker 2y ago
    If you hit the drop-down menu for the size of the model, then tap “view all”, you will see the size and hash of the model you have selected and can compare it to the full list below it that has the quantization specs in the name.
    - diggan 2y ago
      Still, I don't see a way (from the web library) to see the default quantization (from Ollama's POV) at all, is that possible somehow?
      - navbaker 2y ago
        The model displayed in the drop-down when you access the web library is the default that will be pulled. Compare the size and hash to the more detailed model listing below it and you will see what quantization you have. Example: the default model weights for Llama 3.3 70b, after hitting the “view all” have this hash and size listed next to it - a6eb4748fd29 • 43GB Now scroll down through the list and you will find the one that matches that hash and size is “70b-instruct-q4_K_M”. That tells you that the default weights for Llama 3.3 70B from Ollama are 4-bit quantized (q4) while the “K_M” tells you a bit about what techniques were used during quantization to balance size and performance.
- mkesper 2y ago
  If you leave the :27b off from that URL you'll see the default size which is 9b. Ollama seems to always use Q4_0 even if other quants are better.

diggan 2y ago

When you run that, what quantization do you get? The library website of Ollama (https://ollama.com/library/gemma2:27b https://ollama.com/library/gemma2:27b) isn't exactly a good use case in surfacing useful information like what the default quantization is.

fkyoureadthedoc 2y ago
not sure how to tell, but here's the full output from ollama serve https://pastes.io/ollama-run-gemma2-27b https://pastes.io/ollama-run-gemma2-27b
- diggan 2y ago
  Thanks, that seems to indicate Q4 for the quantization, you're probably able to run that on the 4090 as well FWIW, the size of the model is just 14.55 GiB.
- navbaker 2y ago
  If you hit the drop-down menu for the size of the model, then tap “view all”, you will see the size and hash of the model you have selected and can compare it to the full list below it that has the quantization specs in the name.
  - diggan 2y ago
    Still, I don't see a way (from the web library) to see the default quantization (from Ollama's POV) at all, is that possible somehow?
    - navbaker 2y ago
      The model displayed in the drop-down when you access the web library is the default that will be pulled. Compare the size and hash to the more detailed model listing below it and you will see what quantization you have. Example: the default model weights for Llama 3.3 70b, after hitting the “view all” have this hash and size listed next to it - a6eb4748fd29 • 43GB Now scroll down through the list and you will find the one that matches that hash and size is “70b-instruct-q4_K_M”. That tells you that the default weights for Llama 3.3 70B from Ollama are 4-bit quantized (q4) while the “K_M” tells you a bit about what techniques were used during quantization to balance size and performance.
mkesper 2y ago
If you leave the :27b off from that URL you'll see the default size which is 9b. Ollama seems to always use Q4_0 even if other quants are better.