3 ms·

Does using 2x24GB VRAM mean that the model can be fully loaded into memory if it's between 24 and 48 GB in size? I somehow doubt it, at least ollama wouldn't wo

Does using 2x24GB VRAM mean that the model can be fully loaded into memory if it's between 24 and 48 GB in size? I somehow doubt it, at least ollama wouldn't work like that I think. But does anyone know?

memhole 2y ago
No. Hopefully, someone with more knowledge can explain better. But you need room for the kvcache is my understanding. You also need to factor in the size of the context window. If anyone has good resources on this, that would be awesome. Presently, it feels very much like a dark art to host these without crashing or being massively over-provisioned.
- htrp 2y ago
  The dark art is to massively overprovision hardware.
  - cratermoon 2y ago
    Thus you get Open AI spending billions while DeepSeek comes long with, shocking, actual understanding of the hardware and how to optimize for it[1] and spends $6 Million[2] 1. https://arxiv.org/abs/2412.19437v1 https://arxiv.org/abs/2412.19437v1 2. Quibble over the exact figure. Far less than Open AI, doing more with less.
Eisenstein 2y ago
No, you need to have extra space for the context (which requires more space the larger the model is). But it should be said that basing model quality on its size in GB is like qualifying a video based on its size in GB. You can have the same video be small or huge with anywhere from negligible to huge differences in quality between the two. You will be running quantizied model weights, which can range in precision from 1 to 16 bits per parameter (the B for billion in the model name). Model weights at Q8 are generally their parameter size without the B in GB (Llama 3 8B at Q8 would be ~8GB). There are many different strategies for quantizing as well, so this is just a rough guide. So basically if you can't fit the 48GB model into your 48GB of VRAM, just download a lower precision quant.
michaelt 2y ago
For a great many LLMs, you can find someone on HuggingFace who has produced a set of different quantised versions, with approximate RAM requirements. For example, if you want to run "CodeLlama 70B" from https://huggingface.co/TheBloke/CodeLlama-70B-Python-GGUF https://huggingface.co/TheBloke/CodeLlama-70B-Python-GGUF where's a table saying the "Q4_K_M" quantised version is a 41.42 GB download and runs in 43.92 GB of memory.

memhole 2y ago

No. Hopefully, someone with more knowledge can explain better. But you need room for the kvcache is my understanding. You also need to factor in the size of the context window. If anyone has good resources on this, that would be awesome. Presently, it feels very much like a dark art to host these without crashing or being massively over-provisioned.

htrp 2y ago
The dark art is to massively overprovision hardware.
- cratermoon 2y ago
  Thus you get Open AI spending billions while DeepSeek comes long with, shocking, actual understanding of the hardware and how to optimize for it[1] and spends $6 Million[2] 1. https://arxiv.org/abs/2412.19437v1 https://arxiv.org/abs/2412.19437v1 2. Quibble over the exact figure. Far less than Open AI, doing more with less.

Eisenstein 2y ago

No, you need to have extra space for the context (which requires more space the larger the model is). But it should be said that basing model quality on its size in GB is like qualifying a video based on its size in GB. You can have the same video be small or huge with anywhere from negligible to huge differences in quality between the two. You will be running quantizied model weights, which can range in precision from 1 to 16 bits per parameter (the B for billion in the model name). Model weights at Q8 are generally their parameter size without the B in GB (Llama 3 8B at Q8 would be ~8GB). There are many different strategies for quantizing as well, so this is just a rough guide. So basically if you can't fit the 48GB model into your 48GB of VRAM, just download a lower precision quant.

michaelt 2y ago

For a great many LLMs, you can find someone on HuggingFace who has produced a set of different quantised versions, with approximate RAM requirements. For example, if you want to run "CodeLlama 70B" from https://huggingface.co/TheBloke/CodeLlama-70B-Python-GGUF https://huggingface.co/TheBloke/CodeLlama-70B-Python-GGUF where's a table saying the "Q4_K_M" quantised version is a 41.42 GB download and runs in 43.92 GB of memory.