3 ms·
> mixtral works well Do you mean mistral? mixtral is 8x7B and requires like 100GB of RAM Edit: (without quant as others have pointed out) can definitely be l
by jasonjmcghee 3y ago
> mixtral works well
Do you mean mistral?
mixtral is 8x7B and requires like 100GB of RAM
Edit: (without quant as others have pointed out) can definitely be lower, but haven't heard of a 3.4GB version
- kwerk 3y agoI have two 3090s and it runs fine with `ollama run mixtral`. Although OP definitely meant mistral with the 7B note
- _ea1k 3y agoollama run mixtral will default to the quantized version (4bit IIRC). I'd guess this is why it can fit with two 3090s.
- K0balt 3y agoI run mixtral 6 bit quant very happily on my MacBook with 64 gb.
- deleted 3y ago[deleted]
- ranger_danger 3y agoI'm using mixtral-8x7b-v0.1.Q4_K_M.gguf with llama.cpp and it only requires 25GB.
- chpatrick 3y agoThe quantized one works fine on my 24GB 3090.
- XCSme 3y agoSorry, it was from memory. I have those models in Ollama: I have those: dolphin-mixtral:latest (24.6GB) mistral:latest (3.8GB)
- XCSme 3y agoI have 128GB, but something is weird with Ollama. Even though for the Ollama Docker I only allow 90GB, it ends up using 128GB/128GB, so the system because very slow (mouse freezes).
- InitEnabler 3y agoWhat docker flags are you running?
- XCSme 3y agoNone? The default ones from their docs. The Docker also shows minimal usage for the ollama server which is also strange.
- Havoc 3y agoThe smaller quants still require a 24gb card. 16 might work but doubt it