3 ms·
I guess I look forward to seeing 'TheBloke' do a finetune with vram reqs. If my 3090 could fit a good quality quant I'd be interested. I tried to run a mixtra
by wing-_-nuts 3y ago
I guess I look forward to seeing 'TheBloke' do a finetune with vram reqs. If my 3090 could fit a good quality quant I'd be interested. I tried to run a mixtral with vram / sys ram offloading and got garbage ouput, but that's probably down to some mistake I made.
- kken 3y agoTry Ollama or LM Studio. Mixtral and its finetunes work perfectly for me on my RTX3090 with offloading.
- tosh 3y agoGGUF: https://huggingface.co/Qwen/Qwen1.5-32B-Chat-GGUF https://huggingface.co/Qwen/Qwen1.5-32B-Chat-GGUF AWQ: https://huggingface.co/Qwen/Qwen1.5-32B-Chat-AWQ https://huggingface.co/Qwen/Qwen1.5-32B-Chat-AWQ