4 ms·
According to https://huggingface.co/TheBloke/Mixtral-8x7B-v0.1-GGUF https://huggingface.co/TheBloke/Mixtral-8x7B-v0.1-GGUF : 18.14GB in 2bit, which is still to
by Jackson__ 3y ago
According to https://huggingface.co/TheBloke/Mixtral-8x7B-v0.1-GGUF https://huggingface.co/TheBloke/Mixtral-8x7B-v0.1-GGUF :
18.14GB in 2bit, which is still too high for your GPU, and most likely borders on unusable in terms of quality. You could probably split it between CPU and GPU, if you don't mind the slowdown.