4 ms·
How are you accessing it?
by eurekin 3y ago
How are you accessing it?
- gexla 3y agoYou can access it through Poe.com.
- wruza 3y agoLocal setup, “text generation webui”, TheBloke/Mixtral-8x7B-Instruct-v0.1-GGUF (Q4_K_M) on HF. You can run it on a decent intel cpu, takes around 32.5GB of ram including os (8gb for me). GPU with tensorcores can speed up few layers if you have one, but isn’t required. I get around 2.5-3 t/s with 8700 and 4070ti, that’s enough for chats that require some thinking. Edit: I was using 2k window, a larger one would probably eat more ram. But even with 2k it didn’t feel like it loses context or something.
- lhl 3y agoFor macOS and Linux, Ollama is probably the easiest way to try Mixtral (and a large number of models) locally. LM Studio is also nice and available for Mac, Windows, and Linux. As these models can be quite large and memory intensive, if you want to just give it a quick spin, huggingface.co/chat, chat.nbox.ai, and labs.pplx.ai all have Mixtral hosted atm.