4 ms·
I just spoke all night to 8x7B and can say that it sucks much less than 3.5. It doesn’t screw up and apologize all the time (and screw up again) and doesn’t rep
by wruza 3y ago
I just spoke all night to 8x7B and can say that it sucks much less than 3.5. It doesn’t screw up and apologize all the time (and screw up again) and doesn’t repeat what I just said verbatim. That is on topics I have a decent expertise in myself. Never had this experience of periodically forgetting that it’s not a human company with 3.5.
- eurekin 3y agoHow are you accessing it?
- gexla 3y agoYou can access it through Poe.com.
- wruza 3y agoLocal setup, “text generation webui”, TheBloke/Mixtral-8x7B-Instruct-v0.1-GGUF (Q4_K_M) on HF. You can run it on a decent intel cpu, takes around 32.5GB of ram including os (8gb for me). GPU with tensorcores can speed up few layers if you have one, but isn’t required. I get around 2.5-3 t/s with 8700 and 4070ti, that’s enough for chats that require some thinking. Edit: I was using 2k window, a larger one would probably eat more ram. But even with 2k it didn’t feel like it loses context or something.
- lhl 3y agoFor macOS and Linux, Ollama is probably the easiest way to try Mixtral (and a large number of models) locally. LM Studio is also nice and available for Mac, Windows, and Linux. As these models can be quite large and memory intensive, if you want to just give it a quick spin, huggingface.co/chat, chat.nbox.ai, and labs.pplx.ai all have Mixtral hosted atm.