3 ms·
Mixtral looks interesting, but I haven't dabbled in locally hosted LLMs. Would you mind linking to a concise text which could lead me through setting up Mixtra
by high_priest 3y ago
Mixtral looks interesting, but I haven't dabbled in locally hosted LLMs.
Would you mind linking to a concise text which could lead me through setting up Mixtral on my own machine?
- findjashua 3y agoLM Studio is the easiest way to do it
- Sabinus 3y agoThat's what I've been playing with. I can load 9 layers of a mixtral descendant into the 12gb vram for GPU and the rest into ~28gb ram for the CPU to work on. It chugs the system sometimes but the models are interestingly capable.
- djjtiro8 3y agoIt took me just as long to setup llama.cpp as it did to get other tools working well (ollama or other frontend that abstract away the actual config) It’s always read HOWTO, attempt to recreate state, so I prefer sticking with low level where I also learn a bit more about the internals C/C++ user friendliness has come as far as all the other languages and the ecosystems. Really the only reason to “fear” it is propagated memes to do so. It’s not a gun. So I’d suggest just compile llama.cpp and install huggingface-cli to download GGUF format models, which is all ollama is doing but with even more dependencies and much more opaque outcome
- elwebmaster 3y agoOllama gui in WSL2
- baq 3y agoHad zero experience, too. Turns out ollama does everything, literally. You just tell it to run a model and wait a bit for it to download. One (1) shell command total.
- aussieguy1234 3y agoI'm using Mixtral, but rather than shell out for a gaming laptop with an expensive GPU, I simply run it via Together.ai APIs which works out alot cheaper. There's a few similar services out there.