7 ms·
Many options for running Mistral models in your terminal using LLM
- bitdeep 3y agoYou missed ollama option
- M4v3R 3y agoWhich honestly is the easiest option of them all if you own an Apple Silicon based Mac. You just download the ollama and then run `ollama run mixtral` (or choose a quantization from their models page if you don't have enough ram to run the defalt q4 model) and that's it.
- deleted 3y ago[deleted]
- eurekin 3y agoI tried an hour ago and had a can't load model error. Everything up to date. Is there any special step?
- simonw 3y agoI was hoping I could run my LLM CLI tool against Ollama via their localhost API, but it looks like they don't offer an OpenAI-compatible endpoint yet. If they add that it will work out of the box: https://llm.datasette.io/en/stable/other-models.html#openai-compatible-models https://llm.datasette.io/en/stable/other-models.html#openai-... Otherwise someone would need to write a plugin for it, which would probably be pretty simple - I imagine it would look a bit like the llm-mistral plugin but adapted for the Ollama API design: https://github.com/simonw/llm-mistral/blob/main/llm_mistral.py https://github.com/simonw/llm-mistral/blob/main/llm_mistral....
- throwup238 3y agoMixtral-8x7B-Instruct is also available in a llamafile now: https://github.com/Mozilla-Ocho/llamafile#other-example-llamafiles https://github.com/Mozilla-Ocho/llamafile#other-example-llam...
- simonw 3y agoJust added that option - you can talk to it from my LLM CLI tool using its OpenAI-compatible API endpoint: https://simonwillison.net/2023/Dec/18/mistral/#llamafile-openai https://simonwillison.net/2023/Dec/18/mistral/#llamafile-ope...
- fbdab103 3y agoIf I have the baseline Mixtral-8x7B .pth, how do I convert that to a llamafile compatible .gguf?
- tarruda 3y agoJust download the GGUF from huggingface. @TheBloke always uploads GGUF versions of the popular models
- ekianjo 3y agoUseless as he never updates them post release. You are better served doing it yourself.
- tarruda 3y agoWhere did you get your information from? Just 4 days ago he redid a model after the author updated the EOS token: https://huggingface.co/TheBloke/openchat-3.5-1210-GGUF/discussions/1#657c2f1bc8c6e8dceed9289a https://huggingface.co/TheBloke/openchat-3.5-1210-GGUF/discu...
- ekianjo 3y agoJust make the gguf yourself it takes 5 mins and you dont need thebloke
- MilaM 3y agoThis is a bit off-topic and probably a noob question. But maybe someone with experience could help me. I'm about to buy a M2 or M3 Mac to replace my Intel Mac, and would like to play around with locally run LLMs in the future. How much RAM do I have to buy in order not to run into bottlenecks? I'm aware that you would probably want a beefier GPU for better performance. But as I said, I'm not worried about speed so much.
- azeirah 3y agoIf you want to run decently heavy models, I'd recommend getting at a minimum getting 48GB. This allows you to run 34b llama models with ease, 70b models quantized, mixtral without problems. If you want to run most models, get 64GB. This just gives you some more room to work with. If you want to run anything, get 128GB or more. Unquantized 70b? Check. Goliath 120b? Check. Note that high end consumer gpus end at 24GB VRAM. I have one 7900xtx for running llms, and the best it can reliably run is 4-bit quantized 34b models, anything larger is partially in regular ram.
- MilaM 3y agoThank you for this detailed response. I'm not sure if it was clear, but I was going to use just the Apple Silicon CPU/GPU, not an external one from Nvidia. Is there anything useful you can do with 24 or 32GB of RAM with llms? Regular M2 Mac minis can only be ordered with up to 24GB of RAM. The Pro Mac mini M2 is upgradable to 32GB RAM.
- simonw 3y ago32GB will run a good quantized Mixtral, though I can't confidently explain how much of a quality difference there is from unquantized.
- cschneid 3y agoData Point: I am currently having issues getting Mixtral Q4_K_M running in LMStudio on my 32gb M1 Max. I'm trying Q3 to see if it fits. I can have it run in on 'cpu' which is very slow, but offloading to the GPU runs out of memory.
- mrbonner 3y agoAny option to use a SaaS UI to use those OS models? I’m ok with paying the API access for Anya ale for example and use the ApI for the UI.
- ntonozzi 3y agoI’ve been using baseten (https://baseten.co https://baseten.co) and it’s been fun and has reasonable prices. Sometimes you can run some of these models from the hugging face model page, but it’s hit or miss.
- Der_Einzige 3y agoAll this needless difficulty when you can just do the right thing and use oogabooga. https://github.com/oobabooga/text-generation-webui https://github.com/oobabooga/text-generation-webui
- simonw 3y agoLooks like that provides an OpenAI-compatible API endpoint, which means you can use my LLM command-line utility against with the same pattern as for Llamafile: https://simonwillison.net/2023/Dec/18/mistral/#llamafile-openai https://simonwillison.net/2023/Dec/18/mistral/#llamafile-ope...
- brcmthrowaway 3y agoThis may be a dumb question but how do you update an LLM with new information after its knowledge cutoff?
- eurekin 3y ago- in context learning (simply put it into context) - finetuning, as in restarting the models checkpoint and relearning it on the previous + new data - adding extra neurons (e.g. LoRA adapters) at certain places and restarting learning Oh, in classic machine learning there's also the "bagging/boosting classifiers" option, but I have no knowledge if that can be applied to a ANN.
- simonw 3y agoHave you seen LoRA adapters actually work for this kind of thing? The leaked Google "We have no moat" memo was very excited about LoRA style techniques, but it's not clear to me that it's been proven as a technique yet.
- eurekin 3y agoNo, I have tried about 3 times and all failed. Altough it might be that I'm simply doing it wrong. There are people (can point at a discord server) claiming it works for them and that they even sell finetuned models to business clients. EDIT: I found one of the articles I tried to follow: https://www.mlexpert.io/prompt-engineering/chatbot-with-local-llm-using-langchain https://www.mlexpert.io/prompt-engineering/chatbot-with-loca... EDIT2: Ignore above. This seemed much more promising: https://www.youtube.com/watch?v=pnwVz64jNvw https://www.youtube.com/watch?v=pnwVz64jNvw . Author provides consulting services and seemed very nice and approachable
- simonw 3y agoGenerally, you train it again entirely from scratch. It's possible to introduce new information by fine-tuning a new model on top of the existing model, but it's debatable how effective that is for introducing new information - most fine-tuning success stories I've seen focus on teaching a model how to perform new kinds of task as opposed to teaching it new "facts". If you want a model to have access to updated information, the best way to do that is still via Retrieval Augmented Generation. That's the fancy name for the trick where you give the model the ability to run searches for information relevant to the user's questions and then invisibly paste that content into the prompt - effectively what Bing, Bard and ChatGPT do when they run searches against their attached search engines.
- syntaxing 3y agoAnybody have a Mistral invitation they can spare? I’m super curious how the medium performs compared to GPT3.5/4
- simonw 3y agoAre you on their waitlist? I got through that in less than 48 hours, so it might be worth a shot.
- syntaxing 3y agoI am, I signed up late though and got an email about it last night “ Access to our API is currently invitation-only, but we'll let you know when you can subscribe to get access to our best models.”
- kristianpaul 3y agoI run in Linux system it via cli using Ollama, very easy to setup https://ollama.ai/library/mistral https://ollama.ai/library/mistral curl https://ollama.ai/install.sh https://ollama.ai/install.sh | sh ollama run mistral:text
- traverseda 3y agoHuh, I've fought with a few of these things on my laptop, no nvidia GPU, limited ram, etc. This actually worked as advertised.
- 3abiton 3y agoWhat's the performance like (quality and speed wise)?
- all2 3y agoMistral 7b is serviceable for short contexts. If you have a longer conversation, token generation can start to lag a lot.
- tarruda 3y agoYesterday I tried mixtral 7bx8 running on the CPU. With an Intel 11th gen chip and 64gb DDR4 at 3200mhz, I got around 2-4 tokens/second in a small context, this gets progressively slower as the context grows. You would get a much better experience with apple silicon and lots of RAM
- cloudking 3y agoIt's great that there are many comments about being able to run models locally, but I rarely see comments about what folks are using them for. Would love to learn more about use cases and problems being solved.
- Der_Einzige 3y agoLet's be honest, they're using them mostly for ahem personal time.
- throwup238 3y agoIt's a goldrush and no one wants to share their secrets. Here's a good example from previous HN comments: https://news.ycombinator.com/item?id=38482347 https://news.ycombinator.com/item?id=38482347
- eurekin 3y agoI recently made them work with meeting presentation transcript. Mistral 7b (tiny one) was enough to reliably extract quite a lot of information! - It could derive speaker names, based purely on how people called themselves in the conversation - Drew a mermaid sequence diagram that wasn't perfect, but wasn't a complete garbage either. With few back and forth corrections it was on-point. - Created a truly usable meeting notes It was a much better UX than having to hunt down relevant video section, watch it and force to focus despite a lot of filler communication. That works very well for those kinds of 30 min. upward meeting, where the real content is in the spoken words and slides just reitarate that. I was really pleasantly surprised how much I liked that. Also, some models can rate each speaker contribution :o)
- kossTKR 3y agoAre people actually using this to 10x themselves / create actually profitable services already? Or more like finetuning a model to have an edge on the leaderboards for a day or two then taking VC money, or integrating som "AI magic" into existing userbases? I've been following the locallama sub on reddit and the few services created seem very niche besides tons of hobby stuff.
- 3y ago
- pram 3y agoI’ve been wondering, without stuff like llama.cpp and ollama etc how exactly were you intended to run Mistral when they released it?
- tarruda 3y agoThe main framework for running LLMs was pytorch, which AFAIK has a slow CPU implementation, so only GPU/CUDA users would be covered. Llama.cpp was the project that popularized running LLMs on the CPU due to its very efficient implementation. Ollama is a frontend to it.
- EVa5I7bHFq9mnYK 3y agoAny GUI option for us mere mortals not proficient with command line? faraday.dev has a mistral, but not that 8x option.
- dudeinhawaii 3y agoHonestly, Ollama (posted by someone earlier) was surprisingly simple to use. If you have WSL (on windows) or Linux/OSX it's a one-line install and a one-line use. I was up and running using a lowly 6GB VRAM GPU in about 3 minutes (time it took for initial download of models). Installing WSL (on Windows) is similarly straightforward nowadays. In your search bar, lookup the Microsoft Store, open the app, search for Ubuntu, install it, run it, follow the one-liner for installing Ollama. https://ollama.ai/library/mistral https://ollama.ai/library/mistral
- EVa5I7bHFq9mnYK 3y agobut will that wsl ubuntu be able to access my gpu card? do i need to install drivers for that? At untold CLI pain?
- u_name 3y agoMaybe i did something wrong, but the first mistral gguf i downloaded from hugging face and used with llama2 ended every answer with something along the lines of "let us know in the comments" and answers felt like blog posts. :D
- Havoc 3y ago> race to the bottom in terms of pricing from other LLM hosting providers. >This trend makes me a little nervous, since it actively disincentivizes future open model releases from Mistral and from other providers who are hoping to offer their own hosted versions. That does indeed seem ominous. I guess they’ll just introduce a significant lag till they release it in future