10 ms·
Mixtral 8x7B: A sparse Mixture of Experts language model
- SubiculumCode 3y agoquestions running through my head: Is there some magic with the number 8? Why not 6? 11? Each of these 8 models were 7B models. What about using 80 x tinyllama 1B models?
- lee101k 3y ago[dead]
- pushfoo 3y agoTL;DR: It looks like a convenient and explainable starting point for the proof-of-concept. Manually combining specialist variants is a known technique. This paper automates it with a router component which mixes 2 sub-models at any given time. Training 8 slight variants of a base seems safe and configurable compared to n > 16 specialists. The latter seems like the parts could interact unpredictably. Also, the memory usage seems predictable: it follows 2^m memory conventions by mixing 2 models at a time, so ~2x the memory is actively used at a time. I'm not up to date on the hardware implications, so it might not mean anything yet. It might one day if this approach works well enough to design around.
- smcleod 3y agoWasn’t this what was released at the end of last year?
- IanCal 3y agoThe model itself was, but was the writeup? This submission is from yesterday. I've not been following their releases too well but it seemed they were very much on the side of releasing models asap.
- ricopags 3y agoThe model's weights were released in a torrent, but this is the much anticipated paper detailing some of the work.
- Donz1 3y ago[dead]
- simonw 3y agoYes, the Mixtral magnet link was tweeted on the 8th of December: https://twitter.com/mistralai/status/1733150512395038967 https://twitter.com/mistralai/status/1733150512395038967 The paper just came out today: https://twitter.com/dchaplot/status/1744547220983005478l https://twitter.com/dchaplot/status/1744547220983005478l
- grepfru_it 3y agoAlso available on Ollama: https://ollama.ai/library/mixtral https://ollama.ai/library/mixtral
- cuuupid 3y agoI’d like to note that this model’s parameter usage is low enough (13b) to run smoothly at high quality on a 3090 while beating GPT-3.5 on humaneval and sporting 32k context. 3090s are consumer grade and common on gaming rigs. I’m hoping game devs start experimenting with locally deployed Mixtral in their games. e.g. something like CIV but with each leader powered via LLM
- minimaxir 3y agoThe average gamer doesn't have a 3090-equivalent, or even an Nvidia GPU. Running LLMs locally to create custom dialogue for games is still years away.
- somnic 3y agoVR isn't pragmatically accessible to the average gamer due to hardware requirements and the necessity of setting up the right physical environment but there are still VR games.
- __loam 3y agoThere's a few after almost a decade of vr being a thing.
- Cyphase 3y agoHow did you arrive at "almost a decade"? There's been VR stuff since at least the late 1980s. On the flip side you could say it wasn't "a thing" until just a few years ago. Or that it isn't "a thing" yet.
- __loam 3y agoI think it's pretty obvious that I'm talking about post oculus rift vr and the crop of games that use controllers similar to that device.
- kytazo 3y ago
- justinl33 3y agotldr; - 'sparse mixture of experts' means each layer contains 8 mini-neural-nets (experts), each trained to be good at certain data/tasks - a router network passes each token (word) to 2 experts which suit it best - since only 2/8 experts are utilized, the model effectively only uses 13/47B parameters during inference (text generation) - this expert mechanism makes it very efficient and effective (uses less params, is able to speciailize to tokens) - beats llmana 70b and gpt-3.5, especially at math and coding and language - has fine tuned model able to beat gemini pro, as well as llmana 70b and gpt-3.5
- deleted 3y ago[deleted]
- xrd 3y agoIn particular, Mixtral vastly outperforms Llama 2 70B on mathematics, code generation, and multilingual benchmarks I'm interested in seeing how it does with mathematics. That has always seemed like a particular weakness that no one yet has effectively cracked.
- justinl33 3y agoit's sort of a weakness inherent to LLM's... next word prediction isn't really supposed to be good at math. I doubt it will be ever 'cracked' with better LLM's, only multimodal ones with access to program execution and calculators.
- xrd 3y agoAgreed. But they do call it out here and I'm interested in understanding why.
- foota 3y agoThere's been attempts to use different embeddings for numbers, which helps a lot. E.g., https://news.ycombinator.com/item?id=37936005 https://news.ycombinator.com/item?id=37936005
- cjbprime 3y ago> it's sort of a weakness inherent to LLM's... next word prediction isn't really supposed to be good at math. FWIW I don't agree with this in a theoretical sense. The reason LLMs can do as much as they can is because next token prediction attempts to infer a world model for the processes that generated each next-token in the training set. I don't see a reason that this would preclude learning arithmetic in order to better predict next tokens that require arithmetic. I'd guess that arithmetic will become suddenly reliable with one of the next significant (e.g. 2-5x) jumps in parameter count.
- justinl33 3y agoI disagree; 1. the parameters are there to make the model better at learning textual patterns, not arithmetic patterns. 2. Next token prediction is a terrible way to perform arithmetic. 3. Perhaps most importantly, the loss function does not incentivise being good at arithmetic at all. Any perceived arithmetic ability is just a textual coincidence. I agree that a sufficiently intelligent LLM, like 300+ IQ would have an excellent model of how multiplication works. It may even assist in finding new theorems, but a calculator will always be better at 926*725.
- kromem 3y agoI'm curious when we'll start to see open access multimodal models being released. The advancement in text only models has been amazing, but a lot of the 'emergent' behavior in GPT-4 may be because of multimodal training and not just MoE or parameter sizes. I'll be curious to see if multimodal smaller models see similar leaps.
- ijustlovemath 3y agoI've heard that Google actually got the jump on OpenAI in this regard (just from people in a FAANG), and they're playing a bit of catch-up. OpenAI still has a distinct advantage on the language side, though. This is all hearsay, of course.
- minimaxir 3y agoLLaVA is open, although not the leap you are expecting: https://llava-vl.github.io/ https://llava-vl.github.io/ Meta also released a (non-commercial) multimodal model among 6 modalities: https://ai.meta.com/blog/imagebind-six-modalities-binding-ai/ https://ai.meta.com/blog/imagebind-six-modalities-binding-ai...
- coder543 3y agoCogVLM is very good in my (brief) testing: https://github.com/THUDM/CogVLM https://github.com/THUDM/CogVLM The model weights seem to be under a non-commercial license, not true open source, but it is "open access" as you requested. It would be nice if someone trained a CogVLM-compatible model from scratch under an open source license.
- beoberha 3y agoWhen I tried last, it couldn’t be run on M-series Macs.
- invert_franklin 3y agoDoes anyone know what Figure 8 at the end shows? It looks like each expert is used interchangeably with no clear pattern. And earlier they say "Surprisingly, we do not observe obvious patterns in the assignment of experts based on the topic." So then, what is the point of the "expert"? Could this extra performance just be through the 8 expert architectural design, and not be based on the underlying training material? For example, if all 8 experts were ArXiv papers, would the performance be different?
- jakderrida 3y agoNot sure if it answers your question, but the expertise of each expert arise from the training process and aren't assigned by a human. Thus, they wouldn't be discernable to a human. The choice of the model to use, I think, is something called like a "gating network". That's also trained to favor most appropriate model based on training.
- xrd 3y agoIs this a model that can be run using Simon Wilison's LLM tool? I cannot find any mention of Mixtral in the issues nor in the discussions. Is there an easy way to play with this model from the command line other than that?
- ilaksh 3y agoollama or llama.cpp
- ignoramous 3y agoUnsure what Simon Wilson's program does, but you can pull models via many methods. For CLI, Ollama: https://ollama.ai/library/mixtral https://ollama.ai/library/mixtral With UI, GPT4All: https://gpt4all.io/index.html https://gpt4all.io/index.html (doesn't yet support Mixtral) In-app, superagent.sh: https://github.com/homanp/superagent https://github.com/homanp/superagent
- gsharma 3y agoOllama is probably the easiest way. https://ollama.ai/library/mixtral https://ollama.ai/library/mixtral LM Studio is another option.
- xrd 3y agohttps://simonwillison.net/2023/Dec/18/mistral/ https://simonwillison.net/2023/Dec/18/mistral/
- vessenes 3y agoThis paper details the model that's been in the wild for approximately a month now. Mixtral 8x7B is very, very good. It's roughly sized at 13B, and ranked much, much higher than competitively sized models by, e.g. https://www.reddit.com/r/LocalLLaMA/comments/1916896/llm_comparisontest_confirm_leaderboard_big_news/ https://www.reddit.com/r/LocalLLaMA/comments/1916896/llm_com.... Ravenwolf notes that the model slightly outperforms some of its benchmark testing, and this is my experience. It's surprisingly good for a model of its size, and a very capable daily driver on a Mac for chat, code input and other uses. Something that has come to light since the release of the weights, and not mentioned in this paper is that it looks like fairly likely that the 8 experts were all seeded by Mistral 7B and subsequently diverged. This has generated a lot of experimentation in the local LLM community with cloning models as a way to cheaply generate experts. It was generally thought likely that training an 8x7B network would be as much work as training 8 7B networks, but this seems not to have been true for Mistral, which is super interesting. There's still a lot of rapid innovation happening in this space, with papers like Calm from DeepMind this week, and a lot of the adhoc experimental layer combining happening in the wild, (see, e.g. Goliath-120b), I think we're likely to see some pretty interesting architectural improvements this year in the LLM space. Calm seems to point the way to a next step after MoE, and models like Goliath seem to indicate that even a really really lazy version of Calm (no Linear layer combination, just literally alternating layers at full weights) can be very impactful. Overall I think we will see really, really strong models that are performant on consumer hardware in 2024, likely first half of this year.
- Casteil 3y agoI've had excellent results with Mixtral too - it's genuinely impressive. Only problem is that it's a relatively big model that's difficult to run with full GPU inference on consumer hardware (vs the 7b/13b models people typically use). So far, the main consumer platform capable of running it without 'ruining' the quality of its output (with high levels of quantization) is the newer Apple Silicon Macs with unified memory - generally >=48GB. It can apparently be done on 32 or 36GB, but there's not much headroom. Edit: As coder543 points out, yes - you can run it without more lossy levels of quantization on multi-GPU setups providing those have enough combined vram.
- noman-land 3y agoIf anyone wants to try out this model, I believe it's one of the ones released as a Llamafile by Mozilla/jart[0]. 1) Download llamafile[1] (30.03 GB): https://huggingface.co/jartine/Mixtral-8x7B-Instruct-v0.1-llamafile/resolve/main/mixtral-8x7b-instruct-v0.1.Q5_K_M.llamafile?download=true https://huggingface.co/jartine/Mixtral-8x7B-Instruct-v0.1-ll... 2) chmod +x mixtral-8x7b-instruct-v0.1.Q5_K_M.llamafile 3) ./mixtral-8x7b-instruct-v0.1.Q5_K_M.llamafile [0] https://hacks.mozilla.org/2023/11/introducing-llamafile/ https://hacks.mozilla.org/2023/11/introducing-llamafile/ [1] https://github.com/Mozilla-Ocho/llamafile#quickstart https://github.com/Mozilla-Ocho/llamafile#quickstart
- sp332 3y ago[flagged]
- timschmidt 3y agoThat's easy, just write better software than she does.
- sp332 3y agoWe don't actually have to let the people who want to disenfranchise others make a play for power.
- timschmidt 3y agoYou have the same attitude as OP. You could both be working on better software, but are regulating each other instead.
- aunty_helen 3y agoOn mac silicon: https://ollama.ai/ https://ollama.ai/ ollama pull mixtral For a chatgpt-esk web ui https://github.com/ollama-webui/ollama-webui https://github.com/ollama-webui/ollama-webui docker run -d -p 3000:8080 --add-host=host.docker.internal:host-gateway -v ollama-webui:/app/backend/data --name ollama-webui --restart always ghcr.io/ollama-webui/ollama-webui:main Navigate to http://localhost:3000 http://localhost:3000 You can also use ollama in langchain.
- viraptor 3y agoThere's also some unlocked fine tunings available. Dolphin seems to be a very popular one. (Trained on more coding data) If you want to fit under 32gb, there's https://ollama.ai/library/dolphin-mixtral:8x7b-v2.7-q3_K_M https://ollama.ai/library/dolphin-mixtral:8x7b-v2.7-q3_K_M
- NwpierratorR 3y agoI haven't played with dolphin-mixtral, but dolphin-mistral gave me a good impression for generic rag application. There's also developments and experimentation in making it more factual via dpo and laser(afaik so far not very unsuccessful).
- vunderba 3y agoI wouldn't use Ollama (or any LLM host) through docker on Mac m1 since as far as I'm aware there's still no support for Metal.
- Luc 3y agoThe UI is in Docker but the Ollama server isn't.
- dang 3y agoRecent and related: Mixtral of experts - https://news.ycombinator.com/item?id=38598559 https://news.ycombinator.com/item?id=38598559 - Dec 2023 (300 comments) Mistral-8x7B-Chat - https://news.ycombinator.com/item?id=38594578 https://news.ycombinator.com/item?id=38594578 - Dec 2023 (69 comments) Mistral "Mixtral" 8x7B 32k model [magnet] - https://news.ycombinator.com/item?id=38570537 https://news.ycombinator.com/item?id=38570537 - Dec 2023 (239 comments)
- bddppq 3y ago[dead]
- cgeier 3y agoI haven't read a lot of LLM papers, but I believe this is a rather weak paper low on details (note: not the results achieved of the LLM, but the paper itself). If it had landed on my desk for a review, I probably would have sent it back just based on that. For example, they never really say how they trained the experts or which dataset they used. Is this the current standard in the field?
- ShamelessC 3y ago> Is this the current standard in the field? It’s becoming pretty common, yeah. The two things you mentioned: training particulars and dataset mixture are also basically the only competitive advantage companies have. Since the code/architecture is trivial to reproduce, anyone with enough money can make a competing model “easily”. OpenAI started this trend and cemented it with GPT4’s “technical report” which didn’t even specify the number of parameters in the model. They’ve been historically vague about their dataset for far longer than that though.
- MichaelRazum 3y agoExactly, same thought. Actually I would expect, that they trained each expert separately and later together, since you need to train the router network as well. I'm far from an expert in LLMs. But this would be interesting to know, especially how different training setups influence the performance.
- Mashimo 3y agoAnyone know of a decent model for coding assistant that can run on a 16GB vram rtx 4060ti?
- stavros 3y agoThe chatbot arena has tons of good models that can do that, and Mistral, at the very least, isn't half bad.
- gsharma 3y agoIt will probably require some trial and error, but this leaderboard is a good starting point. https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderb... ~13B models should work well with plenty of room for other applications. Lately, I’ve heard good things about Solar10B, but new models come in a dozen by day, so it might have already been changed.
- lee101k 3y ago[dead]
- nl 3y agoIn their recent interview on the A16Z podcast the Mistral founder said they have multiple internal models between chatGPT and GPT4 level of quality. Given their high quality releases so far they means exciting times for open source LLMs.
- ignoramous 3y agoExcept there's no indication those more powerful Mistral models will also be FOSS.
- nl 3y agoWell he said that's the plan for the whole company so it'd be surprising if they aren't.
- coder543 3y agoMistral Medium is available via their API, but not available for download, for example, so I find that confusing if you’re claiming their CEO claimed the plan is to be open for all models.
- Reubend 3y agoIs there a description of each "expert"? Does one of the 8 models specialize in multi-lingual translation, while another specializes in coding? I don't see any answer to this in the paper, although I only skimmed it.
- arxiv_papers 3y ago[dead]