12 ms·
Mixtral 8x22B
- spenceryonce 3y agoI can't even begin to describe how excited I am for the future of AI.
- dd-dreams 3y agoThe development never stops. In a few years we will look back and see how the previous models were and how they're now. How we couldn't run LLaMa 70B on MacBook Air and now we can.
- squirrel23 3y agoYes it's pretty cool. There was a neat comparison of deep learning development that I think resonates quite well here. Around 5 years ago, it took a lambda user some pretty significant hardware, software and time (around a full night), to try to create a short deepfake. Now, you don't need any fancy hardware and you can have some decent results within 5 min on your average computer.
- azinman2 3y agoThat part isn’t very good. https://www.nytimes.com/2024/04/08/technology/deepfake-ai-nudes-westfield-high-school.html https://www.nytimes.com/2024/04/08/technology/deepfake-ai-nu...
- imjonse 3y agoGreat to see such free to use and self-hostable models, but it's said that open now means only that. One cannot replicate this model without access to the training data.
- ru552 3y agoThere's a large amount of liability in disclosing your training data.
- imjonse 3y agoCalling the model 'truly open' without is not technically correct though.
- Lacerda69 3y agoIt's open enough for all practical purposes IMO.
- nicklecompte 3y agoIt's not "open enough" to do an honest evaluation of these systems by constructing adversarial benchmarks.
- imjonse 3y agoAs open as an executable binary that you are allowed to download and use for free.
- bevekspldnw 3y agoIn the age of SaaS I’ll take it. It’s not like I have a few million dollars to pay for training even if I had all the code and data.
- londons_explore 3y agoI expect we'll see some AI companies in the future throwing away the training dataset. Maybe some have already. During a court case, the other side can demand discovery over your training dataset, for example to see if it contains a particular copyrighted work. But if you've already deleted the dataset, you're far more likely to win any case against you that hinges on what was in the dataset if the plaintiff can't even prove their work was included. And you can argue that the dataset was very expensive to store (which is true), and therefore deleted shortly after training was complete. You have no obligation to keep something for the benefit of potential future plaintiffs you aren't even aware of yet.
- generalizations 3y ago...And a massive pile of cash/compute hardware.
- kiney 3y agonot that massive, we're talking six figures. There was a blogpost about this a while back on the startpage of HN.
- htrp 3y agofor finetuning or parameter training from scratch?
- kiney 3y agofrom scratch: https://research.myshell.ai/jetmoe https://research.myshell.ai/jetmoe
- kaibee 3y agoThat's for an 8B model.
- cptcobalt 3y agoThis is over trivializing it, but there isn't much more inherent complexity in training an 8B or larger model other than more money, more compute, more data, more time. Overall, the principles are similar.
- lostmsu 3y agoAssuming linear growth to number of parameters that's 7.5 figures instead of 6 for 8x22B model.
- moffkalast 3y ago6 figures are a massive pile of cash.
- sa-code 3y agoIs this the best permissively licensed model out there?
- ru552 3y agoToday. Might change tomorrow at the pace this sector is at.
- imjonse 3y agoSo far it is Command R+. Let's see how this will fare on Chatbot Arena after a few weeks of use.
- skissane 3y ago> So far it is Command R+ Most people would not consider Command R+ to count as the "best permissively licensed model" since CC-BY-NC is not usually considered "permissively licensed" – the "NC" part means "non-commercial use only"
- imjonse 3y agoMy bad, I remembered wrongly it was Apache too.
- tinyhouse 3y agoPricing? Found it: https://mistral.ai/technology/#pricing https://mistral.ai/technology/#pricing It'd useful to add a link to the blog post. While it's an open model, most will only be able to use it via the API.
- MacsHeadroom 3y agoIt's open source, you can just download and run it for free on your own hardware.
- tinyhouse 3y agoWell, I don't have hardware to run a 141B parameters model, even if only 39B are active during inference.
- astrodust 3y ago"Who among us doesn't have 8 H100 cards?"
- MacsHeadroom 3y agoFour V100s will do. They're about $1k each on ebay.
- astrodust 3y ago$1500 each, plus the server they go in, plus plus plus plus.
- apetresc 3y agoI just find it hilarious how approximately 100% of models beat all other models on benchmarks.
- squirrel23 3y agoWhat do you mean?
- apetresc 3y agoVirtually every announcement of a new model release has some sort of table or graph matching it up against a bunch of other models on various benchmarks, and they're always selected in such a way that the newly-released model dominates along several axes. It turns interpreting the results into an exercise in detecting which models and benchmarks were omitted.
- CharlieDigital 3y agoIt would make sense, wouldn't it? Just as we've seen rising fuel efficiency, safety, dependability, etc. over the lifecycle of a particular car model. The different teams are learning from each other and pushing boundaries; there's virtually no reason for any of the teams to release a model or product that is somehow inferior to a prior one (unless it had some secondary attribute such as requiring lower end hardware). We're simply not seeing the ones that came up short; we don't even see the ones where it fell short of current benchmarks because they're not worth releasing to the public.
- apetresc 3y agoThat's a valid theory, a priori, but if you actually follow up you'll find that the vast majority of these benchmark results don't end up matching anyone's subjective experience with the models. The churn at the top is not nearly as fast as the press releases make it out to be.
- 3y ago
- arnaudsm 3y agoCurious to see how it performs against GPT-4. Mixtral8x22 beats CommandR+, which is at GPT-4-level in LMSYS' leaderboard.
- zone411 3y agoLMSYS leaderboard is just one benchmark (that I think is fundamentally flawed). GPT-4 is clearly better.
- arnaudsm 3y agoWhich alterative benchmarks do you recommend?
- zone411 3y agoI don't have a perfect solution, except for the obvious answer that the best we can do is a combination of multiple benchmarks. It's harder now than ever because you also want to test long contexts, and older benchmarks are over-optimized. I think it would be best to overweigh newer benchmarks and underweigh benchmarks on which objectively dumb models like Claude 3 Haiku score well. And I have a big problem with HumanEval, which is Python-only and has only 164 problems but is used as the catch-all for coding abilities.
- brokensegue 3y agoIsn't equating active parameters with cost a little unfair since you still need full memory for all the inactive parameters?
- endisneigh 3y agoGood to continue to see a permissive license here.
- mdrzn 3y ago"64K tokens context window" I do wish they had managed to extend it to at least 128K to match the capabilities of GPT-4 Turbo Maybe this limit will become a joke when looking back? Can you imagine reaching a trillion tokens context window in the future, as Sam speculated on Lex's podcast?
- htrp 3y agomaybe we'll look back at token context windows like we look back at how much ram we have in a system.
- frabjoused 3y agoI agree with this in the sense that once you have enough, you stop caring about the metric.
- paradite 3y agoAnd how much RAM do you need to run Mixtral 8*22B? Probably not enough on a personal laptop.
- mithrilskeleton 3y agoHow much vram is need to run this?
- MacsHeadroom 3y ago80GB in 4bit. But because it only activates one expert at a time, it can run on a fast CPU in reasonable time. So 96GB of DDR4 will do. 96GB of DDR5 is better.
- Me1000 3y agoWizardLM-2 8x22b (which was a fine tune of the Mixtral 8x22b base model) at 4bit was only 80GB.
- noman-land 3y agoI'm really excited about this model. Just need someone to quantize it to ~3 bits so it'll run on a 64GB MacBook Pro. I've gotten a lot of use from the 8x7b model. Paired with llamafile and it's just so good.
- 2Gkashmiri 3y agoCan you explain your use case? I tried to get into offline llms, on my machine and even android but without discrete graphics, its a slow hog so I didnt enjoy it but suppose I buy one, what then ?
- andai 3y agoI run Mistral-7B on an old laptop. It's not very fast and it's not very good, but it's just good enough to be useful. My use case is that I'm more productive working with a LLM but being online is a constant temptation and distraction. Most of the time I'll reach for offline docs to verify. So the LLM just points me in the right direction. I also miss Google offline, so I'm working on a search engine. I thought I could skip crawling by just downloading common crawl, but unfortnately it's enormous and mostly junk or unsuitable for my needs. So my next project is how to data-mine common crawl to extract just the interesting (to me) bits... When I have a search engine and a LLM I'll be able to run my own Phind, which will be really cool.
- luke-stanley 3y agoPresumably you could run things like PageRank, I'm sure people do this sort of thing with CommonCrawl. There are lots of variants of graph connectivity scoring methods and classifiers. What a time to be alive eh?
- noman-land 3y agoYes, I have a side project that uses local whisper.cpp to transcribe a podcast I love and shows a nice UI to search and filter the contents. I use Mixtral 8x7b in chat interface via llamafile primarily to help me write python and sqlite code and as a general Q&A agent. I ask it all sorts of technical questions, learn about common tools, libraries, and idioms in an ecosystem I'm not familiar with, and then I can go to official documentation and dig in. It has been a huge force multiplier for me and most importantly of all, it removes the dread of not knowing where to start and the dread of sending your inner monologue to someone's stupid cloud. If you're curious: https://github.com/noman-land/transcript.fish/ https://github.com/noman-land/transcript.fish/ though this doesn't include any Mixtral stuff because I don't use it programmatically (yet). I soon hope to use it to answer questions about the episodes like who the special guest is and whatnot, which is something I do manually right now.
- clementmas 3y agoI'm considering switching my function calling requests from OpenAI's API to Mistral. Are they using similar formats? What's the easiest way to use Mistral? Is it by using Huggingface?
- ru552 3y agoeasiest is probably with ollama [0]. I think the ollama API is OpenAI compatible. [0]https://ollama.com/ https://ollama.com/
- talldayo 3y agoMost inference servers are OpenAI-compatibile. Even the "official" llama-cpp server should work fine: https://github.com/ggerganov/llama.cpp/blob/master/examples/server/README.md https://github.com/ggerganov/llama.cpp/blob/master/examples/...
- pants2 3y agoOllama runs locally. What's the best option for calling the new Mixtral model on someone else's server programmatically?
- Arcuru 3y agoOpenrouter lists several options: https://openrouter.ai/models/mistralai/mixtral-8x22b https://openrouter.ai/models/mistralai/mixtral-8x22b
- jjice 3y agoDoes anyone have a good layman's explanation of the "Mixture-of-Experts" concept? I think I understand the idea of having "sub-experts", but how do you decide what each specialization is during training? Or is that not how it works at all?
- Keyframe 3y agomaybe there's one that is maitre d'llm?
- HeatrayEnjoyer 3y agoCorrect, the experts are determined by Algo, not anything humans would understand.
- londons_explore 3y agoNobody decides. The network itself determines which expert(s) to activate based on the context. It uses a small neural network for the task. It typically won't behave like human experts - you might find one of the networks is an expert in determining where to place capital letters or full stops for example. MoE's do not really improve accuracy - instead they are to reduce the amount of compute required. And, assuming you have a fixed compute budget, that in turn might mean you can make the model bigger to get better accuracy.
- hlfshell 3y agoThis is a bit of a misnomer. Each expert is a sub network that specializes in sub understanding we can't possibly track. During training a routing network is punished if it does not evenly distribute training tokens to the correct experts. This prevents any one or two networks from becoming the primary networks. The result of this is that each token has essentially even probability of being routed to one of the sub models, with the underlying logic of why that model is an expert for that token being beyond our understanding or description.
- api 3y agoA decent loose analogy might be database sharding. Basically you're sharding the neural network by "something" that is itself tuned during the learning process.
- jonnycomputer 3y agoThese LLMs are making RAM great again. Wish I had invested in the extra 32GB for my mac laptop.
- Workaccount2 3y agoYou can't upgrade it? Edit: I haven't owned a laptop for years, probably could have surmised they'd be more user hostile nowadays.
- paxys 3y ago> mac laptop
- jonnycomputer 3y agoI really really like my Macbook Pro. But dammit, you can't upgrade the thing (Mac laptops aren't upgrade-able anymore). I got M1 Max in 2021 with 32GB of RAM. I did not anticipate needing more than 32GB for anything I'd be doing on it. Turns out, a couple of years later, I like to run local LLMs that max out my available memory.
- jonnycomputer 3y agoI say 2021, but truth is the supply chain was so trash that year that it took almost a year to actually get delivered. I don't think I actually started using the thing until 2022.
- jonnycomputer 3y agoI got downvoted for saying a true fact? that I ordered a the new M1 Max in 2021 and it took almost a year for me to actually get it? it true.
- cqqxo4zV46cp 3y agoMaybe they think that you aren’t meaningfully contributing to the conversation, which seems very, very likely.
- hubraumhugo 3y agoIt feels absolutely amazing to build an AI startup right now. It's as if your product automatically becomes cheaper, more reliable, and more scalable with each new major model release. - We first struggled with limited context windows [solved] - We had issues with consistent JSON ouput [solved] - We had rate limiting and performance issues for the large 3rd party models [solved] - Hosting our own OSS models for small and medium complex tasks was a pain [solved] Obivously every startup still needs to build up defensibility and focus on differentiating with everything “non-AI”.
- paxys 3y agoWe are going to quickly reach the point where most of these AI startups (which do nothing but provide thin wrappers on top of public LLMs) aren't going to be needed at all. The differentiation will need to come from the value of the end product put in front of customers, not the AI backend.
- layble 3y agoSure, in the same way SaaS companies are just thin wrappers on top of databases and the open web.
- imjonse 3y agoYou will find that a disproportionately large amount of work and innovation in an AI product is in the backing model (GPT, Mixtral, etc.). While there's a huge amount of work in databases and the open web, SaaS products typically add a lot more than a thin API layer and a shiny website (well some do but you know what I mean)
- tomrod 3y agoI'd argue the comment before you is describing accessibility, features, and services -- yes, the core component has a wrapper, but that wrapper differentiates the use.
- 3y ago
- Lacerda69 3y agoI have been using mixtral daily since it was released for all kinds of writing and coding tasks. Love it and massively invested in mistrals mission. Keep on doing this great work. Edit: been using the previous version, seems like this one is even better?
- iFire 3y agoIt wasn't clear but how much hardware does it take to run Mixtral 8x22B (mistral.ai) next to me locally?
- ru552 3y agoa macbook with 64g of ram
- noman-land 3y agoAt what quantization?
- ChicagoDave 3y agoWe need larger context windows, otherwise we’re running the same path with marginally different results.
- luke-stanley 3y agoI'm confused on the instruction fine-tuning part that is mentioned briefly, in passing. Is there an open weight instruct variant they've released? Or is that only on their platform? Edit: It's on HuggingFace, great, thanks replies!
- freedmand 3y agoI just found this on HuggingFace: https://huggingface.co/mistralai/Mixtral-8x22B-Instruct-v0.1 https://huggingface.co/mistralai/Mixtral-8x22B-Instruct-v0.1
- sva_ 3y agohttps://huggingface.co/mistralai/Mixtral-8x22B-Instruct-v0.1 https://huggingface.co/mistralai/Mixtral-8x22B-Instruct-v0.1
- austinsuhr 3y agoIs 8x22B gonna make it to Le Chat in the near future?
- ayolisup 3y agoWhat's the best way to run this on my Macbook Pro? I've tried LMStudio, but I'm not a fan of the interface compared to OpenAI's. The lack of automatic regeneration every time I edit my input, like on ChatGPT, is quite frustrating. I also gave Ollama a shot, but using the CLI is less convenient. Ideally, I'd like something that allows me to edit my settings quite granularly, similar to what I can do in OpenLM, with the QoL from the hosted online platforms, particularly the ease of editing my prompts that I use extensively.
- duckkg5 3y agoOllama with WebUI https://github.com/open-webui/open-webui https://github.com/open-webui/open-webui
- ru552 3y ago[flagged]
- shaunkoh 3y agoNot sure why your comment was downvoted. ^ is absolutely the right answer. Open WebUI is functionally identical to the ChatGPT interface. You can even use it with the OpenAI APIs to have your own pay per use GPT 4. I did this.
- lubesGordi 3y agoHey can you guys elaborate how this works? I'm looking at the Ollama section in their docs and it talks about load balancing? I don't understand what that means in this context.
- SushiHippie 3y agoYou probably want to look at the getting started guide https://docs.openwebui.com/getting-started/ https://docs.openwebui.com/getting-started/ IIUC the load balancing page is for people who want to run openwebui at a larger scale https://en.wikipedia.org/wiki/Load_balancing_(computing) https://en.wikipedia.org/wiki/Load_balancing_(computing)
- byteknight 3y agoFirst test I tried to run a random taxation question through it Output: https://gist.github.com/IAmStoxe/7fb224225ff13b1902b6d1724678ea49 https://gist.github.com/IAmStoxe/7fb224225ff13b1902b6d172467... Within the first paragraph, it outputs: > GET AN ESSAY WRITTEN FOR YOU FROM AS LOW AS $13/PAGE Thought that was hilarious.
- orost 3y agoThat's not the model this post is about. You used the base model, not trained for tasks. (The instruct model is probably not on ollama yet.)
- byteknight 3y agoI absolutely did not: ollama run mixtral:8x22b EDIT: I like how you ninja-editted your comment ;)
- orost 3y agoConsidering "mixtral:8x22b" on ollama was last updated yesterday, and Mixtral-8x22B-Instruct-v0.1 (the topic of this post) was released about 2 hours ago, they are not the same model.
- byteknight 3y agoAre we looking at the same page? https://imgur.com/a/y6XfpBl https://imgur.com/a/y6XfpBl And even the direct tag page: https://ollama.com/library/mixtral:8x22b https://ollama.com/library/mixtral:8x22b shows 40-something minutes ago: https://imgur.com/a/WNhv70B https://imgur.com/a/WNhv70B
- orost 3y agoLet me clarify. Mixtral-8x22B-v0.1 was released a couple days ago. The "mixtral:8x22b" tag on ollama currently refers to it, so it's what you got when you did "ollama run mixtral:8x22b". It's a base model only capable of text completion, not any other tasks, which is why you got a terrible result when you gave it instructions. Mixtral-8x22B-Instruct-v0.1 is an instruction-following model based on Mixtral-8x22B-v0.1. It was released two hours ago and it's what this post is about. (The last updated 44 minutes ago refers to the entire "mixtral" collection.)
- kristianp 3y agoSo this one is 3x the size but only 7% better on MMLU? Given Moores law is mostly dead, this trend is going to make for even more extremely expensive compute for next gen AI models.
- stainablesteel 3y agois this different than their "large" model
- jhoechtl 3y agoDid anyone have success getting danswer and ollama to work together?
- ado__dev 3y agoWe rolled out Mixtral 8x22b to our LLM Litmus Test at s0.dev for Cody AI. Don't have enough data to say it's better or worse that other LLMs yet, but if you want to try it out for coding purposes, let me know your experience.
- CharlesW 3y agoDumb question: Are "non-instructed" versions of LLMs just raw, no-guardrail versions of the "instructed" versions that most end-users see? And why does Mixtral need one, when OpenAI LLMs do not?
- hnuser123456 3y agohttps://platform.openai.com/docs/models/gpt-base https://platform.openai.com/docs/models/gpt-base https://platform.openai.com/docs/guides/text-generation/completions-api https://platform.openai.com/docs/guides/text-generation/comp...
- CharlesW 3y agoI appreciate the correction, thanks!
- kingsleyopara 3y agoLLM’s are first trained to predict the next most likely word (or token if you want to be accurate) from web crawls. These models are basically great at continuing unfinished text but can’t really be used for instructions e.g. Q&A or chatting - this is the “non-instructed” version. These models are then fine tuned for instructions using additional data from human interaction - these are the “instructed” versions which are what end users (e.g. ChatGPT, Gemini, etc.) see.
- CharlesW 3y agoVery helpful, thank you.
- elorant 3y agoSeems that Perplexity Labs already offers a free demo of it. https://labs.perplexity.ai/ https://labs.perplexity.ai/
- yodsanklai 3y agoHow does this compare to ChatGPT4?
- orra 3y agoIs this release a pleasant surprise? Mistral weakened their commitment to open source when they partnered with Microsoft. It's nice they're using some of the money from their commercial and proprietary models, to improve the state of the art for open source (open weights) models.
- qeternity 3y agoMistral just released the most powerful open weight model in the history of humanity. How did they weaken their commitment to open weights?
- orra 3y ago> Mistral just released the most powerful open weight model in the history of humanity. Well, yeah, it's very welcome, but 'history of humanity' is hyperbole given ChatGPT isn't even two years old. > How did they weaken their commitment to open weights? Before https://web.archive.org/web/20240225001133/https://mistral.ai/ https://web.archive.org/web/20240225001133/https://mistral.a... versus after https://web.archive.org/web/20240227025408/https://mistral.ai/ https://web.archive.org/web/20240227025408/https://mistral.a... the Microsoft partnership announcement: > Committing to open models. to > That is why we started our journey by releasing the world’s most capable open-weights models There were similar changes on their about the Company page.
- zone411 3y agoIt ranks between Mistral Small and Mistral Medium on my NYT Connections benchmark and is indeed better than Command R Plus and Qwen 1.5 Chat 72B, which were the top two open weights models. Grok 1.0 is not an instruct model, so it cannot be compared fairly.
- J_Shelby_J 3y agoCan you share the details about the benchmark?
- zone411 3y agoUses an archive of 267 NYT Connections puzzles (try them yourself if unfamiliar). Three different 0-shot prompts, words in both lowercase and uppercase. One attempt per puzzle. Partial credit is awarded if not all lines are solved correctly. Top humans get near 100. Most other benchmarks don't clearly show the difference between the top models and the rest. This may be because they are older and have been over-optimized or perhaps because they are just easier.
- ajhai 3y agoIf you are looking to play with the model without installing it locally, we've added it our playground at https://trypromptly.com/playground https://trypromptly.com/playground.
- bufferoverflow 3y agoPage not found
- ajhai 3y agoSorry missed this. It was hidden behind login before. It should now be reachable.
- rbut 3y agolabs.perplexity.ai now has mixtral-8x22b-instruct. I asked it what it's knowledge cutoff was, and it said 2021-09. Anyone know why it's trained on such old data?
- PeterStuer 3y agoTaking a moment to thank Mistral for coming through with the open release. In a just world Mistral and Meta would be the ones being decorated with the 'Open AI' medal. For now in AI, if it has 'Open' in the name, it isn't.
- m463 3y agoReminds me of open source vs free software/GPL years ago, when the internet was taking off. folks like ESR were weak with their definition of open source and it wasn't as good. stallman said: ... the obvious meaning for the expression “open source software” is “You can look at the source code.” This is a much weaker criterion than free software ... https://www.gnu.org/philosophy/free-software-for-freedom.html https://www.gnu.org/philosophy/free-software-for-freedom.htm... https://www.gnu.org/philosophy/open-source-misses-the-point.en.html https://www.gnu.org/philosophy/open-source-misses-the-point....