4 ms·
They just released a much larger model as a torrent. Probably the first open source mixture-of-experts model. Waiting for them to drop the code and explain it.
by ilaksh 3y ago
They just released a much larger model as a torrent. Probably the first open source mixture-of-experts model. Waiting for them to drop the code and explain it.
- capableweb 3y agoWhich one? Everything on their website is still about the 7B model. 7B is the only model available on their huggingface page too: https://huggingface.co/mistralai https://huggingface.co/mistralai Edit: I'm guessing this is the model referenced: "Mistral "Mixtral" 8x7B 32k model [magnet]" - https://news.ycombinator.com/item?id=38570537 https://news.ycombinator.com/item?id=38570537
- Art9681 3y agoThat's it. The community is working on getting it implemented in the popular inference engines. It came out less then 48 hours ago so give it a few days and we should be able to run it and test it at various quantizations depending on how much compute power you have.
- bugglebeetle 3y agoThere’s an initial version of it already up and running: https://huggingface.co/DiscoResearch/mixtral-7b-8expert https://huggingface.co/DiscoResearch/mixtral-7b-8expert
- ilaksh 3y agoI am doubtful about these efforts because people are just guessing about how it is supposed to work since Mistral didn't release any code or information yet.
- bugglebeetle 3y agoYes, there are probably some implementation pieces missing. But Disco Research also punches way above their weight on open source LLM stuff, so I would gamble it’s not that far off. They also just released a chat fine-tune of the model just moments ago: https://huggingface.co/DiscoResearch/DiscoLM-mixtral-8x7b-v2 https://huggingface.co/DiscoResearch/DiscoLM-mixtral-8x7b-v2
- MacsHeadroom 3y agoPeople had Mixtral 8x7B running on Twitter within 3 hours of the torrent being posted. It's already been independently benchmarked and finetunes are cooking.