4 ms·

Did they release the source code or benchmarks for their new 8x model?

• espadrine 3y ago
  No. Without the inference code, the best we can have are guesses on its implementation, so the benchmark figures we can get could be quite wrong. It does seem better than Llama2-70B in my tests[0], which rely on the work done by Dmytro Dzhulgakov[1] and DiscoResearch[2]. But the point of releasing on bittorrent is to see the effervescence in hobbyist research and early attempts at MoE quantization, which are already ongoing[3]. They are benefitting from the community. [0]: https://twitter.com/espadrine/status/1733505332293288256 https://twitter.com/espadrine/status/1733505332293288256 [1]: https://github.com/dzhulgakov/llama-mistral https://github.com/dzhulgakov/llama-mistral [2]: https://huggingface.co/DiscoResearch/mixtral-7b-8expert https://huggingface.co/DiscoResearch/mixtral-7b-8expert [3]: https://github.com/TimDettmers/bitsandbytes/tree/sparse_moe https://github.com/TimDettmers/bitsandbytes/tree/sparse_moe
• mrkramer 3y ago
  Apparently you can download it via torrent[0]. https://venturebeat.com/ai/mistral-ai-bucks-release-trend-by-dropping-torrent-link-to-new-open-source-llm/ https://venturebeat.com/ai/mistral-ai-bucks-release-trend-by...
  • capableweb 3y ago
    First-hand source: https://twitter.com/MistralAI/status/1706877320844509405 https://twitter.com/MistralAI/status/1706877320844509405
• emadm 3y ago
  Mistral doesn't release source code or data just weights. They did include some details of sliding window attention previously
  • ebalit 3y ago
    They released a reference implementation with Mistral 7B and a fork of a framework they probably used to train the Mixture of Experts for the new one. But it's still early and I guess that they kept just enough mystery to kickstart the conversation around this new model!