Just to be clear, Mistral 7B based models work best with sliding window attention. But they seem to have recently disabled[1][2] SWA for Mixtral, their MoE mode

Just to be clear, Mistral 7B based models work best with sliding window attention. But they seem to have recently disabled[1][2] SWA for Mixtral, their MoE model. I learnt that from this[3] r/LocalLLaMA post, yesterday. [1]: https://huggingface.co/mistralai/Mixtral-8x7B-Instruct-v0.1/commit/125c431e2ff41a156b9f9076f744d2f35dd6e67a https://huggingface.co/mistralai/Mixtral-8x7B-Instruct-v0.1/... [2]: https://huggingface.co/mistralai/Mixtral-8x7B-v0.1/commit/58301445dc1378584211722b7ebf8743ec4e192b https://huggingface.co/mistralai/Mixtral-8x7B-v0.1/commit/58... [3]: https://old.reddit.com/r/LocalLLaMA/comments/18k0fek/ https://old.reddit.com/r/LocalLLaMA/comments/18k0fek/