4 ms·
It is important to note here that SWA is not supported by all models, e.g. Mixtral is not working correctly.
by rf15 3y ago
It is important to note here that SWA is not supported by all models, e.g. Mixtral is not working correctly.
- woadwarrior01 3y agoJust to be clear, Mistral 7B based models work best with sliding window attention. But they seem to have recently disabled[1][2] SWA for Mixtral, their MoE model. I learnt that from this[3] r/LocalLLaMA post, yesterday. [1]: https://huggingface.co/mistralai/Mixtral-8x7B-Instruct-v0.1/commit/125c431e2ff41a156b9f9076f744d2f35dd6e67a https://huggingface.co/mistralai/Mixtral-8x7B-Instruct-v0.1/... [2]: https://huggingface.co/mistralai/Mixtral-8x7B-v0.1/commit/58301445dc1378584211722b7ebf8743ec4e192b https://huggingface.co/mistralai/Mixtral-8x7B-v0.1/commit/58... [3]: https://old.reddit.com/r/LocalLLaMA/comments/18k0fek/ https://old.reddit.com/r/LocalLLaMA/comments/18k0fek/