4 ms·
One thing that's worth mentioning about llama.cpp wrappers like ollama, LM Studio and Faraday is that they don't yet support[1] sliding window attention, and in
by woadwarrior01 3y ago
One thing that's worth mentioning about llama.cpp wrappers like ollama, LM Studio and Faraday is that they don't yet support[1] sliding window attention, and instead use vanilla causal attention from llama2. As noted in the Mistral 7B paper[2], SWA has some benefits in terms of attention span over regular causal attention.
Disclaimer: I have a competing universal macOS/iOS app[3] that does support SWA with Mistral models (using mlc-llm).
[1]: https://github.com/ggerganov/llama.cpp/issues/3377 https://github.com/ggerganov/llama.cpp/issues/3377
[2]: https://arxiv.org/abs/2310.06825 https://arxiv.org/abs/2310.06825
[3]: https://apps.apple.com/us/app/private-llm/id6448106860 https://apps.apple.com/us/app/private-llm/id6448106860
- rf15 3y agoIt is important to note here that SWA is not supported by all models, e.g. Mixtral is not working correctly.
- woadwarrior01 3y agoJust to be clear, Mistral 7B based models work best with sliding window attention. But they seem to have recently disabled[1][2] SWA for Mixtral, their MoE model. I learnt that from this[3] r/LocalLLaMA post, yesterday. [1]: https://huggingface.co/mistralai/Mixtral-8x7B-Instruct-v0.1/commit/125c431e2ff41a156b9f9076f744d2f35dd6e67a https://huggingface.co/mistralai/Mixtral-8x7B-Instruct-v0.1/... [2]: https://huggingface.co/mistralai/Mixtral-8x7B-v0.1/commit/58301445dc1378584211722b7ebf8743ec4e192b https://huggingface.co/mistralai/Mixtral-8x7B-v0.1/commit/58... [3]: https://old.reddit.com/r/LocalLLaMA/comments/18k0fek/ https://old.reddit.com/r/LocalLLaMA/comments/18k0fek/
- mark_l_watson 3y agoI like your app Private LLM. I run it on an old M1 iPad Pro with 16G memory. That said, most of my experiments with integrating LLMs with programs written in Python, Racket, and Common Lisp use Ollama’s REST service. I have a Mac Mini with 32B and it blows my mind how effective an inexpensive energy efficient home computer is for running LLMs.
- swah 3y agoDo the models on your apps work well for coding tasks?
- woadwarrior01 3y agoNot on iOS. On macOS, I personally think WizardLM 13B v1.2 is a very strong model and keep hearing good things about it from users on our discord and in support emails. Now that there's OmniQuant support for Mixtral models[1], I'm plan to add support for Mixtral-8x7B-Instruct-v0.1 in the next version of the macOS app, which in my tests, looks like a very good all purpose model that's also pretty good at coding. It's pretty memory hungry (~41GB of RAM), but that's the price to pay for an uncompromising implementation. Existing quantized implementations quantize the MoE gates, leading to a significant drop in perplexity when compared with results from fp16 inference. [1]: https://github.com/OpenGVLab/OmniQuant/commit/798467 https://github.com/OpenGVLab/OmniQuant/commit/798467
- smcleod 3y agoAh, I tried your app out just the other day. It sounded interesting but I couldn’t see a way to download models (on iOS anyway) and it seemed quite slow compared to MLC Chat but perhaps that was the built in model?
- woadwarrior01 3y agoCurrently, the iOS app only allows downloading the 7B Llama 2 Uncensored model on iPhone 13 Pro and newer phones and also Apple Silicon iPads. I'm working on an update to allow downloading multiple models from the StableLM 3B, Llama 2 7B and Mistral 7B based models. As before, due to memory and compute constraints, 7B models can only be downloaded on the aforementioned devices.