4 ms·
Looking at how to deploy 1B and 3B Llama models on Android for inference. Some posts online recommend using Termux (an amazing app) to have an emulated shell an
by yuvalr1 2y ago
Looking at how to deploy 1B and 3B Llama models on Android for inference. Some posts online recommend using Termux (an amazing app) to have an emulated shell and then install as if it's Linux, using ollama for example. However, this forces you into a manual installation process, and also most of the people don't know what Termux is, and would be afraid to install it from F-Droid.
Maybe someone can recommend a way to deploy Llama to Android without Termux, maybe even something that can be potentially fully implemented inside an app?
I'm currently looking into compiling llama.cpp for Android and bundling it inside an app. Is that a viable path? Would love to hear from someone who tried something similar.
- tugdual 2y agoI actually did something similar using llama.cpp a while back, would be curious to see the speedup with this model. https://github.com/TugdualKerjan/bunny/tree/main https://github.com/TugdualKerjan/bunny/tree/main
- antonvs 2y agoThis might be of use: https://github.com/a-ghorbani/pocketpal-ai https://github.com/a-ghorbani/pocketpal-ai
- deleted 2y ago[deleted]
- niutech 2y agoYou can use MLC LLM: https://llm.mlc.ai/ https://llm.mlc.ai/