4 ms·
If y'all are looking for another iOS port, try a llama 7B or 13B storytelling model running on llama.cpp/koboldcpp or the mlc-llm backend. 7B in particular sho
by brucethemoose2 3y ago
If y'all are looking for another iOS port, try a llama 7B or 13B storytelling model running on llama.cpp/koboldcpp or the mlc-llm backend.
7B in particular should be fast and usable on most iOS devices. 13B should work on 8GB of RAM. Both are quite coherent and "smart."
Theres a tragic gap between these optimized implementations and mobile apps, for some reason.
- omeze 3y agoyea, I was also surprised at the lack of apps. I guess we can take a shot at llama-7b, maybe a base model and a storytelling fine tune. we'll have to look into which compiler is best, I do want to tinker with the mlc toolchain. will post to HN when we get it working!
- brucethemoose2 3y agoMLC and llama.cpp both have performant metal ports, and mlc has a demo ios app! For story finetunes, I am out of the loop... You have to dig through HF by date to find them, for instance: https://huggingface.co/zarakiquemparte/hermesboros-limarp-7b https://huggingface.co/zarakiquemparte/hermesboros-limarp-7b https://huggingface.co/models?p=1&sort=modified&search=7b https://huggingface.co/models?p=1&sort=modified&search=7b You can also look through what people are hosting on Kobold Horde (a storytelling/rp community) ignoring the old pygmalion/opt models: https://lite.koboldai.net/# https://lite.koboldai.net/# Also, you should at least host an instruct model instead of vanilla llama 7B.