3 ms·
What kind of functionality do you need from the model? For basic conversation and RAG, you can use tinyllama or qwen-2.5-0.5b, both of which run on a raspberry
by nico 2y ago
What kind of functionality do you need from the model?
For basic conversation and RAG, you can use tinyllama or qwen-2.5-0.5b, both of which run on a raspberry pi at around 5-20 tokens per second
- justanotheratom 2y agoI am looking for structured output at about 100-200 tokens/second on iPhone 14+. Any pointers?
- nico 2y agoThe qwq-2.5-0.5b is the tiniest useful model I've used, and pretty easy to fine-tune locally on a Mac. Haven't tried it on an iPhone, but given it runs at about 150-200 tokens/second on a Mac, I'm kinda doubtful it could do the same on an iPhone. But I guess you'd just have to try