5 ms·
I am looking for structured output at about 100-200 tokens/second on iPhone 14+. Any pointers?
by justanotheratom 1y ago
I am looking for structured output at about 100-200 tokens/second on iPhone 14+. Any pointers?
- nico 1y agoThe qwq-2.5-0.5b is the tiniest useful model I've used, and pretty easy to fine-tune locally on a Mac. Haven't tried it on an iPhone, but given it runs at about 150-200 tokens/second on a Mac, I'm kinda doubtful it could do the same on an iPhone. But I guess you'd just have to try