23 ms·

Simple on-device iOS AI demo app of TinyStories-1M

5 points by omeze 3y ago

omeze 3y ago
Hey all, After reading the TinyStories paper (using small language models of 1m to 33m parameter size to generate short stories), me and a friend decided to port it as a test to see what performance of tiny LMs look like on mobile devices. The stories it generates aren't great or anything, it's really just a toy tech demo to play with. We used HuggingFace's exporters library to do this. You can try it out here: https://apps.apple.com/us/app/tinystories-on-device/id6451497115 https://apps.apple.com/us/app/tinystories-on-device/id645149... (we are not affiliated with MS research or the authors)
brucethemoose2 3y ago
If y'all are looking for another iOS port, try a llama 7B or 13B storytelling model running on llama.cpp/koboldcpp or the mlc-llm backend. 7B in particular should be fast and usable on most iOS devices. 13B should work on 8GB of RAM. Both are quite coherent and "smart." Theres a tragic gap between these optimized implementations and mobile apps, for some reason.
- omeze 3y ago
  yea, I was also surprised at the lack of apps. I guess we can take a shot at llama-7b, maybe a base model and a storytelling fine tune. we'll have to look into which compiler is best, I do want to tinker with the mlc toolchain. will post to HN when we get it working!
  - brucethemoose2 3y ago
    MLC and llama.cpp both have performant metal ports, and mlc has a demo ios app! For story finetunes, I am out of the loop... You have to dig through HF by date to find them, for instance: https://huggingface.co/zarakiquemparte/hermesboros-limarp-7b https://huggingface.co/zarakiquemparte/hermesboros-limarp-7b https://huggingface.co/models?p=1&sort=modified&search=7b https://huggingface.co/models?p=1&sort=modified&search=7b You can also look through what people are hosting on Kobold Horde (a storytelling/rp community) ignoring the old pygmalion/opt models: https://lite.koboldai.net/# https://lite.koboldai.net/# Also, you should at least host an instruct model instead of vanilla llama 7B.

5 points by omeze 3y ago

Hey all, After reading the TinyStories paper (using small language models of 1m to 33m parameter size to generate short stories), me and a friend decided to port it as a test to see what performance of tiny LMs look like on mobile devices. The stories it generates aren't great or anything, it's really just a toy tech demo to play with. We used HuggingFace's exporters library to do this. You can try it out here: https://apps.apple.com/us/app/tinystories-on-device/id6451497115 https://apps.apple.com/us/app/tinystories-on-device/id645149... (we are not affiliated with MS research or the authors)

brucethemoose2 3y ago

If y'all are looking for another iOS port, try a llama 7B or 13B storytelling model running on llama.cpp/koboldcpp or the mlc-llm backend. 7B in particular should be fast and usable on most iOS devices. 13B should work on 8GB of RAM. Both are quite coherent and "smart." Theres a tragic gap between these optimized implementations and mobile apps, for some reason.

omeze 3y ago
yea, I was also surprised at the lack of apps. I guess we can take a shot at llama-7b, maybe a base model and a storytelling fine tune. we'll have to look into which compiler is best, I do want to tinker with the mlc toolchain. will post to HN when we get it working!
- brucethemoose2 3y ago
  MLC and llama.cpp both have performant metal ports, and mlc has a demo ios app! For story finetunes, I am out of the loop... You have to dig through HF by date to find them, for instance: https://huggingface.co/zarakiquemparte/hermesboros-limarp-7b https://huggingface.co/zarakiquemparte/hermesboros-limarp-7b https://huggingface.co/models?p=1&sort=modified&search=7b https://huggingface.co/models?p=1&sort=modified&search=7b You can also look through what people are hosting on Kobold Horde (a storytelling/rp community) ignoring the old pygmalion/opt models: https://lite.koboldai.net/# https://lite.koboldai.net/# Also, you should at least host an instruct model instead of vanilla llama 7B.