3 ms·
How many tokens / sec are you getting on an M1 Air? Curious since I'm at work and can't try this on my Air yet hah.
by Karrot_Kream 3y ago
How many tokens / sec are you getting on an M1 Air? Curious since I'm at work and can't try this on my Air yet hah.
- anonyfox 3y agofeels roughly like the same speed as GPT3.5 in the browser UI
- Patrick_Devine 3y agoI'm getting >30 tokens/sec using it with ollama and an M2 Pro. That might be a little slow though because I have a background finetuning job running.
- minzi 3y agoBit of a tangential question here, but any recommendations on how to get started fine tuning this model (or ones like it)? I feel like there are a million different tutorial and ways of doing it when I google.
- Karrot_Kream 3y agoTake a look at the QLoRA repo https://github.com/artidoro/qlora/ https://github.com/artidoro/qlora/ which has an example finetuning Llama. Made by the authors of the QLoRA paper.
- brucethemoose2 3y agohttps://github.com/OpenAccess-AI-Collective/axolotl https://github.com/OpenAccess-AI-Collective/axolotl This is a wrapper around many training methods, and it has yielded many excellent community finetunes already.
- brucethemoose2 3y agoIts the same speed as llama 7B, so very quick.
- benkaiser 3y ago~13 tokens per second. I asked for a story about goldilocks and this was the timings on my M1 air using `ollama run mistral --verbose`: total duration: 33.964492834s load duration: 1.471584ms prompt eval count: 8 token(s) prompt eval duration: 385.334ms prompt eval rate: 20.76 tokens/s eval count: 418 token(s) eval duration: 31.887033s eval rate: 13.11 tokens/s