4 ms·
I'm out of the loop for local models. For my M3 24gb ram macbook, what token throughput can I expect? Edit: I tried it out, I have no idea in terms of of token
by jcmontx 1y ago
I'm out of the loop for local models. For my M3 24gb ram macbook, what token throughput can I expect?
Edit: I tried it out, I have no idea in terms of of tokens but it was fluid enough for me. A bit slower than using o3 in the browser but definitely tolerable. I think I will set it up in my GF's machine so she can stop paying for the full subscription (she's a non-tech professional)
- steinvakt2 1y agoWondering about the same for my M4 max 128 gb
- jcmontx 1y agoIt should fly on your machine
- steinvakt2 1y agoYeah, was super quick and easy to set up using Ollama. I had to kill some processes first to avoid memory swap though (even with 128gb memory). So a slightly more quantized version is maybe ideal, for me at least. Edit: I'm talking about the 120B model of course
- coolspot 1y ago40 t/s
- dantetheinferno 1y agoApple M4 Pro w/ 48GB running the smaller version. I'm getting 43.7t/s
- GHanku 1y ago[dead]
- albertgoeswoof 1y ago3 year old M1 MacBook Pro 32gb, 42 tokens/sec on lm studio Very much usable
- ivape 1y agoCurious if anyone is running this on a AMD Ryzen AI Max+ 395 and knows the t/s.