5 ms·
When people talk about running a (quantized) medium-sized model on a Mac Mini, what types of latency and throughput times are they talking about? Do they mean l
by bogtog 1y ago
When people talk about running a (quantized) medium-sized model on a Mac Mini, what types of latency and throughput times are they talking about? Do they mean like 5 tokens per second or at an actually usable speed?
- deleted 1y ago[deleted]
- n42 1y agohere's a quick recording from the 20b model on my 128GB M4 Max MBP: https://asciinema.org/a/AiLDq7qPvgdAR1JuQhvZScMNr https://asciinema.org/a/AiLDq7qPvgdAR1JuQhvZScMNr and the 120b: https://asciinema.org/a/B0q8tBl7IcgUorZsphQbbZsMM https://asciinema.org/a/B0q8tBl7IcgUorZsphQbbZsMM I am, um, floored
- Davidzheng 1y agothe active param count is low so it should be fast.
- Rhubarrbb 1y agoGeneration is usually fast, but prompt processing is the main limitation with local agents. I also have a 128 GB M4 Max. How is the prompt processing on long prompts? processing the system prompt for Goose always takes quite a while for me. I haven't been able to download the 120B yet, but I'm looking to switch to either that or the GLM-4.5-Air for my main driver.
- anonymoushn 1y agoit's odd that the result of this processing cannot be cached.
- lostmsu 1y agoIt can be and it is by most good processing frameworks.
- ghc 1y agoHere's a sample of running the 120b model on Ollama with my MBP: ``` total duration: 1m14.16469975s load duration: 56.678959ms prompt eval count: 3921 token(s) prompt eval duration: 10.791402416s prompt eval rate: 363.34 tokens/s eval count: 2479 token(s) eval duration: 1m3.284597459s eval rate: 39.17 tokens/s ```
- andai 1y agoYou mentioned "on local agents". I've noticed this too. How do ChatGPT and the others get around this, and provide instant responses on long conversations?
- bluecoconut 1y agoNot getting around it, just benefiting from parallel compute / huge flops of GPUs. Fundamentally, it's just that prefill compute is itself highly parallel and HBM is just that much faster than LPDDR. Effectively H100s and B100s can chew through the prefill in under a second at ~50k token lengths, so the TTFT (Time to First Token) can feel amazingly fast.
- mike_hearn 1y agoThey cache the intermediate data (KV cache).
- phonon 1y agoHere's a 4bit 70B parameter model, https://www.youtube.com/watch?v=5ktS0aG3SMc https://www.youtube.com/watch?v=5ktS0aG3SMc (deepseek-r1:70b Q4_K_M) on a M4 Max 128 GB. Usable, but not very performant.
- a_wild_dandan 1y agoGLM-4.5-air produces tokens far faster than I can read on my MacBook. That's plenty fast enough for me, but YMMV.
- davio 1y agoOn a M1 MacBook Air with 8GB, I got this running Gemma 3n: 12.63 tok/sec • 860 tokens • 1.52s to first token I'm amazed it works at all with such limited RAM
- v5v3 1y agoI have started a crowdfunding to get you a MacBook air with 16gb. You poor thing.
- bookofjoe 1y agoUp the ante with an M4 chip
- backscratches 1y agonot meaningfully different, m1 virtually as fast as m4
- wahnfrieden 1y agohttps://github.com/devMEremenko/XcodeBenchmark https://github.com/devMEremenko/XcodeBenchmark M4 is almost twice as fast as M1
- andai 1y agoIn this table, M4 is also twice as fast as M4.
- wahnfrieden 1y agoYou're comparing across vanilla/Pro/Max tiers. within equivalent tier, M4 is almost 2x faster than M1
- v5v3 1y agoTwice the cost too.