34 ms·
> It currently generates 5–6 tok/s on an 8 GB M2 MacBook Air and 31–35 tok/s on an M5 MacBook Pro. Where does this big a performance spread come from? I wouldn
by addaon 2mo ago
> It currently generates 5–6 tok/s on an 8 GB M2 MacBook Air and 31–35 tok/s on an M5 MacBook Pro.
Where does this big a performance spread come from? I wouldn't naïvely expect SSD performance difference to be that big, and I would expect SSD performance to dominate...
- afzalive 2mo agoThe M5 MBP has 24GB of RAM, more context in RAM perhaps?
- gitpusher42 2mo agoThe process stays at around 2 GB with 16 slots and a 4K context on both the M5 and M2. But yeah, Apple might be doing some magic under the hood
- petu 2mo agoUnused RAM is wasted RAM. So not really Apple magic, about every OS uses "free" memory as disk cache. Try to leave only a gigabyte or two free, speed likely would drop dramatically. Edit: or do some calculation / logging of experts read speed, to see if it's faster than SSD spec.
- deleted 2mo ago[deleted]
- deleted 2mo ago[deleted]
- gitpusher42 2mo agoyeah, looks like a page cache matters a lot I tested on mine m5 pro with 8gb memory pressure, got 27t/s instead of 35t/s Someone tested on m4 max. In regular state it was 48tok/s, but 32-42 with memory pressure
- wongarsu 2mo agoMy suspicion is that this is simply due to the M5 having more memory, and the OS already having most of the file cached. The M2 has more memory pressure and would cache fewer of the SSD reads If that's true, inference speed would be even lower if you have only 2GB total, including OS caches
- GeekyBear 2mo agoThe base level M5 doesn't just have more memory than the base level M2. The memory bandwidth is bumped up by 50%, and the size of the on-die system level cache is bumped up by 50% as well.
- icedchai 2mo agoI recently got an M5 Air as a second machine. I question if I'll even need a Pro machine in the future, as long as I can get enough RAM with the base chip (which seems unlikely, but...)
- GeekyBear 2mo agoThe M5 SSD's performance uplift was fairly substantial, even when compared to the prior generation. > In the Blackmagic Disk Speed Test, the SSD in the M5 MacBook Pro achieved read speeds of up to 6,323 MB/s, compared to just 2,031 MB/s on the M4 MacBook Pro. It's not like the M4 is "slow" in a vacuum, but the M5 SSD is over three times faster, which is a great generation uplift. https://www.tomshardware.com/laptops/macbooks/m5-macbook-pros-ssd-is-2-5x-faster-on-average-than-last-gen-m4-exceeding-apples-own-claims-m5-achieves-6-000-mb-s-across-both-read-and-write-speeds https://www.tomshardware.com/laptops/macbooks/m5-macbook-pro...
- ambicapter 2mo agoExtremely impressive.
- Barbing 2mo agoNot sure Data (Star Trek TNG) could read that fast. “Accessing”
- deleted 2mo ago[deleted]
- Schiendelman 2mo agoNot only is it older, so Pro:Pro it would have much slower SSD, but doesn't the Air also have slower SSD than the Pro in the same generation? And maybe narrower memory bandwidth?
- tracker1 2mo agoIIRC, depends on the SSD size.. larger sizes had 2x the bandwidth, so it depends. That's combined/offset with the M5 improvements even further.
- gitpusher42 2mo agoI am relying quite heavily on system caching and pread. And yeah, M5 is a way faster and I can guess Mac can cache something, even if process stays under 2gb. It was 83ms read per token for M2 and 12ms on M5 pro. Total is 163ms/tok vs 30ms/tok for M5. So yeah, there is a faster read and faster gpu processing