3 ms·
Running it on a MacBook with M1 Pro chip and 32 GB of RAM is quite slow. I expected to be as fast as phi4 but it's much slower.
by mike31fr 2y ago
Running it on a MacBook with M1 Pro chip and 32 GB of RAM is quite slow. I expected to be as fast as phi4 but it's much slower.
- mike31fr 2y agoWith eval rate numbers: - phi4: 12 tokens/s - mistral-small: 9 tokens/s On Nvidia RTX 4090 laptop: - phi4: 36 tokens/s - mistral-small: 16 tokens/s