4 ms·
> The joke is that macs are famously slow at prompt prefill and you are not getting anything back in 3 seconds. Your knowledge is out of date. In truth it depe
by EagnaIonat 26d ago
> The joke is that macs are famously slow at prompt prefill and you are not getting anything back in 3 seconds.
Your knowledge is out of date. In truth it depends on the Mac and the models used.
I asked this question on M5 Max 128GB, using Ollama model Quen3.8:27b-mlx, with thinking enabled.
Question: "Give me a python code snippet that opens a file and sorts the lines of text. "
In 2.4 seconds it gave me 4 examples that work with different sorting configurations and a summary of when to use each.
Compare that to an older model of gpt-oss:20b, took 5 seconds to finish thinking and 2 seconds to stream the answer. It gave me one python example snippet and two one liners that do the same thing.
- desterothx 26d agoWe are talking about models of the flash size, 100s of billions of parameters, don't listen to the media, size does matter
- EagnaIonat 26d agoI was just pointing out your claim that you can't get a response in 3 seconds. If I had asked the model just for the code it was under a second. Local models are good enough that it's not an issue. But keep changing the goalposts if it makes you happy.