3 ms·

The author put in the numbers, but maybe you didn’t read them. 45 t/s a second is perfectly respectable especially with no limits and 24/7 uptime with very lit

The author put in the numbers, but maybe you didn’t read them. 45 t/s a second is perfectly respectable especially with no limits and 24/7 uptime with very little power draw on the Studio. Luna is at around 100 t/s for comparison, but it’s a worse model than 5.3 Flash

sho 24d ago
The joke is that macs are famously slow at prompt prefill and you are not getting anything back in 3 seconds, or probably even 30. Once they get generating, it can be acceptable, but the TTFT is horrendous. There's a ton of well-understood things Apple can and hopefully will do to massively accelerate every stage of this pipeline and hopefully they're hard at work implementing most of them for m7.
- EagnaIonat 24d ago
  > The joke is that macs are famously slow at prompt prefill and you are not getting anything back in 3 seconds. Your knowledge is out of date. In truth it depends on the Mac and the models used. I asked this question on M5 Max 128GB, using Ollama model Quen3.8:27b-mlx, with thinking enabled. Question: "Give me a python code snippet that opens a file and sorts the lines of text. " In 2.4 seconds it gave me 4 examples that work with different sorting configurations and a summary of when to use each. Compare that to an older model of gpt-oss:20b, took 5 seconds to finish thinking and 2 seconds to stream the answer. It gave me one python example snippet and two one liners that do the same thing.
  - desterothx 24d ago
    We are talking about models of the flash size, 100s of billions of parameters, don't listen to the media, size does matter
    - EagnaIonat 24d ago
      I was just pointing out your claim that you can't get a response in 3 seconds. If I had asked the model just for the code it was under a second. Local models are good enough that it's not an issue. But keep changing the goalposts if it makes you happy.
mike_hearn 24d ago
I think it's been pretty much proven by now that there are no cases where local inferencing is better than remote inferencing, unless absolute privacy is a hard requirement. The efficiencies that come with datacenter scale and hw can't be beaten.
- akmarinov 24d ago
  Yeah, but data centers don't usually host abliterated models, hence the point of the article.
  - big-and-small 24d ago
    You can rent B300 at hourly rate and run whatever model you want.

sho 24d ago

The joke is that macs are famously slow at prompt prefill and you are not getting anything back in 3 seconds, or probably even 30. Once they get generating, it can be acceptable, but the TTFT is horrendous. There's a ton of well-understood things Apple can and hopefully will do to massively accelerate every stage of this pipeline and hopefully they're hard at work implementing most of them for m7.

EagnaIonat 24d ago
> The joke is that macs are famously slow at prompt prefill and you are not getting anything back in 3 seconds. Your knowledge is out of date. In truth it depends on the Mac and the models used. I asked this question on M5 Max 128GB, using Ollama model Quen3.8:27b-mlx, with thinking enabled. Question: "Give me a python code snippet that opens a file and sorts the lines of text. " In 2.4 seconds it gave me 4 examples that work with different sorting configurations and a summary of when to use each. Compare that to an older model of gpt-oss:20b, took 5 seconds to finish thinking and 2 seconds to stream the answer. It gave me one python example snippet and two one liners that do the same thing.
- desterothx 24d ago
  We are talking about models of the flash size, 100s of billions of parameters, don't listen to the media, size does matter
  - EagnaIonat 24d ago
    I was just pointing out your claim that you can't get a response in 3 seconds. If I had asked the model just for the code it was under a second. Local models are good enough that it's not an issue. But keep changing the goalposts if it makes you happy.

mike_hearn 24d ago

I think it's been pretty much proven by now that there are no cases where local inferencing is better than remote inferencing, unless absolute privacy is a hard requirement. The efficiencies that come with datacenter scale and hw can't be beaten.

akmarinov 24d ago
Yeah, but data centers don't usually host abliterated models, hence the point of the article.
- big-and-small 24d ago
  You can rent B300 at hourly rate and run whatever model you want.