4 ms·

I'm surprised no one has else has mentioned - low power mode. With no speculative decoding, using high power mode, I get 80 t/s on 35B A3B - and it gets hot an

I'm surprised no one has else has mentioned - low power mode. With no speculative decoding, using high power mode, I get 80 t/s on 35B A3B - and it gets hot and spins up. On low power mode I get 38 t/s - no fans, cool to warm laptop. If you currently don't use speculative decoding and you start using it, it can nearly offset the difference between high and low power, and it's night and day experience. I almost always keep my laptop on low power mode.

anon373839 4mo ago
Can you mention what inference stack you're using? I've tried MTP several times with that model and it always seems to significantly cut my token generation speed from ~60 tokens/sec to ~40 (M3 Max).
- jasonjmcghee 3mo ago
  (see above reply to myself) I misattributed the gain from dflash - it was dflash-mlx library + mlx model, not dflash itself giving me the speedup.
mycall 4mo ago
It is less efficient use of the GPU and uses more electricity overall, no?
- spider-mario 4mo ago
  Oh no, 0.6 kWh a day!
- bigyabai 4mo ago
  Yes, this is a tradeoff that foregoes the efficiency of race-to-idle.
html5cat 4mo ago
Awesome idea! Will try it out. Wish there was a way to enable low power on a per-app basis. Scrolling and reading on low power mode is really annoying.
- the_lucifer 4mo ago
  > Wish there was a way to enable low power on a per-app basis. Since you can control the low power mode setting from the command line: `sudo pmset -a lowpowermode 1`. It should be pretty straightforward to hook this up to Hammerspoon[1] using hs.application.frontmostApplication() to apply the setting based on whatever foreground application you choose. Thinking out loud, that being said, the necessity of sudo might make this slightly more complex. An always on background admin agent might be needed I suppose to bypass the password prompts (or add pmset to the sudoers file, if you prefer). [1]: https://www.hammerspoon.org/ https://www.hammerspoon.org/
  - kmacdough 4mo ago
    Unfortunately doesn't cover scrolling HN while the agent toils away.
c16 4mo ago
Will give this a try later. Enjoy working with A3B Coder, but the heat coming out my 32gb M5 is a lot. This might be the trick - Thanks!
- c16 4mo ago
  Update: This works super well. Thanks.
Lwerewolf 4mo ago
Same with ds4-flash. Low power is 13t/s vs 26t/s, power usage is ~30w vs ~100-120w. I still use high power because my m5 max is basically a server with built-in UPS and screen (and an off-work media machine, blender cycles toy, etc - growing fond of it, actually), but even 10tok/s is usable with a relatively smart model. Caveat is that the code that I touch barely has any boilerplate and I like keeping it lean.
jasonjmcghee 3mo ago
I usually get mlx versions of models and didn't realize I was using non-mlx - misattributing the performance boost to speculative decoding. With dflash-mlx library, with dflash disabled 3.6-35b-a3b mlx model I'm getting ~60 t/s on low power and >100 t/s on high power. Compared with previous message being official qwen huggingface release (non-mlx) using lm studio.

anon373839 4mo ago

Can you mention what inference stack you're using? I've tried MTP several times with that model and it always seems to significantly cut my token generation speed from ~60 tokens/sec to ~40 (M3 Max).

jasonjmcghee 3mo ago
(see above reply to myself) I misattributed the gain from dflash - it was dflash-mlx library + mlx model, not dflash itself giving me the speedup.

mycall 4mo ago

It is less efficient use of the GPU and uses more electricity overall, no?

spider-mario 4mo ago
Oh no, 0.6 kWh a day!
bigyabai 4mo ago
Yes, this is a tradeoff that foregoes the efficiency of race-to-idle.

html5cat 4mo ago

Awesome idea! Will try it out. Wish there was a way to enable low power on a per-app basis. Scrolling and reading on low power mode is really annoying.

the_lucifer 4mo ago
> Wish there was a way to enable low power on a per-app basis. Since you can control the low power mode setting from the command line: `sudo pmset -a lowpowermode 1`. It should be pretty straightforward to hook this up to Hammerspoon[1] using hs.application.frontmostApplication() to apply the setting based on whatever foreground application you choose. Thinking out loud, that being said, the necessity of sudo might make this slightly more complex. An always on background admin agent might be needed I suppose to bypass the password prompts (or add pmset to the sudoers file, if you prefer). [1]: https://www.hammerspoon.org/ https://www.hammerspoon.org/
- kmacdough 4mo ago
  Unfortunately doesn't cover scrolling HN while the agent toils away.

c16 4mo ago

Will give this a try later. Enjoy working with A3B Coder, but the heat coming out my 32gb M5 is a lot. This might be the trick - Thanks!

c16 4mo ago
Update: This works super well. Thanks.

Lwerewolf 4mo ago

Same with ds4-flash. Low power is 13t/s vs 26t/s, power usage is ~30w vs ~100-120w. I still use high power because my m5 max is basically a server with built-in UPS and screen (and an off-work media machine, blender cycles toy, etc - growing fond of it, actually), but even 10tok/s is usable with a relatively smart model. Caveat is that the code that I touch barely has any boilerplate and I like keeping it lean.

jasonjmcghee 3mo ago

I usually get mlx versions of models and didn't realize I was using non-mlx - misattributing the performance boost to speculative decoding. With dflash-mlx library, with dflash disabled 3.6-35b-a3b mlx model I'm getting ~60 t/s on low power and >100 t/s on high power. Compared with previous message being official qwen huggingface release (non-mlx) using lm studio.