4 ms·

Doesn’t need to be a winner head to head. If it can do 90% of the tasks the big boys do, at 50% speed, for virtually no extra overhead cost save for the power c

Doesn’t need to be a winner head to head. If it can do 90% of the tasks the big boys do, at 50% speed, for virtually no extra overhead cost save for the power consumed by a prompt - that’s gonna work for a lot of people. And that’s also basically where we’re at today. Qwen3.6 35b running quantized on 10 year old hardware solves basically all of my uses cases for agents except for coding. The frontier models are faster, and better at coding, but not so much that i’ll pay $200/month for them.

drnick1 4mo ago
Consider this. One of the smallest Qwen models (4B parameters) powers my home automation voice assistant, and runs on CPU alone at >20 tok/s. It is enough for that use case, and could be made even better/faster with a modest GPU. It isn't as smart as some cloud-connected thingamajig, but I would never allow a literal Google or Amazon bug in my home. Huge SOTA models aren't relevant everywhere. Most people use LLMs for rather trivial tasks such as finding typos or drafting text.
- dainiusse 4mo ago
  Curious, what exactly does it do for you? I has bad luck with these small models to do anything useful tbh.
  - drnick1 4mo ago
    It's a voice assistant that can respond to commands such as "turn on the light," or explain things (within the abilities of the small model).
- marci 4mo ago
  But with Apple's AFM 3 architecture, we might end up with huge SOTA adjacent on devices with limited RAM. They use a technique where you only load between 1B and 4B of a 20B dense model for an entire prompt run, not token by token like a MoE, and use mostly the low power ANE instead of GPU cores. Now, imagine if/when they scale up to 100B or more? On a chip using 2W?
  - kinnth 4mo ago
    I think we're also ignoring a potential innovative move in how models work. If someone could splinter or fragment the models into more specific tasks i.e "spellchecker AI" and get these working as well as Sonnet 4.6-4.8 on those tasks on a personal laptop. You then question the $100 a month fee. Bear in mind these laptops are likely to be $5000 or so because of the memory, HDD and M7 chip they likely need. It feels to me like the beginning of the inflection point but software updates not hardware updates will be the accelerant.
    - marci 4mo ago
      "That’s where EMO comes in. We show that EMO – a 1B-active, 14B-total-parameter (8-expert active, 128-expert total) MoE trained on 1 trillion tokens – supports selective expert use: for a given task or domain, we can use only a small subset of experts (just 12.5% of total experts) while retaining near full-model performance." https://allenai.org/blog/emo https://allenai.org/blog/emo
overfeed 4mo ago
> If it can do 90% of the tasks the big boys do, at 50% speed I want to live in this world too, but these numbers, as of today, are very aspirational and far removed from reality. I'm no tokenmaxxer; I find my modest local setup useful, I also know the limitations, it's slow and it sucks (relatively) at high-level and/or long-context planning, compared to frontier models. Only a minority of my prompts are max-effort - its not all I do, but, it also means frontier labs aren't dying any time soon
- mikestorrent 4mo ago
  Consider also that right now LLMs run slowly enough you can watch them think. I've seen a demo of an LLM running at an absurdly high speed and it reminds me of when I moved from a 2400 baud modem to a 14.4 - BBS screens that I could watch draw were all of a sudden nigh-interactive. Faster-than-realtime video generation is also coming, and will also continue to require huge hardware for a long while yet. I love local models - I have a machine at home that runs a few for me and it's a lot of fun - but for the time being they are not super trustworthy on tool calls and staying on script. Another year or so might change all that!
  - ChickeNES 4mo ago
    What does your local setup look like?
    - deleted 4mo ago
      [deleted]
    - mikestorrent 4mo ago
      1650 watts of liquid cooled Risc-V
  - KoolKat23 4mo ago
    If anyone wishes to see the future. A fast LLM is quite eye-opening. I think chatjimmy uses Talaas' chips where models are hardcoded into the silicon. https://chatjimmy.ai/ https://chatjimmy.ai/
SanjayMehta 4mo ago
This is what makes sense for me as well. All I need a local model is for playing with simple graphics: no gradients, at most ten colours which I can push through VTracer to get an SVG. Draw Things does the job, usually in 120 seconds or less. Sometimes, I need a quick throwaway bit of python. That can take 30 minutes of my time.

drnick1 4mo ago

Consider this. One of the smallest Qwen models (4B parameters) powers my home automation voice assistant, and runs on CPU alone at >20 tok/s. It is enough for that use case, and could be made even better/faster with a modest GPU. It isn't as smart as some cloud-connected thingamajig, but I would never allow a literal Google or Amazon bug in my home. Huge SOTA models aren't relevant everywhere. Most people use LLMs for rather trivial tasks such as finding typos or drafting text.

dainiusse 4mo ago
Curious, what exactly does it do for you? I has bad luck with these small models to do anything useful tbh.
- drnick1 4mo ago
  It's a voice assistant that can respond to commands such as "turn on the light," or explain things (within the abilities of the small model).
marci 4mo ago
But with Apple's AFM 3 architecture, we might end up with huge SOTA adjacent on devices with limited RAM. They use a technique where you only load between 1B and 4B of a 20B dense model for an entire prompt run, not token by token like a MoE, and use mostly the low power ANE instead of GPU cores. Now, imagine if/when they scale up to 100B or more? On a chip using 2W?
- kinnth 4mo ago
  I think we're also ignoring a potential innovative move in how models work. If someone could splinter or fragment the models into more specific tasks i.e "spellchecker AI" and get these working as well as Sonnet 4.6-4.8 on those tasks on a personal laptop. You then question the $100 a month fee. Bear in mind these laptops are likely to be $5000 or so because of the memory, HDD and M7 chip they likely need. It feels to me like the beginning of the inflection point but software updates not hardware updates will be the accelerant.
  - marci 4mo ago
    "That’s where EMO comes in. We show that EMO – a 1B-active, 14B-total-parameter (8-expert active, 128-expert total) MoE trained on 1 trillion tokens – supports selective expert use: for a given task or domain, we can use only a small subset of experts (just 12.5% of total experts) while retaining near full-model performance." https://allenai.org/blog/emo https://allenai.org/blog/emo

overfeed 4mo ago

> If it can do 90% of the tasks the big boys do, at 50% speed I want to live in this world too, but these numbers, as of today, are very aspirational and far removed from reality. I'm no tokenmaxxer; I find my modest local setup useful, I also know the limitations, it's slow and it sucks (relatively) at high-level and/or long-context planning, compared to frontier models. Only a minority of my prompts are max-effort - its not all I do, but, it also means frontier labs aren't dying any time soon

mikestorrent 4mo ago
Consider also that right now LLMs run slowly enough you can watch them think. I've seen a demo of an LLM running at an absurdly high speed and it reminds me of when I moved from a 2400 baud modem to a 14.4 - BBS screens that I could watch draw were all of a sudden nigh-interactive. Faster-than-realtime video generation is also coming, and will also continue to require huge hardware for a long while yet. I love local models - I have a machine at home that runs a few for me and it's a lot of fun - but for the time being they are not super trustworthy on tool calls and staying on script. Another year or so might change all that!
- ChickeNES 4mo ago
  What does your local setup look like?
  - deleted 4mo ago
    [deleted]
  - mikestorrent 4mo ago
    1650 watts of liquid cooled Risc-V
- KoolKat23 4mo ago
  If anyone wishes to see the future. A fast LLM is quite eye-opening. I think chatjimmy uses Talaas' chips where models are hardcoded into the silicon. https://chatjimmy.ai/ https://chatjimmy.ai/

SanjayMehta 4mo ago

This is what makes sense for me as well. All I need a local model is for playing with simple graphics: no gradients, at most ten colours which I can push through VTracer to get an SVG. Draw Things does the job, usually in 120 seconds or less. Sometimes, I need a quick throwaway bit of python. That can take 30 minutes of my time.