5 ms·

> Though, I've been saying for a while that the local AI inflectiom point is the death knell for these frontier labs. "Death knell" is a touch hyperbolic. Hard

> Though, I've been saying for a while that the local AI inflectiom point is the death knell for these frontier labs. "Death knell" is a touch hyperbolic. Hardware that can only run quantized models that take up GBs in VRAM falls short of even an A100 (by almost an order of magnitude[0]), which in turn falls short of what an 8xH100 cluster can do (also by another order of magnitude[0]). I'm an avid believer in local LLMs, but I cannot deceive myself - data center accelerators will win on power dissipation numbers alone[1], even when giving generous allowances for higher efficiency on Apple chips - and assuming the Apple-efficiency advantage persists on the same TSMC process node. 0. Based on my unscientific fine-tuning training experiments across local and rented GPUs. YMMV for inference. 1. Unless Apple surprises everyone and brings back the XServe with M7, if not, then laptop and desktop for factors simply can't dump heat fast enough to compete head-to-head, and will be designed for lower input wattage.

kennywinker 4mo ago
Doesn’t need to be a winner head to head. If it can do 90% of the tasks the big boys do, at 50% speed, for virtually no extra overhead cost save for the power consumed by a prompt - that’s gonna work for a lot of people. And that’s also basically where we’re at today. Qwen3.6 35b running quantized on 10 year old hardware solves basically all of my uses cases for agents except for coding. The frontier models are faster, and better at coding, but not so much that i’ll pay $200/month for them.
- drnick1 4mo ago
  Consider this. One of the smallest Qwen models (4B parameters) powers my home automation voice assistant, and runs on CPU alone at >20 tok/s. It is enough for that use case, and could be made even better/faster with a modest GPU. It isn't as smart as some cloud-connected thingamajig, but I would never allow a literal Google or Amazon bug in my home. Huge SOTA models aren't relevant everywhere. Most people use LLMs for rather trivial tasks such as finding typos or drafting text.
  - dainiusse 4mo ago
    Curious, what exactly does it do for you? I has bad luck with these small models to do anything useful tbh.
    - drnick1 4mo ago
      It's a voice assistant that can respond to commands such as "turn on the light," or explain things (within the abilities of the small model).
  - marci 4mo ago
    But with Apple's AFM 3 architecture, we might end up with huge SOTA adjacent on devices with limited RAM. They use a technique where you only load between 1B and 4B of a 20B dense model for an entire prompt run, not token by token like a MoE, and use mostly the low power ANE instead of GPU cores. Now, imagine if/when they scale up to 100B or more? On a chip using 2W?
    - kinnth 4mo ago
      I think we're also ignoring a potential innovative move in how models work. If someone could splinter or fragment the models into more specific tasks i.e "spellchecker AI" and get these working as well as Sonnet 4.6-4.8 on those tasks on a personal laptop. You then question the $100 a month fee. Bear in mind these laptops are likely to be $5000 or so because of the memory, HDD and M7 chip they likely need. It feels to me like the beginning of the inflection point but software updates not hardware updates will be the accelerant.
      - marci 4mo ago
        "That’s where EMO comes in. We show that EMO – a 1B-active, 14B-total-parameter (8-expert active, 128-expert total) MoE trained on 1 trillion tokens – supports selective expert use: for a given task or domain, we can use only a small subset of experts (just 12.5% of total experts) while retaining near full-model performance." https://allenai.org/blog/emo https://allenai.org/blog/emo
- overfeed 4mo ago
  > If it can do 90% of the tasks the big boys do, at 50% speed I want to live in this world too, but these numbers, as of today, are very aspirational and far removed from reality. I'm no tokenmaxxer; I find my modest local setup useful, I also know the limitations, it's slow and it sucks (relatively) at high-level and/or long-context planning, compared to frontier models. Only a minority of my prompts are max-effort - its not all I do, but, it also means frontier labs aren't dying any time soon
  - mikestorrent 4mo ago
    Consider also that right now LLMs run slowly enough you can watch them think. I've seen a demo of an LLM running at an absurdly high speed and it reminds me of when I moved from a 2400 baud modem to a 14.4 - BBS screens that I could watch draw were all of a sudden nigh-interactive. Faster-than-realtime video generation is also coming, and will also continue to require huge hardware for a long while yet. I love local models - I have a machine at home that runs a few for me and it's a lot of fun - but for the time being they are not super trustworthy on tool calls and staying on script. Another year or so might change all that!
    - ChickeNES 4mo ago
      What does your local setup look like?
      - deleted 4mo ago
        [deleted]
      - mikestorrent 4mo ago
        1650 watts of liquid cooled Risc-V
    - KoolKat23 4mo ago
      If anyone wishes to see the future. A fast LLM is quite eye-opening. I think chatjimmy uses Talaas' chips where models are hardcoded into the silicon. https://chatjimmy.ai/ https://chatjimmy.ai/
- SanjayMehta 4mo ago
  This is what makes sense for me as well. All I need a local model is for playing with simple graphics: no gradients, at most ten colours which I can push through VTracer to get an SVG. Draw Things does the job, usually in 120 seconds or less. Sometimes, I need a quick throwaway bit of python. That can take 30 minutes of my time.
rubyn00bie 4mo ago
We'll likely see a transformation in how frontier models are trained as a result of a push towards local inference. While it seems unlikely now, given current pricing for RAM, in 10-15 years it's not unthinkable to assume we could see individual machines with 10-12TB (and well beyond that) of RAM which are accessible to the GPU. Min/max system RAM increased a LOT from 2010-2025 and largely because it was cheap. Once the hyperscalers aren't generating revenue for the RAM manufacturers, I wouldn't be surprised to see a massive push towards consumers in order to maintain gross profit. Not to mention new players who enter the market because the margins are measurably absurd right now. At some point there will be diminishing returns towards the "just throw more RAM at it" approach the current frontier models are taking. Commoditization is just as inevitable as it ever was... and in doing so will enable actual leaps of what AI/ML is capable of. That's not to say there won't be a place for 99.999999% accurate vs 99.99999% but those cases will be limited and likely prime to disruption based on real innovation vs access to capital.
- Culonavirus 4mo ago
  The 1080ti is out there for almost 10 years now. It has 11GB of VRAM. A 5090 has 32GB. SOCs with unified memory have shifted this a bit forward, but they're also expensive as shit. 10TB ram in a consumer device is simply not happening in the next 10 years.
  - rjzzleep 4mo ago
    Half a year ago you could get a AI max 395+ with 128GB ram in mobile form factor for ~$2200. The same thing costs $3700. Same SoC, same memory.
    - rbanffy 4mo ago
      10TB is about 80 times that, 200K in today’s money. A lot of capacity is coming online in the next 5 years and it’s reasonable to think we can get there with better process and stacking (the latter does little for pricing, but enables shorter latencies).
deleted 4mo ago
[deleted]
bitwize 4mo ago
Indeed. Local models becoming available and halfway decent don't obviate the laws of scale. And because there's no ceiling to what scaling more will buy you in terms of capability, there's no reason not to scale more, there's no incentive for billionaires not to grab all the fab capacity they can. Enjoy paying $1000 or more for a little 4 GiB cloud terminal that connects you to all your online accounts where all your actual work gets done. This is the future.
- colechristensen 4mo ago
  >there's no ceiling to what scaling more will buy you in terms of capability This is highly doubtful. Rule of thumb: everything people think is exponential is actually an S curve.
  - bitwize 4mo ago
    There's a limit that won't be breached without a fundamental breakthrough in physics of computation, but we're not there yet by a long shot. You can train bigger models, faster, and infer with them faster and more precisely, by throwing more compute at the problem for the foreseeable.
    - rbanffy 4mo ago
      At some point, and I can already see it, they’ll be better than us at writing code. We are still in the loop to coerce them into architecting well, but that’s nothing magical. What’s frontier now is prosumer in a couple years and commonplace in a couple more.
  - nok22kon 4mo ago
    better rule: exponentials are overlapping S-curves
lumost 4mo ago
The big question for local LLMs is whether there is a 100 tok/s model which requires less than 16 GB of memory and is competitive on most tasks with the cloud models. There is some signal that this is possible through both hardware innovation and training/data improvements. Cloud models have their own constraints - I can’t have opus4.8 spend 4 hours on a deep research question I had in the shower without spending money. I can’t do real time video game upscaling and graphics work in the cloud period. A laptop is about an order of magnitude cheaper than a cloud server thanks to economies of scale, uptime requirements, and other factors.
- nok22kon 4mo ago
  if you do the electricity math you'll see that you pay more on local models while getting less (local is more heavily quantized) compared with OpenRouter. I'm not talking local Gemma/Qwen vs cloud Opus, but against OpenRouter same Gemma/Qwen there are reasons to run local - privacy, availability, but cost is not one of them
  - manarth 4mo ago
    That's assuming consumption pricing remains as-is. There has been a lot of market-subsidy in AI which is starting to fade away: e.g. the copilot quotas/pricing. When VC switches from investing to wanting a return, the price equation is likely to change.
    - nok22kon 4mo ago
      There is no subsidy on most OpenRouter providers, they are profitable today. You buy a big GPU, you serve LLMs, you print money.
      - brianwawok 4mo ago
        And if you skip open router and go direct you save another 5%
  - JumpCrisscross 4mo ago
rjzzleep 4mo ago
Is it hyperbolic though? One of the best things about the compute and memory shortage is that people are going to insane lengths to optimize things to run on lower memory / lower compute devices. If we keep this up for a while and then ramp up memory and local compute production, that AI inflection point may actually come. Of course, these are a lot of ifs.
- spiderfarmer 4mo ago
  If we advance just 2x in hardware plus 2x in software, all coding can be done on local hardware imho.
  - rbanffy 4mo ago
    That’s about 4 years in hardware cadence alone. There is a lot of room to improve memory bandwidth, and performance is a given with every process node. IBM has shown yesterday they can do limited runs on 0.7nm (density equivalent).
wisty 4mo ago
I'm not paying for a super computer to do my taxes if a cheap pc can do it for free. So yeah, commercially it might be a death knell. Yes there's still a market for super computers, but would your rather own Apple or Cray?
- rbanffy 4mo ago
  > would your rather own Apple or Cray? I would consider an HPE tower server with a processor on the same league as an M6 or M7 under the Cray brand.
dgellow 4mo ago
The thing is, with the level of hard investment AI vendors have, even a small reduction of their addressable market is significant. They aren’t profitable, and inference is getting commoditized fast, so even if they eventually become profitable (not via financial engineering) they won’t be able to have good margin. The pressure of both open models AND local models is pretty bad imho
everfrustrated 4mo ago
The established AI players have no financial interest to make LLM available locally. They aren't hardware companies and if running LLM requires paying them to host the models as well then they can naturally capture more of the value chain = more revenue. Apple is the only player here where it would play into their natural hardware incentive to get you to pay more for better hardware. It would make sense for them to find a way to run LLM locally (eg, newer architectures that others here have pointed out). Interesting times.
interludead 4mo ago
I think this is right but it also depends on what "compete" means
AnthonyMouse 4mo ago
> Hardware that can only run quantized models that take up GBs in VRAM That's the today hardware. Now suppose Apple goes to any of Samsung/Micron/Hynix and says "we'll pay you the entire cost of building another DRAM fab and in exchange we want its entire output" and then releases M7 devices with enough memory and compute to run bigger models. > Unless Apple surprises everyone and brings back the XServe with M7, if not, then laptop and desktop for factors simply can't dump heat fast enough to compete head-to-head, and will be designed for lower input wattage. Laptops maybe. Desktops can dissipate more heat than the amount of electricity you can draw from a typical household wall outlet.
- overfeed 4mo ago
  > Now suppose Apple goes to any of Samsung/Micron/Hynix and says "we'll pay you the entire cost of building another DRAM fab and in exchange we want its entire output" It's revealing that they aren't doing this: no one wants to fund that gamble on the state of AI demand 12-18 months out, but ate happy to capitalize on their current product lines/capacity. > Desktops can dissipate more heat than the amount of electricity you can draw from a typical household wall outlet 100% agree, but the data center power and cooling infra are not limited by home wiring, and go way beyond what a wall outlet can safely provide (1,440W max on a typical 15A circuit at 120V). A single H100 maxes out at 700W
  - AnthonyMouse 3mo ago
    > It's revealing that they aren't doing this They are doing this. All three of the DRAM manufacturers are actively expanding capacity. It's just not an instantaneous process. > 100% agree, but the data center power and cooling infra are not limited by home wiring, and go way beyond what a wall outlet can safely provide (1,440W max on a typical 15A circuit at 120V). A single H100 maxes out at 700W A desktop with the performance and the 160GB of VRAM of two H100s would sell like hotcakes. The power consumption is also largely proportional to the amount of compute rather than memory, and can be significantly improved by using lower clock speeds and then either making up for it with wider silicon or just not caring that it can run the same model but is e.g. 25% slower than hardware that uses twice as much electricity.

kennywinker 4mo ago

Doesn’t need to be a winner head to head. If it can do 90% of the tasks the big boys do, at 50% speed, for virtually no extra overhead cost save for the power consumed by a prompt - that’s gonna work for a lot of people. And that’s also basically where we’re at today. Qwen3.6 35b running quantized on 10 year old hardware solves basically all of my uses cases for agents except for coding. The frontier models are faster, and better at coding, but not so much that i’ll pay $200/month for them.

drnick1 4mo ago
Consider this. One of the smallest Qwen models (4B parameters) powers my home automation voice assistant, and runs on CPU alone at >20 tok/s. It is enough for that use case, and could be made even better/faster with a modest GPU. It isn't as smart as some cloud-connected thingamajig, but I would never allow a literal Google or Amazon bug in my home. Huge SOTA models aren't relevant everywhere. Most people use LLMs for rather trivial tasks such as finding typos or drafting text.
- dainiusse 4mo ago
  Curious, what exactly does it do for you? I has bad luck with these small models to do anything useful tbh.
  - drnick1 4mo ago
    It's a voice assistant that can respond to commands such as "turn on the light," or explain things (within the abilities of the small model).
- marci 4mo ago
  But with Apple's AFM 3 architecture, we might end up with huge SOTA adjacent on devices with limited RAM. They use a technique where you only load between 1B and 4B of a 20B dense model for an entire prompt run, not token by token like a MoE, and use mostly the low power ANE instead of GPU cores. Now, imagine if/when they scale up to 100B or more? On a chip using 2W?
  - kinnth 4mo ago
    I think we're also ignoring a potential innovative move in how models work. If someone could splinter or fragment the models into more specific tasks i.e "spellchecker AI" and get these working as well as Sonnet 4.6-4.8 on those tasks on a personal laptop. You then question the $100 a month fee. Bear in mind these laptops are likely to be $5000 or so because of the memory, HDD and M7 chip they likely need. It feels to me like the beginning of the inflection point but software updates not hardware updates will be the accelerant.
    - marci 4mo ago
      "That’s where EMO comes in. We show that EMO – a 1B-active, 14B-total-parameter (8-expert active, 128-expert total) MoE trained on 1 trillion tokens – supports selective expert use: for a given task or domain, we can use only a small subset of experts (just 12.5% of total experts) while retaining near full-model performance." https://allenai.org/blog/emo https://allenai.org/blog/emo
overfeed 4mo ago
> If it can do 90% of the tasks the big boys do, at 50% speed I want to live in this world too, but these numbers, as of today, are very aspirational and far removed from reality. I'm no tokenmaxxer; I find my modest local setup useful, I also know the limitations, it's slow and it sucks (relatively) at high-level and/or long-context planning, compared to frontier models. Only a minority of my prompts are max-effort - its not all I do, but, it also means frontier labs aren't dying any time soon
- mikestorrent 4mo ago
  Consider also that right now LLMs run slowly enough you can watch them think. I've seen a demo of an LLM running at an absurdly high speed and it reminds me of when I moved from a 2400 baud modem to a 14.4 - BBS screens that I could watch draw were all of a sudden nigh-interactive. Faster-than-realtime video generation is also coming, and will also continue to require huge hardware for a long while yet. I love local models - I have a machine at home that runs a few for me and it's a lot of fun - but for the time being they are not super trustworthy on tool calls and staying on script. Another year or so might change all that!
  - ChickeNES 4mo ago
    What does your local setup look like?
    - deleted 4mo ago
      [deleted]
    - mikestorrent 4mo ago
      1650 watts of liquid cooled Risc-V
  - KoolKat23 4mo ago
    If anyone wishes to see the future. A fast LLM is quite eye-opening. I think chatjimmy uses Talaas' chips where models are hardcoded into the silicon. https://chatjimmy.ai/ https://chatjimmy.ai/
SanjayMehta 4mo ago
This is what makes sense for me as well. All I need a local model is for playing with simple graphics: no gradients, at most ten colours which I can push through VTracer to get an SVG. Draw Things does the job, usually in 120 seconds or less. Sometimes, I need a quick throwaway bit of python. That can take 30 minutes of my time.

rubyn00bie 4mo ago

We'll likely see a transformation in how frontier models are trained as a result of a push towards local inference. While it seems unlikely now, given current pricing for RAM, in 10-15 years it's not unthinkable to assume we could see individual machines with 10-12TB (and well beyond that) of RAM which are accessible to the GPU. Min/max system RAM increased a LOT from 2010-2025 and largely because it was cheap. Once the hyperscalers aren't generating revenue for the RAM manufacturers, I wouldn't be surprised to see a massive push towards consumers in order to maintain gross profit. Not to mention new players who enter the market because the margins are measurably absurd right now. At some point there will be diminishing returns towards the "just throw more RAM at it" approach the current frontier models are taking. Commoditization is just as inevitable as it ever was... and in doing so will enable actual leaps of what AI/ML is capable of. That's not to say there won't be a place for 99.999999% accurate vs 99.99999% but those cases will be limited and likely prime to disruption based on real innovation vs access to capital.

Culonavirus 4mo ago
The 1080ti is out there for almost 10 years now. It has 11GB of VRAM. A 5090 has 32GB. SOCs with unified memory have shifted this a bit forward, but they're also expensive as shit. 10TB ram in a consumer device is simply not happening in the next 10 years.
- rjzzleep 4mo ago
  Half a year ago you could get a AI max 395+ with 128GB ram in mobile form factor for ~$2200. The same thing costs $3700. Same SoC, same memory.
  - rbanffy 4mo ago
    10TB is about 80 times that, 200K in today’s money. A lot of capacity is coming online in the next 5 years and it’s reasonable to think we can get there with better process and stacking (the latter does little for pricing, but enables shorter latencies).

deleted 4mo ago

[deleted]

bitwize 4mo ago

Indeed. Local models becoming available and halfway decent don't obviate the laws of scale. And because there's no ceiling to what scaling more will buy you in terms of capability, there's no reason not to scale more, there's no incentive for billionaires not to grab all the fab capacity they can. Enjoy paying $1000 or more for a little 4 GiB cloud terminal that connects you to all your online accounts where all your actual work gets done. This is the future.

colechristensen 4mo ago
>there's no ceiling to what scaling more will buy you in terms of capability This is highly doubtful. Rule of thumb: everything people think is exponential is actually an S curve.
- bitwize 4mo ago
  There's a limit that won't be breached without a fundamental breakthrough in physics of computation, but we're not there yet by a long shot. You can train bigger models, faster, and infer with them faster and more precisely, by throwing more compute at the problem for the foreseeable.
  - rbanffy 4mo ago
    At some point, and I can already see it, they’ll be better than us at writing code. We are still in the loop to coerce them into architecting well, but that’s nothing magical. What’s frontier now is prosumer in a couple years and commonplace in a couple more.
- nok22kon 4mo ago
  better rule: exponentials are overlapping S-curves

lumost 4mo ago

The big question for local LLMs is whether there is a 100 tok/s model which requires less than 16 GB of memory and is competitive on most tasks with the cloud models. There is some signal that this is possible through both hardware innovation and training/data improvements. Cloud models have their own constraints - I can’t have opus4.8 spend 4 hours on a deep research question I had in the shower without spending money. I can’t do real time video game upscaling and graphics work in the cloud period. A laptop is about an order of magnitude cheaper than a cloud server thanks to economies of scale, uptime requirements, and other factors.

nok22kon 4mo ago
if you do the electricity math you'll see that you pay more on local models while getting less (local is more heavily quantized) compared with OpenRouter. I'm not talking local Gemma/Qwen vs cloud Opus, but against OpenRouter same Gemma/Qwen there are reasons to run local - privacy, availability, but cost is not one of them
- manarth 4mo ago
  That's assuming consumption pricing remains as-is. There has been a lot of market-subsidy in AI which is starting to fade away: e.g. the copilot quotas/pricing. When VC switches from investing to wanting a return, the price equation is likely to change.
  - nok22kon 4mo ago
    There is no subsidy on most OpenRouter providers, they are profitable today. You buy a big GPU, you serve LLMs, you print money.
    - brianwawok 4mo ago
      And if you skip open router and go direct you save another 5%
- JumpCrisscross 4mo ago

rjzzleep 4mo ago

Is it hyperbolic though? One of the best things about the compute and memory shortage is that people are going to insane lengths to optimize things to run on lower memory / lower compute devices. If we keep this up for a while and then ramp up memory and local compute production, that AI inflection point may actually come. Of course, these are a lot of ifs.

spiderfarmer 4mo ago
If we advance just 2x in hardware plus 2x in software, all coding can be done on local hardware imho.
- rbanffy 4mo ago
  That’s about 4 years in hardware cadence alone. There is a lot of room to improve memory bandwidth, and performance is a given with every process node. IBM has shown yesterday they can do limited runs on 0.7nm (density equivalent).

wisty 4mo ago

I'm not paying for a super computer to do my taxes if a cheap pc can do it for free. So yeah, commercially it might be a death knell. Yes there's still a market for super computers, but would your rather own Apple or Cray?

rbanffy 4mo ago
> would your rather own Apple or Cray? I would consider an HPE tower server with a processor on the same league as an M6 or M7 under the Cray brand.

dgellow 4mo ago

The thing is, with the level of hard investment AI vendors have, even a small reduction of their addressable market is significant. They aren’t profitable, and inference is getting commoditized fast, so even if they eventually become profitable (not via financial engineering) they won’t be able to have good margin. The pressure of both open models AND local models is pretty bad imho

everfrustrated 4mo ago

The established AI players have no financial interest to make LLM available locally. They aren't hardware companies and if running LLM requires paying them to host the models as well then they can naturally capture more of the value chain = more revenue. Apple is the only player here where it would play into their natural hardware incentive to get you to pay more for better hardware. It would make sense for them to find a way to run LLM locally (eg, newer architectures that others here have pointed out). Interesting times.

interludead 4mo ago

I think this is right but it also depends on what "compete" means

AnthonyMouse 4mo ago

> Hardware that can only run quantized models that take up GBs in VRAM That's the today hardware. Now suppose Apple goes to any of Samsung/Micron/Hynix and says "we'll pay you the entire cost of building another DRAM fab and in exchange we want its entire output" and then releases M7 devices with enough memory and compute to run bigger models. > Unless Apple surprises everyone and brings back the XServe with M7, if not, then laptop and desktop for factors simply can't dump heat fast enough to compete head-to-head, and will be designed for lower input wattage. Laptops maybe. Desktops can dissipate more heat than the amount of electricity you can draw from a typical household wall outlet.

overfeed 4mo ago
> Now suppose Apple goes to any of Samsung/Micron/Hynix and says "we'll pay you the entire cost of building another DRAM fab and in exchange we want its entire output" It's revealing that they aren't doing this: no one wants to fund that gamble on the state of AI demand 12-18 months out, but ate happy to capitalize on their current product lines/capacity. > Desktops can dissipate more heat than the amount of electricity you can draw from a typical household wall outlet 100% agree, but the data center power and cooling infra are not limited by home wiring, and go way beyond what a wall outlet can safely provide (1,440W max on a typical 15A circuit at 120V). A single H100 maxes out at 700W
- AnthonyMouse 3mo ago
  > It's revealing that they aren't doing this They are doing this. All three of the DRAM manufacturers are actively expanding capacity. It's just not an instantaneous process. > 100% agree, but the data center power and cooling infra are not limited by home wiring, and go way beyond what a wall outlet can safely provide (1,440W max on a typical 15A circuit at 120V). A single H100 maxes out at 700W A desktop with the performance and the 160GB of VRAM of two H100s would sell like hotcakes. The power consumption is also largely proportional to the amount of compute rather than memory, and can be significantly improved by using lower clock speeds and then either making up for it with wider silicon or just not caring that it can run the same model but is e.g. 25% slower than hardware that uses twice as much electricity.