3 ms·

I’ve been running LLMs on my laptop (M3 Max 64GB) for a year now and I think they are ready, especially with how good mid sized models are getting. I’m pretty s

I’ve been running LLMs on my laptop (M3 Max 64GB) for a year now and I think they are ready, especially with how good mid sized models are getting. I’m pretty sure unified memory and energy efficient GPUs will be more than just a thing on Apple laptops in the next few years.

allovertheworld 11mo ago
Only because of Apples unified memory architecture. The groundwork is there, we just need memory to be cheaper so we can fit 512+GB now ;)
- seanmcdirmid 11mo ago
  Memory prices will rise short term and generally fall long term, even with the current supply hiccup the answer is to just build out more capacity (which will happen if there is healthy competition). I meant, I expect the other mobile chip providers to adopt unified architecture and beefy GPU cores on chip and lots of bandwidth to connect it to memory (at the max or ultra level, at least), I think AMD is already doing UM at least?
  - spwa4 11mo ago
    > Memory prices will rise short term and generally fall long term, even with the current supply hiccup the answer is to just build out more capacity (which will happen if there is healthy competition) Don't worry! Sam Altman is on it. Making sure there never is healthy competition that is. https://www.mooreslawisdead.com/post/sam-altman-s-dirty-dram-deal https://www.mooreslawisdead.com/post/sam-altman-s-dirty-dram...
    - seanmcdirmid 11mo ago
      We’ve been through multiple cycles of scarcity/surplus DRAM cycles in the last couple of decades. Why do we think it will be different now?
      - re-thc 11mo ago
        > Why do we think it will be different now? Margins. AI usage can pay a lot more. Even if they sell less than can still be more profitable. In the past there wasn’t a high margin usage. Servers didn’t charge such a high premium.
        zozbot234 11mo ago
        High margins are exactly what should create a strong incentive to build more capacity. But that dynamic has been tamped down so far because we're all scared of a possible AI bubble that might pop at any moment.
        seanmcdirmid 11mo ago
        Do you not think that some DRAM producer isn't going to see the high margins as a signal to create more capacity to get ahead of the other DRAM producers? This is how it always has worked before, but somehow it is different this time?
        re-thc 11mo ago
        > Do you not think that some DRAM producer isn't going to see the high margins as a signal to create more capacity to get ahead of the other DRAM producers? They took the bite during COVID and failed, so there's still fear from over supply.
        seanmcdirmid 11mo ago
        It only works if they collude on keeping supply steady. If anyone gets greedy for a bigger share of the AI pie, then it implodes quickly. Not all DRAM is made in South Korea so some nationalism will muddy the waters as well.
- zmmmmm 11mo ago
  There's not in the end all that much point having more memory than you can compute on in a reasonable time. So I think probably the useful amount tops out in the 128GB range where you can still run a 70b model and get a useful token rate out of it.
noman-land 11mo ago
You doing code completion and agentic stuff successfully with local models? Got any tips? I've been out of the game for [checks watch] a few months and am behind on the latest. Is Cline the move?
- seanmcdirmid 11mo ago
  I haven't bothered doing code completion locally yet, though its something I want to try with the QWEN model. I'm mostly using it to generate/fix code CLI style.
  - noman-land 11mo ago
    I had some pretty decent but very non-state-of-the-art success with it even cobbled together with LM Studio and VSCode plugins. I'm excited to keep trying it over the next months and years.

allovertheworld 11mo ago

Only because of Apples unified memory architecture. The groundwork is there, we just need memory to be cheaper so we can fit 512+GB now ;)

seanmcdirmid 11mo ago
Memory prices will rise short term and generally fall long term, even with the current supply hiccup the answer is to just build out more capacity (which will happen if there is healthy competition). I meant, I expect the other mobile chip providers to adopt unified architecture and beefy GPU cores on chip and lots of bandwidth to connect it to memory (at the max or ultra level, at least), I think AMD is already doing UM at least?
- spwa4 11mo ago
  > Memory prices will rise short term and generally fall long term, even with the current supply hiccup the answer is to just build out more capacity (which will happen if there is healthy competition) Don't worry! Sam Altman is on it. Making sure there never is healthy competition that is. https://www.mooreslawisdead.com/post/sam-altman-s-dirty-dram-deal https://www.mooreslawisdead.com/post/sam-altman-s-dirty-dram...
  - seanmcdirmid 11mo ago
    We’ve been through multiple cycles of scarcity/surplus DRAM cycles in the last couple of decades. Why do we think it will be different now?
    - re-thc 11mo ago
      > Why do we think it will be different now? Margins. AI usage can pay a lot more. Even if they sell less than can still be more profitable. In the past there wasn’t a high margin usage. Servers didn’t charge such a high premium.
      - zozbot234 11mo ago
        High margins are exactly what should create a strong incentive to build more capacity. But that dynamic has been tamped down so far because we're all scared of a possible AI bubble that might pop at any moment.
      - seanmcdirmid 11mo ago
        Do you not think that some DRAM producer isn't going to see the high margins as a signal to create more capacity to get ahead of the other DRAM producers? This is how it always has worked before, but somehow it is different this time?
        re-thc 11mo ago
        > Do you not think that some DRAM producer isn't going to see the high margins as a signal to create more capacity to get ahead of the other DRAM producers? They took the bite during COVID and failed, so there's still fear from over supply.
        seanmcdirmid 11mo ago
        It only works if they collude on keeping supply steady. If anyone gets greedy for a bigger share of the AI pie, then it implodes quickly. Not all DRAM is made in South Korea so some nationalism will muddy the waters as well.
zmmmmm 11mo ago
There's not in the end all that much point having more memory than you can compute on in a reasonable time. So I think probably the useful amount tops out in the 128GB range where you can still run a 70b model and get a useful token rate out of it.

noman-land 11mo ago

You doing code completion and agentic stuff successfully with local models? Got any tips? I've been out of the game for [checks watch] a few months and am behind on the latest. Is Cline the move?

seanmcdirmid 11mo ago
I haven't bothered doing code completion locally yet, though its something I want to try with the QWEN model. I'm mostly using it to generate/fix code CLI style.
- noman-land 11mo ago
  I had some pretty decent but very non-state-of-the-art success with it even cobbled together with LM Studio and VSCode plugins. I'm excited to keep trying it over the next months and years.