4 ms·

Great post as expected (I'm a really big fan of Simon), but quick note lest others get prematurely excited, this is on an Apple laptop. Personally I run these

Great post as expected (I'm a really big fan of Simon), but quick note lest others get prematurely excited, this is on an Apple laptop. Personally I run these models because I care about openness, so an Apple laptop is the last I'd buy. Good post though and worth a read as there's plenty of good stuff here for a non-Apple user.

simonw 2y ago
Yeah, I don't have a great perspective right now on what this looks like in non-Apple world. The problem is always RAM. On Apple Silicon RAM is shared between CPU and GPU, which means a 64GB of RAM machine can run really big models with GPU acceleration. On Windows/Linux machines you usually need to consider the VRAM available to your GPU. I haven't got a great feel for what kind of VRAM numbers are common for different consumer setups there, so I don't tend to write much about those. Of course llama.cpp will happily run models on CPU as well, but again I don't have a great feel for how usable that is.
- dismalaf 2y ago
  On non-Mac laptops with integrated graphics it's the same, up to 50% of the RAM can be allocated to the GPU (both AMD and Intel work the same way). Not sure if Windows allows it but on Linux you can definitely see it. So let's say you have a 96gb ram Framework laptop, that's 48gb for the GPU... I think the main impediment is Intel and SYCL not working on all their cards... AMD is good to go, I think.
  - buildbot 2y ago
    The bandwidth is (potentially, depending on the Mac in question) much lower on the framework, which will limit the decode tokens/s
    - dismalaf 2y ago
      That's fair but still better than running on the CPU, and still enough to hold the model in VRAM.
  - freedomben 2y ago
    Interesting! I'll have to give this a try. On my framework 16 with the GPU (running Fedora), ollama runs super well and blazing fast, though only smaller models of course. I have 128 GB of system memory though so would love to try it a bigger model in the integrated GPU. Btw, this is my favorite laptop of all time. Highly recommend it.
- transcriptase 2y ago
  In my experience with a 4070ti (12 gb vram), 7800x3d and 64gb ddr5 it will run… but seemingly at about 0.1-0.5 tokens per second with partial GPU offload. It’s painful, but the results are night and day compared to the models that will fit entirely in vram. Of course perhaps I’m doing something wrong so if others have advice it would be great to hear!
  - hnuser123456 2y ago
    Is it any faster with no GPU offload, given how little would fit in VRAM? Can you try giving a few prompts and getting a precise average token speed? Curious if it would be worth upgrading my RAM to at least run it on CPU. I have a 9800x3d/48gb ddr5/3090 and this model still being ~40GB when quantized is a challenge. Sure would be nice to have a 30B version that can quantize down to ~16GB. Or I guess I could upgrade my PSU and get another 3090 and use NVlink...
- Havoc 2y ago
  It’s not the vram number that’s the problem on windows side. It’s the throughput. For whatever reason they’re a good 5-10x lower. Even if it’s a large amount and gpu accessible if the throughput sucks that badly then 70B models aren’t usable What puzzles me is that the even the newer AMD APUs have this issue. The upcoming strix halo is rumoured to be around 140 ish which puts it at very bottom end of Apple devices
  - hnuser123456 2y ago
    Apple silicon RAM is basically all VRAM, with 150-400 GB/s bandwidth. Windows machines with dual channel DDR5 are only around 80 GB/s bandwidth. Nvidia cards are around 1 TB/s, but models with 40+GB of VRAM are more expensive than macs with equivalent RAM capacity. I would expect that an Nvidia card on Windows would be around twice as fast as apple silicon if you have a model with 40+GB, and from some quick research this seems to be true. edit: "Any Macbook 64GB can run it at about 6-7 Tok/s using Q4 quantization. 2x3090 can do it much faster at about 18 tok/s" https://www.reddit.com/r/LocalLLaMA/comments/1h9kci3/comment/m135clq/ https://www.reddit.com/r/LocalLLaMA/comments/1h9kci3/comment...
    - Havoc 2y ago
      > Apple silicon RAM is basically all VRAM The actual ram is lpddr5 - miles slower than what you’d find in even very old dedicated GPUs It’s as you say though - for some tasks like LLMs that doesn’t matter if you’ve got enough bandwidth to said slow ram…ends up being almost the same thing as having actual fast ram. It’s a neat trick & little bit frustrated that nobody is copying it. Cause using dirt cheap lpddr5 (about a dollar per gig) but paying Apple’s comical ram upgrade prices feels like a cruel joke.
      - hnuser123456 2y ago
        Ah, so do they use 2 channels on standard / 4 on pro / 8 on max? Rumors are there may be a X3D Threadripper in the works, would be nice to not have to choose between RAM bandwidth and IPC again.
diffeomorphism 2y ago
Not just "an" apple laptop but one with 64GB of ram. From a quick look at Apple's website the cheapest option among the current models is 4600€ for the 14 inch macbook pro. So, neat, I guess, but not convinced I would pay four macbook airs for that.
- philshem 2y ago
  The newest Mac Mini with M4 Pro and 64GB of ram costs ~ €2000
  - diffeomorphism 2y ago
    The title says "my laptop". A Mac mini is not a laptop.
    - deleted 2y ago
      [deleted]
- woleium 2y ago
  an m1 off evay can be had for less

simonw 2y ago

Yeah, I don't have a great perspective right now on what this looks like in non-Apple world. The problem is always RAM. On Apple Silicon RAM is shared between CPU and GPU, which means a 64GB of RAM machine can run really big models with GPU acceleration. On Windows/Linux machines you usually need to consider the VRAM available to your GPU. I haven't got a great feel for what kind of VRAM numbers are common for different consumer setups there, so I don't tend to write much about those. Of course llama.cpp will happily run models on CPU as well, but again I don't have a great feel for how usable that is.

dismalaf 2y ago
On non-Mac laptops with integrated graphics it's the same, up to 50% of the RAM can be allocated to the GPU (both AMD and Intel work the same way). Not sure if Windows allows it but on Linux you can definitely see it. So let's say you have a 96gb ram Framework laptop, that's 48gb for the GPU... I think the main impediment is Intel and SYCL not working on all their cards... AMD is good to go, I think.
- buildbot 2y ago
  The bandwidth is (potentially, depending on the Mac in question) much lower on the framework, which will limit the decode tokens/s
  - dismalaf 2y ago
    That's fair but still better than running on the CPU, and still enough to hold the model in VRAM.
- freedomben 2y ago
  Interesting! I'll have to give this a try. On my framework 16 with the GPU (running Fedora), ollama runs super well and blazing fast, though only smaller models of course. I have 128 GB of system memory though so would love to try it a bigger model in the integrated GPU. Btw, this is my favorite laptop of all time. Highly recommend it.
transcriptase 2y ago
In my experience with a 4070ti (12 gb vram), 7800x3d and 64gb ddr5 it will run… but seemingly at about 0.1-0.5 tokens per second with partial GPU offload. It’s painful, but the results are night and day compared to the models that will fit entirely in vram. Of course perhaps I’m doing something wrong so if others have advice it would be great to hear!
- hnuser123456 2y ago
  Is it any faster with no GPU offload, given how little would fit in VRAM? Can you try giving a few prompts and getting a precise average token speed? Curious if it would be worth upgrading my RAM to at least run it on CPU. I have a 9800x3d/48gb ddr5/3090 and this model still being ~40GB when quantized is a challenge. Sure would be nice to have a 30B version that can quantize down to ~16GB. Or I guess I could upgrade my PSU and get another 3090 and use NVlink...
Havoc 2y ago
It’s not the vram number that’s the problem on windows side. It’s the throughput. For whatever reason they’re a good 5-10x lower. Even if it’s a large amount and gpu accessible if the throughput sucks that badly then 70B models aren’t usable What puzzles me is that the even the newer AMD APUs have this issue. The upcoming strix halo is rumoured to be around 140 ish which puts it at very bottom end of Apple devices
- hnuser123456 2y ago
  Apple silicon RAM is basically all VRAM, with 150-400 GB/s bandwidth. Windows machines with dual channel DDR5 are only around 80 GB/s bandwidth. Nvidia cards are around 1 TB/s, but models with 40+GB of VRAM are more expensive than macs with equivalent RAM capacity. I would expect that an Nvidia card on Windows would be around twice as fast as apple silicon if you have a model with 40+GB, and from some quick research this seems to be true. edit: "Any Macbook 64GB can run it at about 6-7 Tok/s using Q4 quantization. 2x3090 can do it much faster at about 18 tok/s" https://www.reddit.com/r/LocalLLaMA/comments/1h9kci3/comment/m135clq/ https://www.reddit.com/r/LocalLLaMA/comments/1h9kci3/comment...
  - Havoc 2y ago
    > Apple silicon RAM is basically all VRAM The actual ram is lpddr5 - miles slower than what you’d find in even very old dedicated GPUs It’s as you say though - for some tasks like LLMs that doesn’t matter if you’ve got enough bandwidth to said slow ram…ends up being almost the same thing as having actual fast ram. It’s a neat trick & little bit frustrated that nobody is copying it. Cause using dirt cheap lpddr5 (about a dollar per gig) but paying Apple’s comical ram upgrade prices feels like a cruel joke.
    - hnuser123456 2y ago
      Ah, so do they use 2 channels on standard / 4 on pro / 8 on max? Rumors are there may be a X3D Threadripper in the works, would be nice to not have to choose between RAM bandwidth and IPC again.

diffeomorphism 2y ago

Not just "an" apple laptop but one with 64GB of ram. From a quick look at Apple's website the cheapest option among the current models is 4600€ for the 14 inch macbook pro. So, neat, I guess, but not convinced I would pay four macbook airs for that.

philshem 2y ago
The newest Mac Mini with M4 Pro and 64GB of ram costs ~ €2000
- diffeomorphism 2y ago
  The title says "my laptop". A Mac mini is not a laptop.
  - deleted 2y ago
    [deleted]
woleium 2y ago
an m1 off evay can be had for less