4 ms·
On non-Mac laptops with integrated graphics it's the same, up to 50% of the RAM can be allocated to the GPU (both AMD and Intel work the same way). Not sure if
by dismalaf 2y ago
On non-Mac laptops with integrated graphics it's the same, up to 50% of the RAM can be allocated to the GPU (both AMD and Intel work the same way). Not sure if Windows allows it but on Linux you can definitely see it. So let's say you have a 96gb ram Framework laptop, that's 48gb for the GPU...
I think the main impediment is Intel and SYCL not working on all their cards... AMD is good to go, I think.
- buildbot 2y agoThe bandwidth is (potentially, depending on the Mac in question) much lower on the framework, which will limit the decode tokens/s
- dismalaf 2y agoThat's fair but still better than running on the CPU, and still enough to hold the model in VRAM.
- freedomben 2y agoInteresting! I'll have to give this a try. On my framework 16 with the GPU (running Fedora), ollama runs super well and blazing fast, though only smaller models of course. I have 128 GB of system memory though so would love to try it a bigger model in the integrated GPU. Btw, this is my favorite laptop of all time. Highly recommend it.