4 ms·

So I keep getting told "Macs don't use memory like Windows", "8GB is fine for everything on Mac" (with quotes from people who use it to Photoshop) and "the SSD

So I keep getting told "Macs don't use memory like Windows", "8GB is fine for everything on Mac" (with quotes from people who use it to Photoshop) and "the SSD is so fast if you use virtual memory anyway it's not noticeable"... Is 8GB suitable for LLM use like this, and has anybody actually used it?

RockRobotRock 3y ago
8 GB on a MacBook is kinda rough. I regret it.
jzelinskie 3y ago
Apple really shouldn't sell Macs with less than 16GiB of RAM. The first set of M1s got some bad press because so many folks were plagued by pop-ups saying they were out of memory -- from heavy browser usage not even intense workloads.
- sgarland 3y ago
  As someone with a base model M1 Air, I’ve never experienced this issue from normal workloads. The ONLY times I’ve exceeded memory capacity were doing expensive computations without chunking, and running minikube. I’m neither a Node dev nor an AI person, though, so that probably helps.
  - jsjohnst 3y ago
    Have a half dozen Google Docs open at once and you could force an 8gb machine to start swapping. Just saying.
    - sgarland 3y ago
      This is only slightly hyperbolic, after having checked it. Each GDoc tab (in Safari) took ~600 MiB. What caused the swap was the other browser window with two Gmail tabs, each of which took over 1 GiB, which when combined with some other apps I had open, broke it. I stand corrected. 8 GiB might be enough IFF you're not doing heavy multi-tasking during development, and you're not coding in JS or its offshoots. I will say the comments about "the SSD is so fast you won't notice" is true from an end-user perspective, in that I experienced no visible lag on any app. This may not be true with the smaller-capacity M2 models, since they have slower SSDs than the base M1s.
      - jsjohnst 3y ago
        > This is only slightly hyperbolic, after having checked it. Each GDoc tab (in Safari) took ~600 MiB. It very much depends on the size of the document and how long it’s been open. I’ve actually had Gdocs be over 3gb for a single document in Safari before.
- astrange 3y ago
  That's OS or app bugs. The way that dialog works isn't just how much RAM you have.
LeoPanthera 3y ago
LLMs are notoriously RAM-hungry. It’s not fair to judge what a MacBook comes with against what LLaMa2 needs. That is not normal use of a MacBook.
DrBenCarson 3y ago
8gb on a Mac is better than 8gb on Windows, but 16gb is a whole lot better
loxias 3y ago
I wonder the same thing, I have 64GB on a thinkpad from 3 years ago...
SparkyMcUnicorn 3y ago
8GB of memory is not currently suitable for LLM use. A 7B model @ Q4 needs roughly 6.5GB of memory. I have a 64GB M1. If my memory pressure is high and I start up a model that goes into swap, the machine becomes unbearably slow.
- brucethemoose2 3y ago
  More than that for the 8K+ context people are running now.
- tarruda 3y ago
  > A 7B model @ Q4 needs roughly 6.5GB of memory. Mistral 7B q4 seems to use ~ 5Gb VRAM A few months ago I could not run Llama 7B q4 on my laptop's rtx 3070 (8gb VRAM), but Mistral 7B it runs very well and still has approx 2gb VRAM left even with a big context. Not sure if this is an optimization on inference algorithms or if Mistral 7B is just more efficient
yieldcrv 3y ago
If you need a continuous block of addressable ram, there is nothing that a mac does differently and you will need more than 8gb if you need more than an 8gb block for a process
wryanzimmerman 3y ago
Macs manage memory and swap in a way that makes them feel much more responsive when you have a lot of common applications open at once (or something like a lot of browser tabs at once), but if you’re literally doing a single task that uses a given amount of ram, you need that much ram or you’ll slow way down.
Geezus_42 3y ago
I use an M1 Pro every day for work, and while it runs fantastically, I call bullshit on the 8GB claim.
brucethemoose2 3y ago
Before I got a desktop, I used to run my laptop (16GB RAM + a 6GB Nvidia GPU) headless to just barely squeeze the 33B models onto the GPU + RAM, and play with it on another device. ...I am not an OSX person, but if you can boot headless, it can squeeze 7B on there with full context

RockRobotRock 3y ago

8 GB on a MacBook is kinda rough. I regret it.

jzelinskie 3y ago

Apple really shouldn't sell Macs with less than 16GiB of RAM. The first set of M1s got some bad press because so many folks were plagued by pop-ups saying they were out of memory -- from heavy browser usage not even intense workloads.

sgarland 3y ago
As someone with a base model M1 Air, I’ve never experienced this issue from normal workloads. The ONLY times I’ve exceeded memory capacity were doing expensive computations without chunking, and running minikube. I’m neither a Node dev nor an AI person, though, so that probably helps.
- jsjohnst 3y ago
  Have a half dozen Google Docs open at once and you could force an 8gb machine to start swapping. Just saying.
  - sgarland 3y ago
    This is only slightly hyperbolic, after having checked it. Each GDoc tab (in Safari) took ~600 MiB. What caused the swap was the other browser window with two Gmail tabs, each of which took over 1 GiB, which when combined with some other apps I had open, broke it. I stand corrected. 8 GiB might be enough IFF you're not doing heavy multi-tasking during development, and you're not coding in JS or its offshoots. I will say the comments about "the SSD is so fast you won't notice" is true from an end-user perspective, in that I experienced no visible lag on any app. This may not be true with the smaller-capacity M2 models, since they have slower SSDs than the base M1s.
    - jsjohnst 3y ago
      > This is only slightly hyperbolic, after having checked it. Each GDoc tab (in Safari) took ~600 MiB. It very much depends on the size of the document and how long it’s been open. I’ve actually had Gdocs be over 3gb for a single document in Safari before.
astrange 3y ago
That's OS or app bugs. The way that dialog works isn't just how much RAM you have.

LeoPanthera 3y ago

LLMs are notoriously RAM-hungry. It’s not fair to judge what a MacBook comes with against what LLaMa2 needs. That is not normal use of a MacBook.

DrBenCarson 3y ago

8gb on a Mac is better than 8gb on Windows, but 16gb is a whole lot better

loxias 3y ago

I wonder the same thing, I have 64GB on a thinkpad from 3 years ago...

SparkyMcUnicorn 3y ago

8GB of memory is not currently suitable for LLM use. A 7B model @ Q4 needs roughly 6.5GB of memory. I have a 64GB M1. If my memory pressure is high and I start up a model that goes into swap, the machine becomes unbearably slow.

brucethemoose2 3y ago
More than that for the 8K+ context people are running now.
tarruda 3y ago
> A 7B model @ Q4 needs roughly 6.5GB of memory. Mistral 7B q4 seems to use ~ 5Gb VRAM A few months ago I could not run Llama 7B q4 on my laptop's rtx 3070 (8gb VRAM), but Mistral 7B it runs very well and still has approx 2gb VRAM left even with a big context. Not sure if this is an optimization on inference algorithms or if Mistral 7B is just more efficient

yieldcrv 3y ago

If you need a continuous block of addressable ram, there is nothing that a mac does differently and you will need more than 8gb if you need more than an 8gb block for a process

wryanzimmerman 3y ago

Macs manage memory and swap in a way that makes them feel much more responsive when you have a lot of common applications open at once (or something like a lot of browser tabs at once), but if you’re literally doing a single task that uses a given amount of ram, you need that much ram or you’ll slow way down.

Geezus_42 3y ago

I use an M1 Pro every day for work, and while it runs fantastically, I call bullshit on the 8GB claim.

brucethemoose2 3y ago

Before I got a desktop, I used to run my laptop (16GB RAM + a 6GB Nvidia GPU) headless to just barely squeeze the 33B models onto the GPU + RAM, and play with it on another device. ...I am not an OSX person, but if you can boot headless, it can squeeze 7B on there with full context