3 ms·
> running CodeBooga 34b @ 4bit on my work laptop. How much RAM or VRAM does that require?
by janice1999 3y ago
> running CodeBooga 34b @ 4bit on my work laptop.
How much RAM or VRAM does that require?
- valine 3y agoIt’ll be around 17GB + context. For longer contexts you’ll add a couple GBs.
- _boffin_ 3y agoMax ram is saying 22.72GB. They gave me a laptop with a M2 MAX and 64GB of ram. I'm loving the fact that i can toss any piece of code in there without having to worry about data privacy issues and or any retention policies of service providers.
- rf15 3y agoIn my experience the vastly oversimplified-beyond-actual-usability calculation is that you take the parameter number (34) and count that as the amount in GB you'd need for an 8 bit model. with a 4-bit quantization that's halved, so your ball park is 17 GB plus a few GB for context. You can run these in RAM, but it's not very fast with (if I had to guess) slightly less than 1 Token/sec. on most non-apple consumer devices.