In my experience the vastly oversimplified-beyond-actual-usability calculation is that you take the parameter number (34) and count that as the amount in GB you

In my experience the vastly oversimplified-beyond-actual-usability calculation is that you take the parameter number (34) and count that as the amount in GB you'd need for an 8 bit model. with a 4-bit quantization that's halved, so your ball park is 17 GB plus a few GB for context. You can run these in RAM, but it's not very fast with (if I had to guess) slightly less than 1 Token/sec. on most non-apple consumer devices.