Its not so straightforward, as theres some overhead aside from the weights, especially with 7B at ~4 bit. But this is probably capable of squeezing onto a 6GB

Its not so straightforward, as theres some overhead aside from the weights, especially with 7B at ~4 bit. But this is probably capable of squeezing onto a 6GB (or less?) GPU with the right backend.