4 ms·

Yeah, 70B is much larger and fits on a 24GB, admitedly with very lossy quantization. This is just about right for 24GB. I bet that is intentional on their part

Yeah, 70B is much larger and fits on a 24GB, admitedly with very lossy quantization. This is just about right for 24GB. I bet that is intentional on their part.