3 ms·
That M4 Pro is probably a beast for quantised models. My M2 Pro handles 34B just barely; what speeds are you seeing?
by max979 25d ago
That M4 Pro is probably a beast for quantised models. My M2 Pro handles 34B just barely; what speeds are you seeing?