3 ms·

This, plus metal acceleration works quite well. 7~8B parameter models quantized to 3bpw or so run with good tok/s on my iphone 15 pro

• sanxiyn 2y ago
  It works quite well as long as you don't care about battery.