3 ms·
The bandwidth is (potentially, depending on the Mac in question) much lower on the framework, which will limit the decode tokens/s
by buildbot 2y ago
The bandwidth is (potentially, depending on the Mac in question) much lower on the framework, which will limit the decode tokens/s
- dismalaf 2y agoThat's fair but still better than running on the CPU, and still enough to hold the model in VRAM.