2 ms·
IDK, 8B-class quantized models run pretty fast on commodity laptops, with CPU-only inference. Thanks to the people who figured out quantization and reimplemente
by atemerev 2y ago
IDK, 8B-class quantized models run pretty fast on commodity laptops, with CPU-only inference. Thanks to the people who figured out quantization and reimplemented everything in C++, instead of academic-grade Python.
- actualwitch 2y agoA solid chunk of python is just wrappers around C/C++, most tensor frameworks included.
- atemerev 2y agoI know, and yet early model implementations were quite unoptimized compared to the modern ones.