2 ms·
Thank you for explaining. Do you think there are still opportunities for stack optimizations to meaningfully speed up inference on single consumer-grade GPUs?
by dr_kiszonka 4mo ago
Thank you for explaining. Do you think there are still opportunities for stack optimizations to meaningfully speed up inference on single consumer-grade GPUs?
- gaeld 4mo agoI'm sure there are, and I really hope we can work on consumer-grade GPUs at some point. It should be possible to apply the same methodology (digging deep into the hardware details to understand all its little characteristics, and rethinking the inference stack around that).