4 ms·
The slowest part is loading weights into vram in my experience. I haven't done benchmarking on that. What kind of benchmark would you like to see?
by xena 3y ago
The slowest part is loading weights into vram in my experience. I haven't done benchmarking on that. What kind of benchmark would you like to see?
- ipsum2 3y agoI would like to see time to first inference for typical models (llama-7b first token, SDXL 1 step, etc)