4 ms·
Is the limit on the speed on inference a memory bandwidth issue or compute?
by christkv 3y ago
Is the limit on the speed on inference a memory bandwidth issue or compute?
- thatguysaguy 3y agoMemory bandwidth/latency, especially when you're at smaller batch sizes.
- huac 3y ago"it depends" https://kipp.ly/transformer-inference-arithmetic/ https://kipp.ly/transformer-inference-arithmetic/
- brucethemoose2 3y agoDepends. One might say its sometimes "cache size limited" too.