5 ms·
You would be surprised. The GPU often loses even for small neural nets given the large latency. Anything that needs high throughput or is sized like an HPC prob
by pclmulqdq 2y ago
You would be surprised. The GPU often loses even for small neural nets given the large latency. Anything that needs high throughput or is sized like an HPC problem should use a GPU, but a lot of code benefits from SIMD on small problems.
- gmueckl 2y agoIf you run many small tasks on the GPU, you can increase throughput by overlapping transfers and computation. There may also be other ways to batch problems together, but that depends on the algorithms. The one truly unfixable issue is round-trip latency.
- gopalv 2y ago> The GPU often loses even for small neural nets given the large latency Apple's neural engine shows that you can live in between those two worlds. As you said, the trouble is the latency, the programming model is still great.