3 ms·
most likely he was referring the fact that you need plenty of GPU-fast memory to keep the model, and GPU cards have it.
by oreoftw 1y ago
most likely he was referring the fact that you need plenty of GPU-fast memory to keep the model, and GPU cards have it.
- deleted 1y ago[deleted]
- adastra22 1y agoThere is nothing magical about GPU memory though. It’s just faster. But people have been doing CPU inference since the first llama code came out.