2 ms·
Running inference with LLMs is not very complex. You don't need CUDA for that at all. You don't even need those complex libraries for inference. The market for
by threatripper 2y ago
Running inference with LLMs is not very complex. You don't need CUDA for that at all. You don't even need those complex libraries for inference. The market for inference will grow much more as AI is used in more and more applications by more and more people. NVIDIA has a moat but it's around the smaller castle.
- YetAnotherNick 2y agoYou need CUDA for flash attention. Other than that its just matrix multiplication, embedding and point wise activation.
- f_devd 2y agoOnly for the reference implementation, flash attention is just based on optimization of the memory bandwidth between caches. The ThunderKittens[0] library has what is effectively flash attention v3, and they are working on supporting AMD arch. [0]: https://hazyresearch.stanford.edu/blog/2024-05-12-tk https://hazyresearch.stanford.edu/blog/2024-05-12-tk