3 ms·
I honestly don't see why it's so hard. On my project we wrote our own gemm kernels from scratch so llama.cpp didn't need to depend on cublas anymore. Only took
by jart 3y ago
I honestly don't see why it's so hard. On my project we wrote our own gemm kernels from scratch so llama.cpp didn't need to depend on cublas anymore. Only took a few days and a few hundred lines of code. We had to trade away 5% performance.
- soulbadguy 3y agoFor a given set kernels, and a limited set of architectures, the problem is relatively easy. But covering all the important kernels acros all the crazy architecture out there and with relatively good performance and numerical accuracy ... Much harder