Ask and you shall receive: https://github.com/ryao/llama3.c/commit/dc27f2b1c6884c72bd4bd3d8409a94bd12fdcaaf https://github.com/ryao/llama3.c/commit/dc27f2b1c68

Ask and you shall receive: https://github.com/ryao/llama3.c/commit/dc27f2b1c6884c72bd4bd3d8409a94bd12fdcaaf https://github.com/ryao/llama3.c/commit/dc27f2b1c6884c72bd4b... It is still a WIP, but the CUDA kernels are unlikely to see much improvement for a while as I had gone over them with a fine toothed comb for performance in the past few days. The C code is more likely where I will be making improvements. For example, I need to switch to a structure to stop abusing an array of void pointers for the cudaMemcpy() aggregration trick for the cublasGemmBatchedEx() calls. The current version is performing either on par to or slightly better than llama.cpp's F16 codepath on my 3090 Ti, and I have not even implemented CUDA graphs yet. Both get around 53 tokens per second, although on really good runs, I have seen my code do 54 tokens per second and I have not seen that from llama.cpp so far.