3 ms·
Probably if you have any esoteric flags that pytorch supports. Flash attention 2, for example, was supported way earlier on pt than llama.cpp, so if flash atten
by JackYoustra 2y ago
Probably if you have any esoteric flags that pytorch supports. Flash attention 2, for example, was supported way earlier on pt than llama.cpp, so if flash attention 3 follows the same path it'll probably make more sense to use this when targeting nvidia gpus.
- sunshinesfbay 2y agoIt would appear that Flash-3 is already something that exists for PyTorch based on this joint blog between Nvidia, Together.ai and Princeton about enabling Flash-3 for PyTorch: https://pytorch.org/blog/flashattention-3/ https://pytorch.org/blog/flashattention-3/
- JackYoustra 2y agoRight - my point about "follows the same path" mostly revolves around llama.cpp's latency in adopting it.