4 ms·
FlashAttention-2, 2x faster than FlashAttention
- lucidrains 3y agohuge! thank you Tri!
- bufo 3y agoTri Dao and Tim Dettmers ftw
- deleted 3y ago[deleted]
- whimsicalism 3y agoDoes anyone have resources for a good way to get started with this sort of modern GPU systems work?
- brrrrrm 3y agoI'd start with the example of implementing the fastest reduction you possibly can. Pretty much all complexity in every kernel used in ML extends from this concept (reductions with addition). https://developer.download.nvidia.com/assets/cuda/files/reduction.pdf https://developer.download.nvidia.com/assets/cuda/files/redu...
- whimsicalism 3y agothank you for the suggestion - will take a look!
- FL33TW00D 3y agoI love this PDF, some crazy sticking power since he's referring to a G80 GPU there!
- jahewson 3y agoIf you’d like a practical goal, you probably want to learn PyTorch and have a little background knowledge of the memory architecture of the GPUs. If you want to go deep, learn CUDA: https://docs.nvidia.com/cuda/cuda-c-programming-guide/index.html#programming-model https://docs.nvidia.com/cuda/cuda-c-programming-guide/index....
- whimsicalism 3y agoYes, I know pytorch well at this point and have basic memory architecture understanding. In the process of learning CUDA, but would love pointers for depth/intermediate things to explore.
- jahewson 3y agoI found this talk helpful. https://on-demand.gputechconf.com/gtc/2017/presentation/s7122-stephen-jones-cuda-optimization-tips-tricks-and-techniques.pdf https://on-demand.gputechconf.com/gtc/2017/presentation/s712... Have you tried the Visual Profiler yet?
- luckyt 3y agoI found it helpful to start with CUDA on numba since it lets you write GPU kernels in python. Assuming you're like most ML engineers and you're more familiar with python than C++, this allows you to separately learn CUDA concepts from also learning C++ at the same time. There's also a set of GPU puzzles for beginners [1] using to get started with numba CUDA. [1] https://github.com/srush/GPU-Puzzles https://github.com/srush/GPU-Puzzles
- whimsicalism 3y agoThanks for the link! Sasha is actually my former professor - if this is anything like his past pytorch puzzles I'm sure I'll find it enjoyable.
- ternaus 3y agoI would be very greatfull to see how one can leverage it not for LLMs but for Stable Diffusion models
- m00x 3y agoWhy couldn't it be applied to SD?
- m00x 3y agoIt looks like it's already at thing https://github.com/AUTOMATIC1111/stable-diffusion-webui/blob/f865d3e11647dfd6c7b2cdf90dde24680e58acd8/modules/cmd_args.py#L54 https://github.com/AUTOMATIC1111/stable-diffusion-webui/blob...
- zaptrem 3y agoThat's Flash Attention 1, but yeah there's nothing stopping someone from submitting a PR.
- hunkins 3y agoThis enables V1. V2 is still yet to be integrated into xformers. The team replied saying it should happen this week. See the relevant Github issue here: https://github.com/facebookresearch/xformers/issues/795 https://github.com/facebookresearch/xformers/issues/795
- mcyc 3y agoSD1 uses a head dimension that is not supported by the old version of Flash Attention but is supported by this version.
- hunkins 3y agoExcellent work by the team on this. Excited to see this applied to the diffusers library.