4 ms·
As mentioned, these are all toy implementations and you should not use them in production. If you want to the fast, easy, and extremely optimized way of doing t
by f38zf5vdt 3y ago
As mentioned, these are all toy implementations and you should not use them in production. If you want to the fast, easy, and extremely optimized way of doing things, use torch.nn.MultiheadAttention or torch.nn.functional.scaled_dot_product_attention so that you get the optimal implementations. You can use xformers scaled dot product attention if you want the bleeding edge of performance.
> (Note that the code presented in this article is intended for illustrative purposes. If you plan to implement self-attention for training LLMs, I recommend considering optimized implementations like Flash Attention, which reduce memory footprint and computational load.)
Flash attention is already part of torch's kernels as of torch 2, but the latest versions and optimizations land in xformers first.
- varelse 3y ago[dead]
- radarsat1 3y agoIt seems that there are some popular attention methods such as relative embeddings and rotary embeddings (rope embeddings?) that are not possible to implement using pytorch's implementation, if I understand correctly. Do these then require the "slow path" versions that can be more easily modified?
- deleted 3y ago[deleted]
- rasbt 3y agoYes, totally agree. These implementations are meant for educational purposes. You could in theory use them to train a model though (GPT-2 also had a from-scratch implementation if I recall correctly). In practice, you probably want to use FlashAttention though. You use it through `torch.nn.functional.scaled_dot_product_attention` etc.
- nightski 3y agoDo you typically take tutorial blog posts and put them in production?
- f38zf5vdt 3y agoMore than I would like to admit