4 ms·
The nn.Block should probably be renamed to nn.DecoderBlock as it's not very clear if it's supposed to be an encoder or a decoder block, also an option to disabl
by GaggiX 3y ago
The nn.Block should probably be renamed to nn.DecoderBlock as it's not very clear if it's supposed to be an encoder or a decoder block, also an option to disable the mask from attention. That said, very cool project.
- eduardoleao052 3y agoYou are absolutely right! Will solve that as soon as possible. Thanks for the feedback!
- _ea1k 3y agoIf you have the option to disable the mask, isn't it then a generic nn.Block?
- eduardoleao052 3y agoIt would be with some simple tweaks. For instance, the current block does not support Cross-Attention, just Self-Attention.
- GaggiX 3y agoThe ability to disable the mask in nn.MultiHeadSelfAttention, then having nn.DecoderBlock and a nn.EncoderBlock.
- eduardoleao052 3y agoI think that’s it. I’ll probably add that soon