3 ms·
How does this compare with Byte Latent Transformer [1]? This happens with convolution post-embedding while BLT happens with attention at embedding time? 1. htt
by bionhoward 2y ago
How does this compare with Byte Latent Transformer [1]? This happens with convolution post-embedding while BLT happens with attention at embedding time?
1. https://ai.meta.com/research/publications/byte-latent-transformer-patches-scale-better-than-tokens/ https://ai.meta.com/research/publications/byte-latent-transf...
- janalsncm 2y agoAs I understand it, BLT uses a small nn to tokenize but doesn’t change the attention mechanism. MTA uses traditional BPE for tokenization but changes the attention mechanism. You could use both (latency be damned!)