3 ms·
This one should have you covered :-) one out of every eight layers is a traditional Transformer layer, which should ensure precision, at least over short distan
by samus 3y ago
This one should have you covered :-) one out of every eight layers is a traditional Transformer layer, which should ensure precision, at least over short distances.
- swyx 3y ago> which should ensure precision, at least over short distances. why? i dont follow. transformers should provide some attention over -all- distances, no? why does layering truncate this to "short distances"?
- samus 3y agoI mean "short" in comparison to the unlimited, but lossy recall that the Mamba blocks provide. Transformers are limited to the context length, while Mamba can carry along state. While it can remember things from a lot farther back, it is limited and must thus eventually drop things and/or lose precision.