3 ms·
> which should ensure precision, at least over short distances. why? i dont follow. transformers should provide some attention over -all- distances, no? why do
by swyx 3y ago
> which should ensure precision, at least over short distances.
why? i dont follow. transformers should provide some attention over -all- distances, no? why does layering truncate this to "short distances"?
- samus 3y agoI mean "short" in comparison to the unlimited, but lossy recall that the Mamba blocks provide. Transformers are limited to the context length, while Mamba can carry along state. While it can remember things from a lot farther back, it is limited and must thus eventually drop things and/or lose precision.