2 ms·
The problem is, the 128k window performed terribly and showed that attention was mostly limited to the first and last 20%. Increasing it to 1M just means even
by bearjaws 3y ago
The problem is, the 128k window performed terribly and showed that attention was mostly limited to the first and last 20%.
Increasing it to 1M just means even more data is ignored.
- cubefox 3y agoMaybe their architecture wasn't as good as MAMBA and Google could use the better architecture thanks to being late to the game...