3 ms·
If there's really no catch then this is a groundbreaking paper. Unambiguously beating transformers with margin on all metrics they tested. https://imgur.com/a/
by famouswaffles 3y ago
If there's really no catch then this is a groundbreaking paper.
Unambiguously beating transformers with margin on all metrics they tested.
https://imgur.com/a/WKFF1yv https://imgur.com/a/WKFF1yv
- huac 3y agosome catches I'm looking for: it remains to be seen if the generations are high quality (not just low perplexity) and if the architecture scales beyond 3B, but definitely super promising
- wigglin 3y agoSee section 4 of the paper, 'Empirical Evaluation.' Meets or exceeds performance of similarly sized transformers on various benchmarks such as Hellaswag and Winogrande. Seems very promising.
- deleted 3y ago[deleted]
- cs702 3y agoHi og_kalu! I agree. Let me add that the authors of RWKV and Linear Attention deserve some credit too, given the connection between those linear RNNs and these new "selective" SSM models. See my comment here: https://news.ycombinator.com/item?id=38529943 https://news.ycombinator.com/item?id=38529943
- famouswaffles 3y agoOh for sure. Didn't mean to imply otherwise!
- cs702 3y ago:-)