3 ms·

Samba: Efficient Unlimited Context Language Modeling

• anon373839 2y ago
  > Introducing Samba 3.8B, a simple Mamba+Sliding Window Attention architecture that outperforms Phi3-mini on major benchmarks (e.g., MMLU, GSM8K and HumanEval) by a large margin. And it has an infinite context length with linear complexity. https://x.com/liliang_ren/status/1801027052147216457 https://x.com/liliang_ren/status/1801027052147216457