Regarding how they’re getting to 10M context, I think it’s possible they are using the new SAMBA architecture. Here’s the paper: https://arxiv.org/abs/2312.007

Regarding how they’re getting to 10M context, I think it’s possible they are using the new SAMBA architecture. Here’s the paper: https://arxiv.org/abs/2312.00752 https://arxiv.org/abs/2312.00752 And here’s a great podcast episode on it: https://www.cognitiverevolution.ai/emergency-pod-mamba-memory-and-the-ssm-moment/ https://www.cognitiverevolution.ai/emergency-pod-mamba-memor...