3 ms·
Generative AI and the big buzz about small language models
- milliondreams 3y agoAs we see these systems evolving, I have come to believe specialist small language models with an MoE framework are the future of the industry.
- swimwiththebeat 3y agoDoes anyone know if this is using the Mamba architecture[1] instead of transformers? It looks like it uses a state space model (SSM) layer. [1]: https://arxiv.org/abs/2312.00752 https://arxiv.org/abs/2312.00752
- sal9000 3y agoIt came earlier than Mamba. It uses hyena hierarchy blocks, which are considered SSM but not the same as Mamba.
- milliondreams 3y agoWe covered state space models in a blog post here - https://blog.dragonscale.ai/state-space-models/ https://blog.dragonscale.ai/state-space-models/ It gives overview of Mamba And StrypedHyna.
- compressedgas 3y agoPiece with less detail than the source linked from the article: https://www.together.ai/blog/stripedhyena-7b https://www.together.ai/blog/stripedhyena-7b