3 ms·
RecurrentGemma: Moving Past Transformers for Efficient Open Language Models
- bcherny 3y agoBlog post: https://developers.googleblog.com/2024/04/gemma-family-expands.html https://developers.googleblog.com/2024/04/gemma-family-expan... Github: https://github.com/google-deepmind/recurrentgemma https://github.com/google-deepmind/recurrentgemma HuggingFace: https://huggingface.co/google/recurrentgemma-2b https://huggingface.co/google/recurrentgemma-2b
- huac 3y agorelated previous discussion https://news.ycombinator.com/item?id=39993626 https://news.ycombinator.com/item?id=39993626
- deleted 3y ago[deleted]
- TOMDM 3y agoI'd be curious to see this architecture trained to a comparable level to Llama3. 8B params, 15T tokens of training. Llama3 8B supercedes ChatGPT3.5. If RecurrentGemma scales, then it would be both faster and more capable than its Llama peer.