3 ms·
Sure but can they fit a bigger model? I don’t think they can string together these to fit bigger models like llama3.1 405b
by m3kw9 2y ago
Sure but can they fit a bigger model? I don’t think they can string together these to fit bigger models like llama3.1 405b
- sanxiyn 2y agoI am not sure why not. From the article: "In the weeks to come we will be adding larger models such as Llama3-405B".
- bigcat12345678 2y agohttps://youtu.be/re4QqXPmfgs?t=956 https://youtu.be/re4QqXPmfgs?t=956 CS-3 system is built for single node domain scaling to 24 Trillion parameter models. I.e., they claim you can run the same code without hand-written distributed training code to reach 24 Trillion parameter models.