3 ms·
You mean red pajama? I believe that has already started for 1-14B (need to double check)
by pico_creator 3y ago
You mean red pajama? I believe that has already started for 1-14B (need to double check)
- logicchains 3y agoYep that's the one. Curious roughly how many A100s it'd take to train a 65B RWKV on that.
- pico_creator 3y agoReally bad napkin math as no one has attempted 65B (so +\- 50%) 8 x 8 x 8 A100, should be able to do a 100k++ tokens/s at that size With a dataset of 1.2 trillion tokens. That’s 12 million seconds. Or 140 days (PS: this is why everyone is training <60B, its crazy the cost, even if my math estimate is wrong by 300%, its still a crazy number)
- logicchains 3y agoThank you! 888 is 512 A100s, that is indeed pretty expensive.