4 ms·
>Small models only trained on 800B tokens "These models will be trained on up to 1.5 trillion tokens." on the Github repo. https://github.com/stability-AI/sta
by GaggiX 4y ago
>Small models only trained on 800B tokens
"These models will be trained on up to 1.5 trillion tokens." on the Github repo.
https://github.com/stability-AI/stableLM/#stablelm-alpha https://github.com/stability-AI/stableLM/#stablelm-alpha
- Taek 4y agoDevs confirmed that the small ones use 800B, 1.5T is for the large ones
- GaggiX 4y ago@thunderbird120 asked a Stability employee and say that the plan is going to keep training the models up to 1.5T. So I don't know where do you read this.
- nickthegreek 4y agohttps://github.com/Stability-AI/StableLM#stablelm-alpha https://github.com/Stability-AI/StableLM#stablelm-alpha shows that the 3b and 7B had 800b training tokens.
- Taek 4y agoThat may be, but the weights you can download today were trained on 800B
- GaggiX 4y agoyes of course that's why they use "will be trained" on the GH repo.
- sroussey 4y agoI think they are “checkpoint” models in this case. Will be fun to compare when completed!
- oehtXRwMkIs 4y agoAre not all models checkpoints? I think you may be interpreting it too colloquially.
- youssefabdelm 4y agoThat's great news, but one would think that since they're behind Stable Diffusion, that they'd use the insights behind it and scale data even more than that to result in better quality at a smaller scale model that can run on most people's machines. Like... try 10 trillion or 100 trillion tokens (although that may be absurd, I never did the calculation), and a long context on a 7B parameter model then see if that gets you better results than a 30 or 65B parameter on 1.5 trillion tokens. A lot of these open source projects just seem to be trying to follow and (poorly) reproduce OpenAI's breakthroughs instead of trying to surpass them.
- dragonwriter 4y ago> Like... try 10 trillion or 100 trillion tokens (although that may be absurd, I never did the calculation) But where’s the corpus supposed ro come from?
- GaggiX 4y ago>try 10 trillion or 100 trillion tokens Computation is not free and data is not infinite.
- youssefabdelm 4y agoYou could've said the same to OpenAI when they were scaling GPT from 1 billion to 175 billion parameters. We're all grateful they didn't follow that line of thought. But Stability does have access to a pretty big cluster, so it's not paying cloud compute (I assume), so cost will be less, and data of course is not infinite...never stated that. But considering 3.7 million videos are uploaded to youtube everyday, 2 million scientific articles published every year, yada yada...that argument falls apart. At the very least implement spiral development... 1 trillion... 3 trillion... (oh it seems to be getting WAY better! There seems to be a STEP CHANGE!)... 5 trillion... (holy shit this really works, lets keep going)
- dragonwriter 4y agoThe training corpus is the problem. An extra trillion tokens is (ballpark) an extra million KJV bibles worth of text formatted for ingestion. And you probably picked all of the low hanging fruit in terms of quality prior vetting and being in a standard format for ingestion in your first trillion tokens of training data.