3 ms·

For context, the cerebras models were trained in only a couple weeks and the purpose of the training was to establish a scaling law for compute-optimal training

by ftxbro 4y ago

For context, the cerebras models were trained in only a couple weeks and the purpose of the training was to establish a scaling law for compute-optimal training, presumably for predicting what will happen with larger models where it's more important to train in a compute-optimal way. This is a different goal than that of other research projects that try to get the most power per VRAM on small models.

freeqaz 4y ago
That makes sense especially since they're not intending to deploy this model to production. For models like GPT-3/4 it makes sense why they would train them more because the costs of running the inference "in production" likely dominates the compute costs. (Just like how Youtube will spend 50x more compute to compress a video an extra 2% because bandwidth costs far outstrip the compression costs.) Do you know what percentage, roughly, this model has been trained relative to something like LLaMA? Are we talking 10%? 50%? 90%? It may be possible that it is still useful if it can be trained further by the community!
- gpm 4y ago
  LLaMa 65B and 30B were trained on 1.4 trillion tokens. This model was trained on 260 billion tokens.
  - freeqaz 4y ago
    So ~18.6% trained relative to LLaMa. That's not _nothing_ but it's also not great. Thanks for digging into this!
pama 4y ago
The “law” was previously established empirically already and is only of relevance as a technical detail to a few specialists that may care. I think it was a strategic mistake to only release models that are weaker than what people can get their hands on. Is there a limit on that hardware scaling to larger models? As a hardware company that tries to stay in the game they should show some signs of dominance, not just Apache license.
- UncleEntity 4y ago
  > The “law” was previously established empirically already and is only of relevance as a technical detail to a few specialists that may care. Or, you know, customers who want a quote on how much it’ll cost to train on their data. Not everyone has unlimited budgets to train a model to state of the art performance for giggles.

by ftxbro 4y ago

freeqaz 4y ago

That makes sense especially since they're not intending to deploy this model to production. For models like GPT-3/4 it makes sense why they would train them more because the costs of running the inference "in production" likely dominates the compute costs. (Just like how Youtube will spend 50x more compute to compress a video an extra 2% because bandwidth costs far outstrip the compression costs.) Do you know what percentage, roughly, this model has been trained relative to something like LLaMA? Are we talking 10%? 50%? 90%? It may be possible that it is still useful if it can be trained further by the community!

gpm 4y ago
LLaMa 65B and 30B were trained on 1.4 trillion tokens. This model was trained on 260 billion tokens.
- freeqaz 4y ago
  So ~18.6% trained relative to LLaMa. That's not _nothing_ but it's also not great. Thanks for digging into this!

pama 4y ago

The “law” was previously established empirically already and is only of relevance as a technical detail to a few specialists that may care. I think it was a strategic mistake to only release models that are weaker than what people can get their hands on. Is there a limit on that hardware scaling to larger models? As a hardware company that tries to stay in the game they should show some signs of dominance, not just Apache license.

UncleEntity 4y ago
> The “law” was previously established empirically already and is only of relevance as a technical detail to a few specialists that may care. Or, you know, customers who want a quote on how much it’ll cost to train on their data. Not everyone has unlimited budgets to train a model to state of the art performance for giggles.