9 ms·

Cerebras-GPT vs. LLaMA AI Model Performance Comparison

263 points by freeqaz 4y ago

ftxbro 4y ago
For context, the cerebras models were trained in only a couple weeks and the purpose of the training was to establish a scaling law for compute-optimal training, presumably for predicting what will happen with larger models where it's more important to train in a compute-optimal way. This is a different goal than that of other research projects that try to get the most power per VRAM on small models.
- freeqaz 4y ago
  That makes sense especially since they're not intending to deploy this model to production. For models like GPT-3/4 it makes sense why they would train them more because the costs of running the inference "in production" likely dominates the compute costs. (Just like how Youtube will spend 50x more compute to compress a video an extra 2% because bandwidth costs far outstrip the compression costs.) Do you know what percentage, roughly, this model has been trained relative to something like LLaMA? Are we talking 10%? 50%? 90%? It may be possible that it is still useful if it can be trained further by the community!
  - gpm 4y ago
    LLaMa 65B and 30B were trained on 1.4 trillion tokens. This model was trained on 260 billion tokens.
    - freeqaz 4y ago
      So ~18.6% trained relative to LLaMa. That's not _nothing_ but it's also not great. Thanks for digging into this!
- pama 4y ago
  The “law” was previously established empirically already and is only of relevance as a technical detail to a few specialists that may care. I think it was a strategic mistake to only release models that are weaker than what people can get their hands on. Is there a limit on that hardware scaling to larger models? As a hardware company that tries to stay in the game they should show some signs of dominance, not just Apache license.
  - UncleEntity 4y ago
    > The “law” was previously established empirically already and is only of relevance as a technical detail to a few specialists that may care. Or, you know, customers who want a quote on how much it’ll cost to train on their data. Not everyone has unlimited budgets to train a model to state of the art performance for giggles.
breadchris 4y ago
Wow! I didn't even know that Cerebras was a thing and I have been trying to keep up to date with this stuff!
- ftxbro 4y ago
  other hacker news cerebras discussion is here: https://news.ycombinator.com/item?id=35343763 https://news.ycombinator.com/item?id=35343763
- imaurer 4y ago
  Submit new links here :) https://github.com/imaurer/awesome-decentralized-llm https://github.com/imaurer/awesome-decentralized-llm
sbierwagen 4y ago
>>Is 10000 bigger than 10050? >>Yes, 10000 is bigger than 10050. >But even the mighty ChatGPT often can't do simple math GPT is bad at math because BPE input compression obfuscates individual digits. https://bbot.org/etc/gpt-math.png https://bbot.org/etc/gpt-math.png You'd be bad at math too if every number was scrambled. The graph is from page 22 of the GPT-3 paper from 2020. https://arxiv.org/abs/2005.14165 https://arxiv.org/abs/2005.14165 Even with 175 billion parameters it can't reliably do four digit addition. An example from 4 days ago of ChatGPT being as bad as you'd expect at string reversal: https://news.ycombinator.com/item?id=35297183 https://news.ycombinator.com/item?id=35297183 (Although, I just tested ChatGPT Mar 14 Version against the above question after doing a bunch of math prompting and it got it right...)
- cma 4y ago
  Llama individually tokenized digits, how much did it fix the issue?
  - nl 4y ago
    The most relevant reported benchmark results are on https://arxiv.org/abs/2103.03874 https://arxiv.org/abs/2103.03874 They compare it against PaLM and Minerva, both of which are fine tuned on math datasets. It outperforms both at roughly the same parameter count (eg Llama 65B vs Palm and Minerva 62B) but unclear how much of this is due to encoding vs the many other differences. It is useful to see that the performance increase is clearly not due to fine turning though.
    - cma 4y ago
      Where do I look, the paper was from 2021 before llama but did they update it (or you mean the llama paper mentions that benchmark)?
      - nl 4y ago
        I posted the wrong link sorry! https://arxiv.org/pdf/2302.13971.pdf https://arxiv.org/pdf/2302.13971.pdf (the Llama paper), table 7.
dumbaccount123 4y ago
[flagged]
- knicholes 4y ago
  Nobody is stopping you from living in a cave! ... at least I don't think so.
  - uejfiweun 4y ago
    I dunno man, you seen cave prices these days? The cave market is in a tough spot until the fed starts cutting…
    - deleted 4y ago
      [deleted]
  - flangola7 4y ago
    > Nobody is stopping you from living in a cave! A bunch of tights wearing twats manufactured this idea of "private land" and now violence is waged against those seeking cave life. Also caves don't stop the Shoggoth from finding you.
nl 4y ago
Tl;DR: It's quite a lot worse than LLama 13B in every way that is measured. It benchmarks between GPT-J (6B) and GPT-NeoX-20B, although is worse than GPT-J (6B) in a few benchmarks. There are no good benchmarks for text generation, but it would be surprising if it was competitive with the best open models. It's unclear how much RLHF can help here - AFAIK no one has tried out DataBrick's Dolly to compare a GPT-J based RLHF model.
carlsborg 4y ago
The LLM model performance is a sideshow. They are demoing linear scaling on their "cluster on a wafer" tech (a vertically scaled single chip "the largest computer chip ever built and the fastest AI processor on Earth") vs distributed training. They have a cloud service offering for example fine tuning and pretraining well known LLM models on their chips. (est. $40k to train GPT3 6.7B in ~11 days)
varunkmohan 4y ago
Sad, this whole thing where companies train chinchilla optimal models has to stop - I'm always happy to get open source models but it would be great if it was useful rather than a benchmark.

263 points by freeqaz 4y ago

For context, the cerebras models were trained in only a couple weeks and the purpose of the training was to establish a scaling law for compute-optimal training, presumably for predicting what will happen with larger models where it's more important to train in a compute-optimal way. This is a different goal than that of other research projects that try to get the most power per VRAM on small models.

freeqaz 4y ago
That makes sense especially since they're not intending to deploy this model to production. For models like GPT-3/4 it makes sense why they would train them more because the costs of running the inference "in production" likely dominates the compute costs. (Just like how Youtube will spend 50x more compute to compress a video an extra 2% because bandwidth costs far outstrip the compression costs.) Do you know what percentage, roughly, this model has been trained relative to something like LLaMA? Are we talking 10%? 50%? 90%? It may be possible that it is still useful if it can be trained further by the community!
- gpm 4y ago
  LLaMa 65B and 30B were trained on 1.4 trillion tokens. This model was trained on 260 billion tokens.
  - freeqaz 4y ago
    So ~18.6% trained relative to LLaMa. That's not _nothing_ but it's also not great. Thanks for digging into this!
pama 4y ago
The “law” was previously established empirically already and is only of relevance as a technical detail to a few specialists that may care. I think it was a strategic mistake to only release models that are weaker than what people can get their hands on. Is there a limit on that hardware scaling to larger models? As a hardware company that tries to stay in the game they should show some signs of dominance, not just Apache license.
- UncleEntity 4y ago
  > The “law” was previously established empirically already and is only of relevance as a technical detail to a few specialists that may care. Or, you know, customers who want a quote on how much it’ll cost to train on their data. Not everyone has unlimited budgets to train a model to state of the art performance for giggles.

breadchris 4y ago

Wow! I didn't even know that Cerebras was a thing and I have been trying to keep up to date with this stuff!

ftxbro 4y ago
other hacker news cerebras discussion is here: https://news.ycombinator.com/item?id=35343763 https://news.ycombinator.com/item?id=35343763
imaurer 4y ago
Submit new links here :) https://github.com/imaurer/awesome-decentralized-llm https://github.com/imaurer/awesome-decentralized-llm

sbierwagen 4y ago

>>Is 10000 bigger than 10050? >>Yes, 10000 is bigger than 10050. >But even the mighty ChatGPT often can't do simple math GPT is bad at math because BPE input compression obfuscates individual digits. https://bbot.org/etc/gpt-math.png https://bbot.org/etc/gpt-math.png You'd be bad at math too if every number was scrambled. The graph is from page 22 of the GPT-3 paper from 2020. https://arxiv.org/abs/2005.14165 https://arxiv.org/abs/2005.14165 Even with 175 billion parameters it can't reliably do four digit addition. An example from 4 days ago of ChatGPT being as bad as you'd expect at string reversal: https://news.ycombinator.com/item?id=35297183 https://news.ycombinator.com/item?id=35297183 (Although, I just tested ChatGPT Mar 14 Version against the above question after doing a bunch of math prompting and it got it right...)

cma 4y ago
Llama individually tokenized digits, how much did it fix the issue?
- nl 4y ago
  The most relevant reported benchmark results are on https://arxiv.org/abs/2103.03874 https://arxiv.org/abs/2103.03874 They compare it against PaLM and Minerva, both of which are fine tuned on math datasets. It outperforms both at roughly the same parameter count (eg Llama 65B vs Palm and Minerva 62B) but unclear how much of this is due to encoding vs the many other differences. It is useful to see that the performance increase is clearly not due to fine turning though.
  - cma 4y ago
    Where do I look, the paper was from 2021 before llama but did they update it (or you mean the llama paper mentions that benchmark)?
    - nl 4y ago
      I posted the wrong link sorry! https://arxiv.org/pdf/2302.13971.pdf https://arxiv.org/pdf/2302.13971.pdf (the Llama paper), table 7.

dumbaccount123 4y ago

[flagged]

knicholes 4y ago
Nobody is stopping you from living in a cave! ... at least I don't think so.
- uejfiweun 4y ago
  I dunno man, you seen cave prices these days? The cave market is in a tough spot until the fed starts cutting…
  - deleted 4y ago
    [deleted]
- flangola7 4y ago
  > Nobody is stopping you from living in a cave! A bunch of tights wearing twats manufactured this idea of "private land" and now violence is waged against those seeking cave life. Also caves don't stop the Shoggoth from finding you.

nl 4y ago

Tl;DR: It's quite a lot worse than LLama 13B in every way that is measured. It benchmarks between GPT-J (6B) and GPT-NeoX-20B, although is worse than GPT-J (6B) in a few benchmarks. There are no good benchmarks for text generation, but it would be surprising if it was competitive with the best open models. It's unclear how much RLHF can help here - AFAIK no one has tried out DataBrick's Dolly to compare a GPT-J based RLHF model.

carlsborg 4y ago

The LLM model performance is a sideshow. They are demoing linear scaling on their "cluster on a wafer" tech (a vertically scaled single chip "the largest computer chip ever built and the fastest AI processor on Earth") vs distributed training. They have a cloud service offering for example fine tuning and pretraining well known LLM models on their chips. (est. $40k to train GPT3 6.7B in ~11 days)

varunkmohan 4y ago

Sad, this whole thing where companies train chinchilla optimal models has to stop - I'm always happy to get open source models but it would be great if it was useful rather than a benchmark.