6 ms·

The bottom of https://ai.meta.com/blog/meta-llama-3/ https://ai.meta.com/blog/meta-llama-3/ has in-progress results for the 400B model as well. Looks like it's

by sp332 2y ago

The bottom of https://ai.meta.com/blog/meta-llama-3/ https://ai.meta.com/blog/meta-llama-3/ has in-progress results for the 400B model as well. Looks like it's not quite there yet. Llama 3 400B Base / Instruct MMLU 84.8 86.1 GPQA - 48.0 MATH - 57.8 HumanEval - 84.1 DROP 83.5 -

YetAnotherNick 2y ago
For the still training 400B: Llama 3 GPT 4(Published) BBH 85.3 83.1 MMLU 86.1 86.4 DROP 83.5 80.9 GSM8K 94.1 92.0 MATH 57.8 52.9 HumEv 84.1 74.4 Although it should be noted that the API numbers were generally better than published numbers for GPT4. [1]: https://deepmind.google/technologies/gemini/ https://deepmind.google/technologies/gemini/
- oliwary 2y ago
  Wild! So if this indeed holds up, it looks like OpenAI were about a year ahead when GPT-4 was released, compared to the open source world. However, given the timespan between matching GPT-3.5 (Mixtral perhaps?) and matching GPT-4 has just been a few weeks, I am wondering if the open source models have more momentum. That said, I am very curious what OpenAI has in their labs... Are they actually barely ahead? Or do they have something much better that is not yet public? Perhaps they were waiting for Llama 3 to show it? Exciting times ahead either way!
  - ChildOfChaos 2y ago
    You've also got to consider that we don't really know where OpenAI are though, what they have released in the past year have been tweaks to GPT4, while I am sure the real work is going into GPT5 or whatever it gets called. While all the others are catching up and in some cases being slightly better, I wouldn't be surprised to see a rather large leap back into the lead from OpenAI pretty soon and then a scrabble for some time for others to get close again. We will really see who has the momentum soon, when we see OpenAI's next full release.
- tmikaeld 2y ago
  Hm, how much VRAM would this take to run?
  - asadm 2y ago
    yes
  - mrtranscendence 2y ago
    Back of the envelope, maybe 0.75TB? More than you have, probably ...
  - bearjaws 2y ago
    My guess is around 256GiB but it depends on what level of quantization you are okay with. At full 16bit it will be massive, near 512GiB. I figure we will see some Q4's that can probably fit on 4 4090s with CPU offloading.
    - sp332 2y ago
      With 400 billion parameters and 8 bits per parameter, wouldn't it be ~400 GB? Plus context size which could be quite large.
      - yalok 2y ago
        he said "Q4" - meaning 4-bit weights.
        sp332 2y ago
        Ok but at 16-bit it would be 800GB+, right? Not 512.
        reactordev 2y ago
        Divide not multiply. If a size is estimated in 8-bit, reducing to 4-bit halves the size (and entropy of each value). Difference between INT_MAX and SHORT_MAX (assuming you have such defs). I could be wrong too but that’s my understanding. Like float vs half-float.
- mdeeks 2y ago
  Which specific GPT-4 model is this? gpt-4-0613? gpt-4-0125-preview?
  - YetAnotherNick 2y ago
    This is mostly from technical report from OpenAI[1]. API performs better as I said in my previous comment. API models(0613/0125 etc.) also uses user data for training which could leak the benchmark data. [1]: https://arxiv.org/pdf/2303.08774.pdf https://arxiv.org/pdf/2303.08774.pdf
    - pama 2y ago
      IIRC this model had finished pretraining in the summer of 2022.
- tedsanders 2y ago
  Those numbers are for the original GPT-4 (Mar 2023). Current GPT-4-Turbo (Apr 2024) is better: Llama 3 GPT-4 GPT-4-Turbo* (Apr 2024) MMLU 86.1 86.4 86.7 DROP 83.5 80.9 86.0 MATH 57.8 52.9 73.4 HumEv 84.1 74.4 88.2 *using API prompt: https://github.com/openai/simple-evals https://github.com/openai/simple-evals
  - natrys 2y ago
    I find it somewhat interesting that there is a common perception about GPT-4 at release being actually smart, but that it got gradually nerfed for speed with turbo, which is better tuned but doesn't exhibit intelligence like the original. There were times when I felt that too, but nowadays I predominantly use turbo. It's probably because turbo is faster and cheaper, but in lmsys turbo has 100 elo higher than original, so by and large people simply find turbo to be....better? Nevertheless, I do wonder if not just in benchmarks but in how people use LLMs, intelligence is somewhat under utilised, or possibly offset by other qualities.
    - thelittleone 2y ago
      Have you tried Claude 3 Opus? I've been using that predominantly since release and find it's "smarts" as or better than my experience with GPT-4 (pre turbo).
      - natrys 2y ago
        I did. It definitely exudes more all around personality. Unfortunately in my private test suite (mostly about coding), it did somewhat worse than turbo or phind 70b. Since price influences my calculus, I can't say this for sure, but it seems being slightly smarter is not much of an edge, because it's still dumb by human standards. For most non-coding use the smart doesn't make much difference (like summarisation), I find that cheaper options like mistral-large do just as good as Opus. In the last month I have used Command R+ more and more. Finally had some excuse to write some function calling stuff. I have also been highly impressed by Gemini Pro 1.5 finding technical answers from a dense 650 page pdf manual. I have enjoyed chatting with the WizardLM2 fine-tune for the past few days. Somehow I haven't quite found a consistent use case for Opus.
modeless 2y ago
Not quite there yet, but very close and not done training! It's quite plausible that this model could be state of the art over GPT-4 in some domains when it finishes training, unless GPT-5 comes out first. Although 400B will be pretty much out of reach for any PC to run locally, it will still be exciting to have a GPT-4 level model in the open for research so people can try quantizing, pruning, distilling, and other ways of making it more practical to run. And I'm sure startups will build on it as well.
- int_19h 2y ago
  The real question will be, how much you can quantize that while still retaining sanity. 400b at 2-bit would be possible to run on a Mac Studio - probably at multiple seconds per token, but sometimes that's "fast enough".
  - modeless 2y ago
    Yes. I expect an explosion of research and experimentation in model compression. The good news is I think there are tons of avenues that have barely been explored at all. We are at the very beginning of understanding this stuff, and my bet is that in a few years we'll be able to compress these models 10x or more.
- brandall10 2y ago
  There are rumors about an upcoming M3 or M4 Extreme chip... which would certainly have enough RAM, and probably a 1600-2000 GB/s bandwidth. Still wouldn't be super performant AFA token gen, ~4-6 per second, but certainly runnable. Of course by the time that lands in 6-12 months we'll probably have a 70-100G model that is similarly performant.
jug 2y ago
This is tantalizingly close in multiple benchmarks though. Pretty sure this one will finally be the open GPT-4 match.

by sp332 2y ago

YetAnotherNick 2y ago

For the still training 400B: Llama 3 GPT 4(Published) BBH 85.3 83.1 MMLU 86.1 86.4 DROP 83.5 80.9 GSM8K 94.1 92.0 MATH 57.8 52.9 HumEv 84.1 74.4 Although it should be noted that the API numbers were generally better than published numbers for GPT4. [1]: https://deepmind.google/technologies/gemini/ https://deepmind.google/technologies/gemini/

oliwary 2y ago
Wild! So if this indeed holds up, it looks like OpenAI were about a year ahead when GPT-4 was released, compared to the open source world. However, given the timespan between matching GPT-3.5 (Mixtral perhaps?) and matching GPT-4 has just been a few weeks, I am wondering if the open source models have more momentum. That said, I am very curious what OpenAI has in their labs... Are they actually barely ahead? Or do they have something much better that is not yet public? Perhaps they were waiting for Llama 3 to show it? Exciting times ahead either way!
- ChildOfChaos 2y ago
  You've also got to consider that we don't really know where OpenAI are though, what they have released in the past year have been tweaks to GPT4, while I am sure the real work is going into GPT5 or whatever it gets called. While all the others are catching up and in some cases being slightly better, I wouldn't be surprised to see a rather large leap back into the lead from OpenAI pretty soon and then a scrabble for some time for others to get close again. We will really see who has the momentum soon, when we see OpenAI's next full release.
tmikaeld 2y ago
Hm, how much VRAM would this take to run?
- asadm 2y ago
  yes
- mrtranscendence 2y ago
  Back of the envelope, maybe 0.75TB? More than you have, probably ...
- bearjaws 2y ago
  My guess is around 256GiB but it depends on what level of quantization you are okay with. At full 16bit it will be massive, near 512GiB. I figure we will see some Q4's that can probably fit on 4 4090s with CPU offloading.
  - sp332 2y ago
    With 400 billion parameters and 8 bits per parameter, wouldn't it be ~400 GB? Plus context size which could be quite large.
    - yalok 2y ago
      he said "Q4" - meaning 4-bit weights.
      - sp332 2y ago
        Ok but at 16-bit it would be 800GB+, right? Not 512.
        reactordev 2y ago
        Divide not multiply. If a size is estimated in 8-bit, reducing to 4-bit halves the size (and entropy of each value). Difference between INT_MAX and SHORT_MAX (assuming you have such defs). I could be wrong too but that’s my understanding. Like float vs half-float.
mdeeks 2y ago
Which specific GPT-4 model is this? gpt-4-0613? gpt-4-0125-preview?
- YetAnotherNick 2y ago
  This is mostly from technical report from OpenAI[1]. API performs better as I said in my previous comment. API models(0613/0125 etc.) also uses user data for training which could leak the benchmark data. [1]: https://arxiv.org/pdf/2303.08774.pdf https://arxiv.org/pdf/2303.08774.pdf
  - pama 2y ago
    IIRC this model had finished pretraining in the summer of 2022.
tedsanders 2y ago
Those numbers are for the original GPT-4 (Mar 2023). Current GPT-4-Turbo (Apr 2024) is better: Llama 3 GPT-4 GPT-4-Turbo* (Apr 2024) MMLU 86.1 86.4 86.7 DROP 83.5 80.9 86.0 MATH 57.8 52.9 73.4 HumEv 84.1 74.4 88.2 *using API prompt: https://github.com/openai/simple-evals https://github.com/openai/simple-evals
- natrys 2y ago
  I find it somewhat interesting that there is a common perception about GPT-4 at release being actually smart, but that it got gradually nerfed for speed with turbo, which is better tuned but doesn't exhibit intelligence like the original. There were times when I felt that too, but nowadays I predominantly use turbo. It's probably because turbo is faster and cheaper, but in lmsys turbo has 100 elo higher than original, so by and large people simply find turbo to be....better? Nevertheless, I do wonder if not just in benchmarks but in how people use LLMs, intelligence is somewhat under utilised, or possibly offset by other qualities.
  - thelittleone 2y ago
    Have you tried Claude 3 Opus? I've been using that predominantly since release and find it's "smarts" as or better than my experience with GPT-4 (pre turbo).
    - natrys 2y ago
      I did. It definitely exudes more all around personality. Unfortunately in my private test suite (mostly about coding), it did somewhat worse than turbo or phind 70b. Since price influences my calculus, I can't say this for sure, but it seems being slightly smarter is not much of an edge, because it's still dumb by human standards. For most non-coding use the smart doesn't make much difference (like summarisation), I find that cheaper options like mistral-large do just as good as Opus. In the last month I have used Command R+ more and more. Finally had some excuse to write some function calling stuff. I have also been highly impressed by Gemini Pro 1.5 finding technical answers from a dense 650 page pdf manual. I have enjoyed chatting with the WizardLM2 fine-tune for the past few days. Somehow I haven't quite found a consistent use case for Opus.

modeless 2y ago

Not quite there yet, but very close and not done training! It's quite plausible that this model could be state of the art over GPT-4 in some domains when it finishes training, unless GPT-5 comes out first. Although 400B will be pretty much out of reach for any PC to run locally, it will still be exciting to have a GPT-4 level model in the open for research so people can try quantizing, pruning, distilling, and other ways of making it more practical to run. And I'm sure startups will build on it as well.

int_19h 2y ago
The real question will be, how much you can quantize that while still retaining sanity. 400b at 2-bit would be possible to run on a Mac Studio - probably at multiple seconds per token, but sometimes that's "fast enough".
- modeless 2y ago
  Yes. I expect an explosion of research and experimentation in model compression. The good news is I think there are tons of avenues that have barely been explored at all. We are at the very beginning of understanding this stuff, and my bet is that in a few years we'll be able to compress these models 10x or more.
brandall10 2y ago
There are rumors about an upcoming M3 or M4 Extreme chip... which would certainly have enough RAM, and probably a 1600-2000 GB/s bandwidth. Still wouldn't be super performant AFA token gen, ~4-6 per second, but certainly runnable. Of course by the time that lands in 6-12 months we'll probably have a 70-100G model that is similarly performant.

jug 2y ago

This is tantalizingly close in multiple benchmarks though. Pretty sure this one will finally be the open GPT-4 match.