4 ms·

X.ai's Grok-1 Model Is Officially Open-Source and Larger Than Expected

22 points by RafelMri 3y ago

jejones3141 3y ago
What kind of hardware would one need to use the model with reasonable performance?
- trillic 3y ago
  8xH100s to keep it all in RAM from my uneducated view.
- nirav72 3y ago
  I wouldn’t bother. Probably should wait until someone releases an optimized version.
- deleted 3y ago
  [deleted]
- superkuh 3y ago
  Depends on how many experts are active in any given pass. If it's a 10 expert mix of 33B experts (grok-0 is 33B, grok-1 is ~314B which is ~10x) and only runs two of them (like Mixtral's 2/8) then it'd have about the same inference requirements as a 70B model (2*33=66B). So if this was quantized using ~4 bits per parameter you'd need ~40GB of vram. So you could spread it across 2x 3090 24GB using llama.cpp.
  - dragonwriter 3y ago
    MoE has the same “loading” RAM requirements as any other model with the same total parameters (not just for the fixed portion plus whatever experts are activated at any one time) because it has to load all the parameters. The additional needed because of context may be lower (not sure), but the big difference is that it has much better inference speed (and, as a result, can be tolerable with layers split between VRAM and system RAM where a similarly-sized non-MoE model would not.) > So if this was quantized using ~4 bits per parameter you’d need ~40GB of vram. No, Mixtral 8x7B (which is a total of 45 billion parameters, because there is a shared portion of the 7B, so its not 56 billion) at 4-bit quantization takes ~29GB [0]. A 314B model is ~7 times as large; with a similar architecture its not going to take only another 1/3 as much RAM. [0] https://huggingface.co/TheBloke/Mixtral-8x7B-v0.1-GGUF https://huggingface.co/TheBloke/Mixtral-8x7B-v0.1-GGUF
    - superkuh 3y ago
      You're absolutely right. I don't know what I was thinking.
- zaptrem 3y ago
  I think rule of thumb is 1GB VRAM per 1 billion params quantized to FP8.
  - dragonwriter 3y ago
    Just to load the model without actually running it requires 1GB of whatever RAM it is loading and running in (could be VRAM, system RAM, or a combination, with different performance characteristics for each option) per billion parameters at 8-bit quantization. Though models often are usefully run at 4-5 bit quantization, which saves half (or nearly so) of that. You also need additional RAM that increases as some function of context size (not sure what function, and ISTR there are big-O differences between architectures in how it varies) to actually do inference.
mdaniel 3y ago
the submission that doesn't have a paywall: https://news.ycombinator.com/item?id=39737313 https://news.ycombinator.com/item?id=39737313 or the discussion on the code itself: https://news.ycombinator.com/item?id=39737281 https://news.ycombinator.com/item?id=39737281 TFA: https://x.ai/blog/grok-os https://x.ai/blog/grok-os TFC: https://github.com/xai-org/grok-1#readme https://github.com/xai-org/grok-1#readme
teruakohatu 3y ago
From the official release page [1]: > We are releasing the base model weights and network architecture of Grok-1, our large language model. Grok-1 is a 314 billion parameter Mixture-of-Experts model trained from scratch by xAI. > This is the raw base model checkpoint from the Grok-1 pre-training phase, which concluded in October 2023. This means that the model is not fine-tuned for any specific application, such as dialogue. > We are releasing the weights and the architecture under the Apache 2.0 license. > To get started with using the model, follow the instructions at github.com/xai-org/grok. A little disappointing they are not releasing the weights for the Grok-1 finetuned model. [1] https://x.ai/blog/grok-os https://x.ai/blog/grok-os
numpad0 3y ago
dupe, from 6 hours ago: https://news.ycombinator.com/item?id=39737281 https://news.ycombinator.com/item?id=39737281
threeseed 3y ago
> OpenAI CEO Sam Altman with the express objective of creating open-source AI models Not aware of OpenAI at any point saying they would open-source their models. Only that researchers would be encouraged to share their work with the world: https://web.archive.org/web/20190224031626/https://blog.openai.com/introducing-openai/ https://web.archive.org/web/20190224031626/https://blog.open...

22 points by RafelMri 3y ago

jejones3141 3y ago

What kind of hardware would one need to use the model with reasonable performance?

trillic 3y ago
8xH100s to keep it all in RAM from my uneducated view.
nirav72 3y ago
I wouldn’t bother. Probably should wait until someone releases an optimized version.
deleted 3y ago
[deleted]
superkuh 3y ago
Depends on how many experts are active in any given pass. If it's a 10 expert mix of 33B experts (grok-0 is 33B, grok-1 is ~314B which is ~10x) and only runs two of them (like Mixtral's 2/8) then it'd have about the same inference requirements as a 70B model (2*33=66B). So if this was quantized using ~4 bits per parameter you'd need ~40GB of vram. So you could spread it across 2x 3090 24GB using llama.cpp.
- dragonwriter 3y ago
  MoE has the same “loading” RAM requirements as any other model with the same total parameters (not just for the fixed portion plus whatever experts are activated at any one time) because it has to load all the parameters. The additional needed because of context may be lower (not sure), but the big difference is that it has much better inference speed (and, as a result, can be tolerable with layers split between VRAM and system RAM where a similarly-sized non-MoE model would not.) > So if this was quantized using ~4 bits per parameter you’d need ~40GB of vram. No, Mixtral 8x7B (which is a total of 45 billion parameters, because there is a shared portion of the 7B, so its not 56 billion) at 4-bit quantization takes ~29GB [0]. A 314B model is ~7 times as large; with a similar architecture its not going to take only another 1/3 as much RAM. [0] https://huggingface.co/TheBloke/Mixtral-8x7B-v0.1-GGUF https://huggingface.co/TheBloke/Mixtral-8x7B-v0.1-GGUF
  - superkuh 3y ago
    You're absolutely right. I don't know what I was thinking.
zaptrem 3y ago
I think rule of thumb is 1GB VRAM per 1 billion params quantized to FP8.
- dragonwriter 3y ago
  Just to load the model without actually running it requires 1GB of whatever RAM it is loading and running in (could be VRAM, system RAM, or a combination, with different performance characteristics for each option) per billion parameters at 8-bit quantization. Though models often are usefully run at 4-5 bit quantization, which saves half (or nearly so) of that. You also need additional RAM that increases as some function of context size (not sure what function, and ISTR there are big-O differences between architectures in how it varies) to actually do inference.

mdaniel 3y ago

the submission that doesn't have a paywall: https://news.ycombinator.com/item?id=39737313 https://news.ycombinator.com/item?id=39737313 or the discussion on the code itself: https://news.ycombinator.com/item?id=39737281 https://news.ycombinator.com/item?id=39737281 TFA: https://x.ai/blog/grok-os https://x.ai/blog/grok-os TFC: https://github.com/xai-org/grok-1#readme https://github.com/xai-org/grok-1#readme

teruakohatu 3y ago

From the official release page [1]: > We are releasing the base model weights and network architecture of Grok-1, our large language model. Grok-1 is a 314 billion parameter Mixture-of-Experts model trained from scratch by xAI. > This is the raw base model checkpoint from the Grok-1 pre-training phase, which concluded in October 2023. This means that the model is not fine-tuned for any specific application, such as dialogue. > We are releasing the weights and the architecture under the Apache 2.0 license. > To get started with using the model, follow the instructions at github.com/xai-org/grok. A little disappointing they are not releasing the weights for the Grok-1 finetuned model. [1] https://x.ai/blog/grok-os https://x.ai/blog/grok-os

numpad0 3y ago

dupe, from 6 hours ago: https://news.ycombinator.com/item?id=39737281 https://news.ycombinator.com/item?id=39737281

threeseed 3y ago

> OpenAI CEO Sam Altman with the express objective of creating open-source AI models Not aware of OpenAI at any point saying they would open-source their models. Only that researchers would be encouraged to share their work with the world: https://web.archive.org/web/20190224031626/https://blog.openai.com/introducing-openai/ https://web.archive.org/web/20190224031626/https://blog.open...