10 ms·

Summary: This is a company that makes AI accelerator ICs. They reimplemented Chinchilla and released the model weights under a permissive license.

bogwog 4y ago
In other words, they’re actually incentivized to help make LLMs as accessible as possible, rather than try to keep them locked up to hide them from competitors. Which makes me wonder if Nvidia is doing anything with LLMs too?
- vintermann 4y ago
  NVidia has certainly pushing the envelope on image generation. StyleGAN3 was really cool when it came. But it is an issue that their chips are hardly optimized for LLMs.
  - meghan_rain 4y ago
    How can a GPU be optimized for StyleGAN but not LLMs? Serious question.
    - MacsHeadroom 4y ago
      RAM. GPT-3 is over 600GB, ie just the max RAM of 8xA100s, because that's all the hardware can fit. StableDiffusion plus a whole chain of imagenets can make any visual imagery imaginable in 2GB of RAM. Meanwhile 2GB of RAM barely runs a basic tiny text completion NN that can't do anything intelligent. Text requires a lot more parameters (and more memory/RAM) than images.
      - brucethemoose2 4y ago
        The Cerebras node's actual "RAM" (the 40GB of SRAM) is pretty modest too, but being an enormous chip with the networked storage pools is certainly a better situation than a bunch of A100s reaching out to every other A100. Honestly, all the AI ASIC makers drastically underestimated the RAM requirements of future models. Graphcore's 4GB and Tenstorrent's 8GB per IC is kinda laughable, and it takes them longer to adjust than Nvidia. And Cerebras' original pitch was "fit the entire model into SRAM!"
  - ShamelessC 4y ago
    Have you seen the license on StyleGAN3?
- option 4y ago
  obviously yes, there are larger (20B) models trained on more tokens that are released by NVIDIA under commercial-friendly license on HF https://huggingface.co/nvidia/nemo-megatron-gpt-20B https://huggingface.co/nvidia/nemo-megatron-gpt-20B

bogwog 4y ago

In other words, they’re actually incentivized to help make LLMs as accessible as possible, rather than try to keep them locked up to hide them from competitors. Which makes me wonder if Nvidia is doing anything with LLMs too?

vintermann 4y ago
NVidia has certainly pushing the envelope on image generation. StyleGAN3 was really cool when it came. But it is an issue that their chips are hardly optimized for LLMs.
- meghan_rain 4y ago
  How can a GPU be optimized for StyleGAN but not LLMs? Serious question.
  - MacsHeadroom 4y ago
    RAM. GPT-3 is over 600GB, ie just the max RAM of 8xA100s, because that's all the hardware can fit. StableDiffusion plus a whole chain of imagenets can make any visual imagery imaginable in 2GB of RAM. Meanwhile 2GB of RAM barely runs a basic tiny text completion NN that can't do anything intelligent. Text requires a lot more parameters (and more memory/RAM) than images.
    - brucethemoose2 4y ago
      The Cerebras node's actual "RAM" (the 40GB of SRAM) is pretty modest too, but being an enormous chip with the networked storage pools is certainly a better situation than a bunch of A100s reaching out to every other A100. Honestly, all the AI ASIC makers drastically underestimated the RAM requirements of future models. Graphcore's 4GB and Tenstorrent's 8GB per IC is kinda laughable, and it takes them longer to adjust than Nvidia. And Cerebras' original pitch was "fit the entire model into SRAM!"
- ShamelessC 4y ago
  Have you seen the license on StyleGAN3?
option 4y ago
obviously yes, there are larger (20B) models trained on more tokens that are released by NVIDIA under commercial-friendly license on HF https://huggingface.co/nvidia/nemo-megatron-gpt-20B https://huggingface.co/nvidia/nemo-megatron-gpt-20B