slimesli

3 ms·

1. 1.▲
   by slimesli 2y ago

   for llama 3-8b (smaller model) you need 8GB/16GB of memory: minimum/recommended || for llama 3-70b (larger model) you need 64GB/96GB of memory what kind of memory? there are 3 types of memory: GPU memory: medium capacity | h
2. 2.▲
   by slimesli 3y ago

   It's not a fundamental limit. Google palm had 540B parameters as dense model. But it's a practical limit because models with over 1T would be extremely slow even on newest gpus. Even now, OpenAI has limit of 25 messages. You can r
3. 3.▲
   by slimesli 3y ago

   Because of memory bandwidth. H100 has 3350gB/s of bandwidth, more gpus will give you more memory but not bandwidth. If you load 175b parameters in 8bit then you can get theoretically 3350/175=19 tokens/second. In MoE you nee