3 ms·

How will a 5090 compare against project digits? now that they're both in the front page :)

by lz400 2y ago

IiOmegaI 2y ago
Don't forget that you can link for example two 'Digits' together (~256 GB) if you want to run even larger models or have larger context size. That is 2x$3000 vs 8x$2000. This will make it possible for you to run models up to 405B parameters, like Llama 3.1 405B at 4bit quant or the Grok-1 314B at 6bit quant. Who knows, maybe some better models will be released in the future which are better optimized and won't need that much RAM, but it is easier to buy a second 'Digits' in comparison to building a rack with 8xGPUs. For example, if you look at the latest Llama models, Meta states: 'Llama 3.3 70B approaches the performance of Llama 3.1 405B'. To interfere with Llama3.3-70B-Instruct with ~8k context length (without offloading), you'd need: - Q4 (~44GB): 2x5090; 1x 'Digits' - Q6 (~58GB): 2x5090; 1x 'Digits' - Q8 (~74GB): 3x5090; 1x 'Digits' - FP16 (~144GB): 5x5090; 2x 'Digits' Let's wait and see which bandwidth it will have.
- lz400 2y ago
  > bandwidth Speculation has it at ~5XXgb/s. agreed on the memory. if I can I'll get a few but I fear they'll sell out immediately
ryao 2y ago
We will not really know until memory bandwidth and compute numbers are published. However, Project Digits seems like a successor to the NVIDIA Jetson AGX Orin 64GB Developer Kit, which was based on the Ampere architecture and has 204.8GB/sec memory bandwidth: https://www.okdo.com/wp-content/uploads/2023/03/jetson-agx-orin-64GB-developer-kit-datasheet-web-us.pdf https://www.okdo.com/wp-content/uploads/2023/03/jetson-agx-o... The 3090 Ti had about 5 times the memory bandwidth and 5 times the compute capability. If that ratio holds for blackwell, the 5090 will run circles around it when it has enough VRAM (or you have enough 5090 cards to fit everything into VRAM).
- lz400 2y ago
  Very interesting, thanks! 32gb for the 5090 vs 128gb for digits might put a nasty cap on unleashing all that power for interesting models. Several 5090s together would work but then we're talking about multiple times the cost (4x$2000+PC VS $3000)
  - ryao 2y ago
    Inference presumably will run faster on a 5090. If the 5x memory bandwidth figure holds, then token generation would run 5 times faster. That said, people in the digits discussion predict that the memory bandwidth will be closer to 546GB/sec, which is closer to 1/3 the memory bandwidth of the 5090, so a bunch of 5090 cards would only run 3 times faster at token generation.

How will a 5090 compare against project digits? now that they're both in the front page :)

by lz400 2y ago

How will a 5090 compare against project digits? now that they're both in the front page :)

Don't forget that you can link for example two 'Digits' together (~256 GB) if you want to run even larger models or have larger context size. That is 2x$3000 vs 8x$2000. This will make it possible for you to run models up to 405B parameters, like Llama 3.1 405B at 4bit quant or the Grok-1 314B at 6bit quant. Who knows, maybe some better models will be released in the future which are better optimized and won't need that much RAM, but it is easier to buy a second 'Digits' in comparison to building a rack with 8xGPUs. For example, if you look at the latest Llama models, Meta states: 'Llama 3.3 70B approaches the performance of Llama 3.1 405B'. To interfere with Llama3.3-70B-Instruct with ~8k context length (without offloading), you'd need: - Q4 (~44GB): 2x5090; 1x 'Digits' - Q6 (~58GB): 2x5090; 1x 'Digits' - Q8 (~74GB): 3x5090; 1x 'Digits' - FP16 (~144GB): 5x5090; 2x 'Digits' Let's wait and see which bandwidth it will have.

lz400 2y ago
> bandwidth Speculation has it at ~5XXgb/s. agreed on the memory. if I can I'll get a few but I fear they'll sell out immediately

ryao 2y ago

We will not really know until memory bandwidth and compute numbers are published. However, Project Digits seems like a successor to the NVIDIA Jetson AGX Orin 64GB Developer Kit, which was based on the Ampere architecture and has 204.8GB/sec memory bandwidth: https://www.okdo.com/wp-content/uploads/2023/03/jetson-agx-orin-64GB-developer-kit-datasheet-web-us.pdf https://www.okdo.com/wp-content/uploads/2023/03/jetson-agx-o... The 3090 Ti had about 5 times the memory bandwidth and 5 times the compute capability. If that ratio holds for blackwell, the 5090 will run circles around it when it has enough VRAM (or you have enough 5090 cards to fit everything into VRAM).

lz400 2y ago
Very interesting, thanks! 32gb for the 5090 vs 128gb for digits might put a nasty cap on unleashing all that power for interesting models. Several 5090s together would work but then we're talking about multiple times the cost (4x$2000+PC VS $3000)
- ryao 2y ago
  Inference presumably will run faster on a 5090. If the 5x memory bandwidth figure holds, then token generation would run 5 times faster. That said, people in the digits discussion predict that the memory bandwidth will be closer to 546GB/sec, which is closer to 1/3 the memory bandwidth of the 5090, so a bunch of 5090 cards would only run 3 times faster at token generation.