4 ms·

Nvidia DGX Spark

15 points by tanelpoder 2y ago

deleted 2y ago
[deleted]
andrewSC 2y ago
Am I missing something here or is inference going to be painful given the "low" memory bandwidth compared to, say, HBM2E?
- dragonwriter 2y ago
  Have you seen anything with 128GB of HBM2E at anywhere near the DGX Spark's $3,000 price point?
  - andrewSC 2y ago
    I honestly can't say I have however, that doesn't mean it couldn't physically exist/happen? Perhaps a "little" more cost but I'd be willing to bet people would gladly pay the premium for such a device. I'm also very curious to know what the BOM for an A100 actually is as well as HBM2E per GB.
  - karmakaze 2y ago
    The AMD Radeon VII has 16GB HBM2 and sold for $700 in 2019. I don't know how that would translate to today's HBM2E's pricing like if its price change follows that of GDDR's.
  - deleted 2y ago
    [deleted]
- tanelpoder 2y ago
  I guess since these devices aren't meant for production throughput, but rather about having enough RAM for local experimentation with large enough models, it's an ok tradeoff at this price point...
- sliken 2y ago
  273GB/sec with good FP4 performance should be fine for developers playing with inference. This isn't the kind of thing that you'd use for inference workloads supporting millions of users. I'd like to see a inference benchmark vs the strix halo, which has better memory bandwidth and costs 2/3rds as much.
moondev 2y ago
"ConnectX-7 Smart NIC" https://resources.nvidia.com/en-us-accelerated-networking-resource-library/connectx-7-datasheet https://resources.nvidia.com/en-us-accelerated-networking-re... So those two QSFP ports can be configured for 400GbE IB and/or RoCE? Also is it fair to say that this can run a desktop considering it has an HDMI port?
- dragonwriter 2y ago
  Yes, the whole point is that it is a compact desktop for AI applications, so, yes, it can run a desktop.
  - moondev 2y ago
    Nvidia DGX and HGX servers packed NVME, 8 H100 GPUs and 8 Infiniband cards don't run a desktop. They don't even have an HDMI port.
    - deleted 2y ago
      [deleted]
vaxman 2y ago
Heard coming from the underground parking area at One Apple Park Way earlier today https://youtu.be/BMm4FFrSxMk https://youtu.be/BMm4FFrSxMk Heard coming from the Taiwan-infused PC clone manufacturer industry https://youtu.be/wIE5wwvicew https://youtu.be/wIE5wwvicew Meanwhile, at whatever planet OpenAI lives on https://youtu.be/615lNmMiOSQ https://youtu.be/615lNmMiOSQ

15 points by tanelpoder 2y ago

deleted 2y ago

[deleted]

Am I missing something here or is inference going to be painful given the "low" memory bandwidth compared to, say, HBM2E?

dragonwriter 2y ago
Have you seen anything with 128GB of HBM2E at anywhere near the DGX Spark's $3,000 price point?
- andrewSC 2y ago
  I honestly can't say I have however, that doesn't mean it couldn't physically exist/happen? Perhaps a "little" more cost but I'd be willing to bet people would gladly pay the premium for such a device. I'm also very curious to know what the BOM for an A100 actually is as well as HBM2E per GB.
- karmakaze 2y ago
  The AMD Radeon VII has 16GB HBM2 and sold for $700 in 2019. I don't know how that would translate to today's HBM2E's pricing like if its price change follows that of GDDR's.
- deleted 2y ago
  [deleted]
tanelpoder 2y ago
I guess since these devices aren't meant for production throughput, but rather about having enough RAM for local experimentation with large enough models, it's an ok tradeoff at this price point...
sliken 2y ago
273GB/sec with good FP4 performance should be fine for developers playing with inference. This isn't the kind of thing that you'd use for inference workloads supporting millions of users. I'd like to see a inference benchmark vs the strix halo, which has better memory bandwidth and costs 2/3rds as much.

moondev 2y ago

"ConnectX-7 Smart NIC" https://resources.nvidia.com/en-us-accelerated-networking-resource-library/connectx-7-datasheet https://resources.nvidia.com/en-us-accelerated-networking-re... So those two QSFP ports can be configured for 400GbE IB and/or RoCE? Also is it fair to say that this can run a desktop considering it has an HDMI port?

dragonwriter 2y ago
Yes, the whole point is that it is a compact desktop for AI applications, so, yes, it can run a desktop.
- moondev 2y ago
  Nvidia DGX and HGX servers packed NVME, 8 H100 GPUs and 8 Infiniband cards don't run a desktop. They don't even have an HDMI port.
  - deleted 2y ago
    [deleted]

vaxman 2y ago

Heard coming from the underground parking area at One Apple Park Way earlier today https://youtu.be/BMm4FFrSxMk https://youtu.be/BMm4FFrSxMk Heard coming from the Taiwan-infused PC clone manufacturer industry https://youtu.be/wIE5wwvicew https://youtu.be/wIE5wwvicew Meanwhile, at whatever planet OpenAI lives on https://youtu.be/615lNmMiOSQ https://youtu.be/615lNmMiOSQ