2 ms·

So when can I buy MI350s for my homelab?

touisteur 2mo ago
Hopefully the MI350P is available soon, at last a standard PCIe SKU, if a bit too-much previous-generation and gimped compared to the MI350X
- lostmsu 2mo ago
  MI350 happens to not fit DeepSeek V4 Flash (144GB vs 180+ needed) so it will be dead on arrival.
  - boroboro4 2mo ago
    For production loads? No one will run DS without expert parallelism so it’s not important for model to fit on one gpu.
    - smallerize 2mo ago
      Do you mean people will run multiple GPUs, or that streaming a small % of experts from disk won't completely kill performance?
      - touisteur 2mo ago
        Time for AMD to improve their directstorage story I guess. Is there a good writeup on how one would use it there ? Same for gpudirect (more useful for scale-out or training). Everyone is focused on AI but these are two interesting techs trickling down from the NVIDIA tree, relatively "easy" to use there, which maybe exist in AMD world but I somehow missed the docs and APIs and demos on how to use them...
  - tpurves 2mo ago
    This thing has 50% more memory than GTX PRO 6000, which fetch 18k-25k+ each right now on newegg, and generally sold out. If nvdia can still sell out 96GB PCI cards (on a 2 year old architecture) at those prices, how is an AMD equivalent with better specs dead on arrival?
    - lostmsu 2mo ago
      Market takes time to catch up. When released Qwen 3.6 had a lot of sense and it did fit, but frontier since went too far ahead. Now you'd want DS V4 Flash from yesterday, or you are too far behind. Unless MI350P will be much cheaper than 6000 Pro, which is unlikely given its VRAM, it will lose to it, because you'd need 2 of either.
      - touisteur 2mo ago
        Don't see how it could be cheaper with HBM than the RTX 6000 Blackwell Server Pro but in these times of relative shortage any additionnal supply should be slurped ? I was remarking on the MI350P because I've had a hard time procuring "small" CDNAx systems (for e.g. development, experiments and lower-profile servers) and OAM seemed very niche (not if you're aiming for density and training/inference...). I hope the MI350P fills a lower part of the spectrum and I can start massively porting CUDA stuff or at least work on HIP and ROCm and what I need to make most or some of our CUDA stuff run on AMD HW, then how make it run fast.

Hopefully the MI350P is available soon, at last a standard PCIe SKU, if a bit too-much previous-generation and gimped compared to the MI350X

lostmsu 2mo ago
MI350 happens to not fit DeepSeek V4 Flash (144GB vs 180+ needed) so it will be dead on arrival.
- boroboro4 2mo ago
  For production loads? No one will run DS without expert parallelism so it’s not important for model to fit on one gpu.
  - smallerize 2mo ago
    Do you mean people will run multiple GPUs, or that streaming a small % of experts from disk won't completely kill performance?
    - touisteur 2mo ago
      Time for AMD to improve their directstorage story I guess. Is there a good writeup on how one would use it there ? Same for gpudirect (more useful for scale-out or training). Everyone is focused on AI but these are two interesting techs trickling down from the NVIDIA tree, relatively "easy" to use there, which maybe exist in AMD world but I somehow missed the docs and APIs and demos on how to use them...
- tpurves 2mo ago
  This thing has 50% more memory than GTX PRO 6000, which fetch 18k-25k+ each right now on newegg, and generally sold out. If nvdia can still sell out 96GB PCI cards (on a 2 year old architecture) at those prices, how is an AMD equivalent with better specs dead on arrival?
  - lostmsu 2mo ago
    Market takes time to catch up. When released Qwen 3.6 had a lot of sense and it did fit, but frontier since went too far ahead. Now you'd want DS V4 Flash from yesterday, or you are too far behind. Unless MI350P will be much cheaper than 6000 Pro, which is unlikely given its VRAM, it will lose to it, because you'd need 2 of either.
    - touisteur 2mo ago
      Don't see how it could be cheaper with HBM than the RTX 6000 Blackwell Server Pro but in these times of relative shortage any additionnal supply should be slurped ? I was remarking on the MI350P because I've had a hard time procuring "small" CDNAx systems (for e.g. development, experiments and lower-profile servers) and OAM seemed very niche (not if you're aiming for density and training/inference...). I hope the MI350P fills a lower part of the spectrum and I can start massively porting CUDA stuff or at least work on HIP and ROCm and what I need to make most or some of our CUDA stuff run on AMD HW, then how make it run fast.