4 ms·

that or if you put chips in the hands of your customers, they may start to benchmark it against other equivalent solutions

by htrp 3y ago

latchkey 3y ago
Funny you should mention that. ;-) https://www.reddit.com/r/LocalLLaMA/comments/1bpgrdf/wanted_amd_mi300x_benchmarks/ https://www.reddit.com/r/LocalLLaMA/comments/1bpgrdf/wanted_... I've got about a dozen people signed up. Just working through some hardware issues right now (see above about high rate of failures), and hope to have this resolved next week, so that I can get people onto them and doing their testing.
- htrp 3y ago
  Shiny! You have you guys gotten MI300X's working for non-inference use cases?
  - latchkey 3y ago
    We got them, then gave them to a customer to use for a week, got them back and now we are having some hardware issues that we are in the process of sorting out. I literally haven't had more than a few hours time on them yet. They _should_ work fine for both training and inference, but since nobody has done much in the way of public in-depth benchmarks yet... I was hoping to get people to do it for us in order to stay as unbiased as possible. noticing now: strange that my previous comment was downvoted. Would be nice to understand what someone didn't like about what I said!
    - lostmsu 3y ago
      That sounds like your machine broke after 1 week of intensive use.
      - latchkey 3y ago
        Sadly, not even intensive. The thing that nobody talks about is that there is a high rate of failures on this high end equipment. I've heard as high as 20%, in the first month. I'm not even talking about AMD here. If anyone thinks they can just buy some accelerators and throw them into a rack and expect them to work flawlessly... they've got some hard lessons to learn. This will be less of an issue as we grow as we will have plenty of stock to pull from, but it is a real bummer as we are starting as a proof of concept first. We started working on this business last August, before anyone knew whether or not AMD would even change course on AI. The good news is that we onboarded a customer the day that we announced our availability, we passed that PoC challenge with flying colors and closed significant additional funding immediately after that. Onwards and upwards, just have to roll with the punches.
        touisteur 3y ago
        Ooooh yeah, from experience, even the usual (thorough) suspects who over-over-over-spec cooling and power (as anyone here has heard an HPE Apollo 6500 at full fan speed can attest) have a hard time getting all the interconnects, pcie, firmware stuff up to snuff and running H24. Once it's setup it's amazing but the bringup can be rocky. And I'm not even talking 100/400G network, wonderful wonderful hardware, good luck debugging and getting all the RoCE/RDMA/GPUDirect/StorageDirect/NCCL working (already a bit of pain on nvidia, with a large installed base...). Either you want to learn all this stuff (for reasons) or you're dumping a lot of money on fast-evolving tech.

that or if you put chips in the hands of your customers, they may start to benchmark it against other equivalent solutions

by htrp 3y ago

that or if you put chips in the hands of your customers, they may start to benchmark it against other equivalent solutions

latchkey 3y ago

Funny you should mention that. ;-) https://www.reddit.com/r/LocalLLaMA/comments/1bpgrdf/wanted_amd_mi300x_benchmarks/ https://www.reddit.com/r/LocalLLaMA/comments/1bpgrdf/wanted_... I've got about a dozen people signed up. Just working through some hardware issues right now (see above about high rate of failures), and hope to have this resolved next week, so that I can get people onto them and doing their testing.

htrp 3y ago
Shiny! You have you guys gotten MI300X's working for non-inference use cases?
- latchkey 3y ago
  We got them, then gave them to a customer to use for a week, got them back and now we are having some hardware issues that we are in the process of sorting out. I literally haven't had more than a few hours time on them yet. They _should_ work fine for both training and inference, but since nobody has done much in the way of public in-depth benchmarks yet... I was hoping to get people to do it for us in order to stay as unbiased as possible. noticing now: strange that my previous comment was downvoted. Would be nice to understand what someone didn't like about what I said!
  - lostmsu 3y ago
    That sounds like your machine broke after 1 week of intensive use.
    - latchkey 3y ago
      Sadly, not even intensive. The thing that nobody talks about is that there is a high rate of failures on this high end equipment. I've heard as high as 20%, in the first month. I'm not even talking about AMD here. If anyone thinks they can just buy some accelerators and throw them into a rack and expect them to work flawlessly... they've got some hard lessons to learn. This will be less of an issue as we grow as we will have plenty of stock to pull from, but it is a real bummer as we are starting as a proof of concept first. We started working on this business last August, before anyone knew whether or not AMD would even change course on AI. The good news is that we onboarded a customer the day that we announced our availability, we passed that PoC challenge with flying colors and closed significant additional funding immediately after that. Onwards and upwards, just have to roll with the punches.
      - touisteur 3y ago
        Ooooh yeah, from experience, even the usual (thorough) suspects who over-over-over-spec cooling and power (as anyone here has heard an HPE Apollo 6500 at full fan speed can attest) have a hard time getting all the interconnects, pcie, firmware stuff up to snuff and running H24. Once it's setup it's amazing but the bringup can be rocky. And I'm not even talking 100/400G network, wonderful wonderful hardware, good luck debugging and getting all the RoCE/RDMA/GPUDirect/StorageDirect/NCCL working (already a bit of pain on nvidia, with a large installed base...). Either you want to learn all this stuff (for reasons) or you're dumping a lot of money on fast-evolving tech.