4 ms·

We absolutely hammered the MI50 in internal testing for ages. Was solid as far as I can tell. Rocm is sensitive to matching kernel version to driver version to

We absolutely hammered the MI50 in internal testing for ages. Was solid as far as I can tell. Rocm is sensitive to matching kernel version to driver version to userspace version. Staying very much on the kernel version from a official release and using the corresponding driver is drastically more robust than optimistically mixing different components. In particular, rocm is released and tested as one large blob, and running that large blob on a slightly different kernel version can go very badly. Mixing things from GitHub with things from your package manager is also optimistic. Imagine it as huge ball of code where cross version compatibility of pieces is totally untested.

tucnak 3y ago
I would run simple llama.cpp batch jobs for 10 minutes when it would suddenly fail, and require a restart. Random VM_L2_PROTECTION_FAULT in dmesg, something having to do with doorbells. I did report this, never heard back from them.
- latchkey 3y ago
  Would you like to share the model of GPU and versions of various software used? George has a nice explanation of doorbells: https://youtu.be/AqPIOtUkxNo?feature=shared&t=968 https://youtu.be/AqPIOtUkxNo?feature=shared&t=968
- FeepingCreature 3y ago
  Same here with SD on 7900XTX. Most of the time for me it's sufficient to reset the card with rocm-smi --gpureset -d 0.
  - michaelt 3y ago
    Only "most of the time" ? :( You'd hope at $15,000+ per unit, you wouldn't have to reset it at all...
    - JonChesterfield 3y ago
      It's $1000 per, no? This is one of the gaming cards.
      - FeepingCreature 3y ago
        Yep, bought for $1000. At which price point to be honest, it still shouldn't be needed. AMD are lucky everyone expects this nowadays, or people might consider sueing.
- JonChesterfield 3y ago
  Did you run on the blessed Ubuntu version with the blessed kernel version and the blessed driver version? As otherwise you really are in a development branch. If you can point me to a repro I'll add it to my todo list. You can probably tag me in the github issue if that's where you reported it.
  - Aissen 3y ago
    > blessed Ubuntu version with the blessed kernel version To an SRE, this is a nightmare to read. Cuda is bad in this regard (can often prevent major kernel version updates), but this is worse.
    - latchkey 3y ago
      I feel like this goes both ways. You also don't want to have to run bleeding edge for everything because there are so many bugs in things. You kind of want known stable versions to at least base yourself off of.
  - FeepingCreature 3y ago
    At least the one I run into, which also says stuff with L2 and doorbells, is https://github.com/ROCm/ROCm/issues/2196 https://github.com/ROCm/ROCm/issues/2196 fwiw.

tucnak 3y ago

I would run simple llama.cpp batch jobs for 10 minutes when it would suddenly fail, and require a restart. Random VM_L2_PROTECTION_FAULT in dmesg, something having to do with doorbells. I did report this, never heard back from them.

latchkey 3y ago
Would you like to share the model of GPU and versions of various software used? George has a nice explanation of doorbells: https://youtu.be/AqPIOtUkxNo?feature=shared&t=968 https://youtu.be/AqPIOtUkxNo?feature=shared&t=968
FeepingCreature 3y ago
Same here with SD on 7900XTX. Most of the time for me it's sufficient to reset the card with rocm-smi --gpureset -d 0.
- michaelt 3y ago
  Only "most of the time" ? :( You'd hope at $15,000+ per unit, you wouldn't have to reset it at all...
  - JonChesterfield 3y ago
    It's $1000 per, no? This is one of the gaming cards.
    - FeepingCreature 3y ago
      Yep, bought for $1000. At which price point to be honest, it still shouldn't be needed. AMD are lucky everyone expects this nowadays, or people might consider sueing.
JonChesterfield 3y ago
Did you run on the blessed Ubuntu version with the blessed kernel version and the blessed driver version? As otherwise you really are in a development branch. If you can point me to a repro I'll add it to my todo list. You can probably tag me in the github issue if that's where you reported it.
- Aissen 3y ago
  > blessed Ubuntu version with the blessed kernel version To an SRE, this is a nightmare to read. Cuda is bad in this regard (can often prevent major kernel version updates), but this is worse.
  - latchkey 3y ago
    I feel like this goes both ways. You also don't want to have to run bleeding edge for everything because there are so many bugs in things. You kind of want known stable versions to at least base yourself off of.
- FeepingCreature 3y ago
  At least the one I run into, which also says stuff with L2 and doorbells, is https://github.com/ROCm/ROCm/issues/2196 https://github.com/ROCm/ROCm/issues/2196 fwiw.