4 ms·

> Be advised though, stay as far away as possible from the K80 - the drivers were one of the most painful tech things I've ever had to endure, even if 24GB of V

> Be advised though, stay as far away as possible from the K80 - the drivers were one of the most painful tech things I've ever had to endure, even if 24GB of VRAM for 50 bucks sounds incredibly appealing. I thought the problem was that those cards have loads of RAM but lack really important compute capabilities such that they're kind of useless for actually running AI workloads on. Is that not the case?

almostgotcaught 2y ago
> Is that not the case? it is - they're laughably slow and not even supported by latest CUDA > NVIDIA Driver support for Kepler is removed beginning with R495. CUDA Toolkit development support for Kepler continues through CUDA 11.x.
- GTP 2y ago
  But Deepseek R1 doesn't use CUDA, so maybe for this specific case, it isn't a big deal?
  - almostgotcaught 2y ago
    > it isn't a big deal? friend you shouldn't make comments like this unless you understand the definitions of the words. Deepseek wrote some parts of their kernels using PTX. newsflash: PTX support for features is lockstep with CUDA support for the same features ie the fact that CUDA doesn't support it means you couldn't write the PTX to use those features either.
    - therealfiona 2y ago
      It is poor form to condemn someone from asking a question. Thank you for providing the information to clear up ignorance though.
      - almostgotcaught 2y ago
        this is a question: > is deepseak's use of PTX instead of CUDA relevant here? this is a conclusion/assumption thinly veiled as a question > Deepseek R1 doesn't use CUDA, so ... it isn't a big deal? note, genuine questions don't already presuppose an answer.
        GTP 2y ago
        Asking if it is a big deal or not is definitely a question ;) Thank you for providing the information I was missing though.
  - numpad0 2y ago
    The PTX hack is for backend runner and training infra, the public weights are often executed using existing backends. Especially R1-distill-* models are.
    - almostgotcaught 2y ago
      the two things (weights and kernels) have nothing to do with each other in the slightest. again i wish people would take a beat before commenting out of their depth and consider whether their comment adds to the conversation or not.
TrueDuality 2y ago
I'm running P41s in one of my test boxes. These don't have support for BF16 but they do support F16 and F32 and those are accelerated to a certain degree, they're lacking kernels that are as optimized but its not terribly hard to adapt other ones for the purposes. You don't get great out-of-the-box performance but it only took me three work days or so with no experience writing these to adapt, test, and validate a kernel using the acceleration hardware that was available (no prior experience writing these kernels). They're not as powerful as others but still significantly better than running on a CPU alone and I'd bet my kernel is missing more advanced optimizations. My issue with these was the power cable and fans. The author touches on the fans and I did try a 3D printed shroud and some of the higher pressure fans but I could only run the cards in short stints. I ended up making an enclosure that went straight out of the case using two high pressure SAN array fans I harvested from the IT graveyard per card and making a hole with an angle grinder. The power cable is NOT STANDARD on these. I had to find a weird specific cable to adapt the standard 8-pin GPU connector and each card takes two of these bad boys.

almostgotcaught 2y ago

> Is that not the case? it is - they're laughably slow and not even supported by latest CUDA > NVIDIA Driver support for Kepler is removed beginning with R495. CUDA Toolkit development support for Kepler continues through CUDA 11.x.

GTP 2y ago
But Deepseek R1 doesn't use CUDA, so maybe for this specific case, it isn't a big deal?
- almostgotcaught 2y ago
  > it isn't a big deal? friend you shouldn't make comments like this unless you understand the definitions of the words. Deepseek wrote some parts of their kernels using PTX. newsflash: PTX support for features is lockstep with CUDA support for the same features ie the fact that CUDA doesn't support it means you couldn't write the PTX to use those features either.
  - therealfiona 2y ago
    It is poor form to condemn someone from asking a question. Thank you for providing the information to clear up ignorance though.
    - almostgotcaught 2y ago
      this is a question: > is deepseak's use of PTX instead of CUDA relevant here? this is a conclusion/assumption thinly veiled as a question > Deepseek R1 doesn't use CUDA, so ... it isn't a big deal? note, genuine questions don't already presuppose an answer.
      - GTP 2y ago
        Asking if it is a big deal or not is definitely a question ;) Thank you for providing the information I was missing though.
- numpad0 2y ago
  The PTX hack is for backend runner and training infra, the public weights are often executed using existing backends. Especially R1-distill-* models are.
  - almostgotcaught 2y ago
    the two things (weights and kernels) have nothing to do with each other in the slightest. again i wish people would take a beat before commenting out of their depth and consider whether their comment adds to the conversation or not.

TrueDuality 2y ago

I'm running P41s in one of my test boxes. These don't have support for BF16 but they do support F16 and F32 and those are accelerated to a certain degree, they're lacking kernels that are as optimized but its not terribly hard to adapt other ones for the purposes. You don't get great out-of-the-box performance but it only took me three work days or so with no experience writing these to adapt, test, and validate a kernel using the acceleration hardware that was available (no prior experience writing these kernels). They're not as powerful as others but still significantly better than running on a CPU alone and I'd bet my kernel is missing more advanced optimizations. My issue with these was the power cable and fans. The author touches on the fans and I did try a 3D printed shroud and some of the higher pressure fans but I could only run the cards in short stints. I ended up making an enclosure that went straight out of the case using two high pressure SAN array fans I harvested from the IT graveyard per card and making a hole with an angle grinder. The power cable is NOT STANDARD on these. I had to find a weird specific cable to adapt the standard 8-pin GPU connector and each card takes two of these bad boys.