3 ms·

Those P100s are so cheap because they're more trouble than they're worth. They don't mount easily in most cases, will consume a lot of power at idle, generate l

Those P100s are so cheap because they're more trouble than they're worth. They don't mount easily in most cases, will consume a lot of power at idle, generate lots of heat without active cooling and require a Linux machine to use properly. It's probably possible to engineer a home-scale inferencing network like that, but I've had plenty of success just running GPT-Neo on off-the-shelf ARM chips. It's not worth the extra $800, exploding power bill and engineering effort to get diminishing returns.

mk_stjames 4y ago
Oh I know for sure, it's still just amusing to me to look up these just slightly older cards and see them going for so little compared to what they were new; I do also look at various benchmarks and comparisons over and over because I would like to build a home gpu server for some machine learning fun and I find it interesting that the TFLOPS/$ numbers always come out very much linear... A card that gives 2x performance costs 2x more.... if cards depreciate 50%, it's because new cards have been launched that have double performance for the same price... etc... Even simpler, the CUDA cores-per-dollar is pretty much a constant. What isn't taken into account there is various optimizations new cards have that speed up training or inference, like better use of the tensor cores. At some point incompatibility with newer CUDA compute levels makes the older cards much less useable.
dzdt 4y ago
Not directly applicable for large language models, I was doing some searching for cards with lots of double-precision calc performance. Nvidia has been leaving that out of all of the consumer versions of hardware; cheap P100's seem to be by far the highest DP FLOPS per dollar solution available right now.
- mk_stjames 4y ago
  I have an interest in doing lattice Boltzmann fluid dynamics simulation on GPUs, and this is something I've noticed with the P100's as well. That and memory bandwidth is super important for such calcs, along with total memory. Commercial codes I've seen aren't using any ML ops compute optimizations or tensor cores and anything like that, it's mostly just OpenCL based and thus the P100s throughput per dollar is pretty high.
- bioemerl 4y ago
  MI25 Faster, cheaper, crippled by AMDs poor software support.

mk_stjames 4y ago

Oh I know for sure, it's still just amusing to me to look up these just slightly older cards and see them going for so little compared to what they were new; I do also look at various benchmarks and comparisons over and over because I would like to build a home gpu server for some machine learning fun and I find it interesting that the TFLOPS/$ numbers always come out very much linear... A card that gives 2x performance costs 2x more.... if cards depreciate 50%, it's because new cards have been launched that have double performance for the same price... etc... Even simpler, the CUDA cores-per-dollar is pretty much a constant. What isn't taken into account there is various optimizations new cards have that speed up training or inference, like better use of the tensor cores. At some point incompatibility with newer CUDA compute levels makes the older cards much less useable.

dzdt 4y ago

Not directly applicable for large language models, I was doing some searching for cards with lots of double-precision calc performance. Nvidia has been leaving that out of all of the consumer versions of hardware; cheap P100's seem to be by far the highest DP FLOPS per dollar solution available right now.

mk_stjames 4y ago
I have an interest in doing lattice Boltzmann fluid dynamics simulation on GPUs, and this is something I've noticed with the P100's as well. That and memory bandwidth is super important for such calcs, along with total memory. Commercial codes I've seen aren't using any ML ops compute optimizations or tensor cores and anything like that, it's mostly just OpenCL based and thus the P100s throughput per dollar is pretty high.
bioemerl 4y ago
MI25 Faster, cheaper, crippled by AMDs poor software support.