3 ms·

This is not observable from LLM inference, where you would not encounter uniform matrices. Power limiting does not improve performance but it does improve effi

This is not observable from LLM inference, where you would not encounter uniform matrices. Power limiting does not improve performance but it does improve efficiency. You might be able to get 90% of the performance for only 70% of the power usage, for example. It does not make the card go faster though.

Lerc 5mo ago
When thermal throttling occurs you can perform faster by running slower. This is precicely because of the efficiency. The lower efficiency of the higher speed triggers a much lower performance sooner.
- Aurornis 5mo ago
  > When thermal throttling occurs you can perform faster by running slower. This is not true unless the throttling algorithm is so broken that it's oscillating between extremes. The parts have a curve of clock speed versus voltage. More clock speed means higher performance. That goes further up the voltage curve, meaning more power. Throttling just moves the card further down the voltage to clock speed curve. It reduces clock speed, reducing performance. The cards don't "perform faster by running slower". If you run the card slower, it performs slower.
  - PcChip 5mo ago
    with a lower power cap set, it runs cooler, which sometimes allows the GPU to reach higher boost speeds. This is a real effect on gaming GPUs - however I have no idea if it applies to datacenter GPUs
  - Lerc 5mo ago
    >This is not true unless the throttling algorithm is so broken that it's oscillating between extremes. That algorithm is doing exactly the task I described. If it could temporarily run faster but in a way that would cause occilation, that literally means it can run faster but it is choosing not to to preserve overall performance.
Ardon 5mo ago
This isn't necessarily true, especially with consumer GPUs. Some actually can clock higher with less voltage. It's pretty rare, and mostly comes from factory overclocked cards. For it to help you need a card that 1. Thermal throttles 2. can sustain its max OC with less voltage than set at the factory. In that rare case you are removing thermal pressure which allows you to clock higher for longer. It's the silicon lottery though, and (often) lazy board partners just smashing the voltage up as high as the chip maker allows. You definitely won't get more performance from a datacenter GPU this way.

Lerc 5mo ago

When thermal throttling occurs you can perform faster by running slower. This is precicely because of the efficiency. The lower efficiency of the higher speed triggers a much lower performance sooner.

Aurornis 5mo ago
> When thermal throttling occurs you can perform faster by running slower. This is not true unless the throttling algorithm is so broken that it's oscillating between extremes. The parts have a curve of clock speed versus voltage. More clock speed means higher performance. That goes further up the voltage curve, meaning more power. Throttling just moves the card further down the voltage to clock speed curve. It reduces clock speed, reducing performance. The cards don't "perform faster by running slower". If you run the card slower, it performs slower.
- PcChip 5mo ago
  with a lower power cap set, it runs cooler, which sometimes allows the GPU to reach higher boost speeds. This is a real effect on gaming GPUs - however I have no idea if it applies to datacenter GPUs
- Lerc 5mo ago
  >This is not true unless the throttling algorithm is so broken that it's oscillating between extremes. That algorithm is doing exactly the task I described. If it could temporarily run faster but in a way that would cause occilation, that literally means it can run faster but it is choosing not to to preserve overall performance.

Ardon 5mo ago

This isn't necessarily true, especially with consumer GPUs. Some actually can clock higher with less voltage. It's pretty rare, and mostly comes from factory overclocked cards. For it to help you need a card that 1. Thermal throttles 2. can sustain its max OC with less voltage than set at the factory. In that rare case you are removing thermal pressure which allows you to clock higher for longer. It's the silicon lottery though, and (often) lazy board partners just smashing the voltage up as high as the chip maker allows. You definitely won't get more performance from a datacenter GPU this way.