3 ms·
Unrelated, but on the topic of reducing power consumption, I want to once again note that both AMD and NVidia max out a CPU core per blocking API call, preventi
by FeepingCreature 2y ago
Unrelated, but on the topic of reducing power consumption, I want to once again note that both AMD and NVidia max out a CPU core per blocking API call, preventing your CPU from entering low power states even when doing nothing but waiting on the GPU, for no reason other than to minimally rice benchmarks.
Basically, these APIs are set up to busyspin while waiting for a bus write from the GPU by default (!), rather than use interrupts like every other hardware device on your system.
You turn it off with
NVidia: `cudaSetDeviceFlags(cudaDeviceScheduleBlockingSync)`
AMD: `hipSetDeviceFlags(hipDeviceScheduleBlockingSync)`
On Pytorch
NVidia: `import ctypes \ ctypes.CDLL('libcudart.so').cudaSetDeviceFlags(4)`
AMD: `import ctypes \ ctypes.CDLL('libamdhip64.so').hipSetDeviceFlags(4)`
This saves me 20W whenever my GPU is busy in ComfyUI.
Every single device using the default settings for CUDA/ROCM burns a CPU core per worker thread for no reason.
- bob1029 2y ago> for no reason other than to minimally rice benchmarks. For AI/ML applications, perhaps no one will notice. For gaming, yielding threads of execution to the OS can periodically incur minimum scheduler delays of 10-20ms. Many gamers will notice an ~extra frame of latency being randomly injected.
- FeepingCreature 2y agoSure, but CUDA is an AI/ML API, and anyways you're not doing blocking calls when writing a graphics engine regardless. (Well, you better not.) And anyways, these calls will already busyspin for a few millis before yielding to the OS - it's just that you have to explicitly opt in to the latter part. So these are the sorts of calls that you'd use for high-throughput work, but they behave like calls designed for very-low-latency work. There is no point in shaving a few milliseconds off a low-seconds call other than to make NVidia look a few percent better in benchmarks. The tradeoffs are all wrong, and because nobody knows about it, megawatts of energy are being wasted.
- saagarjha 2y agoThis is important if you are launching many kernels and orchestrating their execution from the CPU.
- FeepingCreature 2y agoIn that case (which tbh is kind of bad design imo), you should have to explicitly opt in to the power-hungry mode.
- saagarjha 2y agoThis is a thing that people want, hence the decision. Unfortunately those people pay Nvidia a lot more money than you do.
- FeepingCreature 2y agoThe thing is, it's really not hard to recognize this access pattern. Just bucket API call times and switch modes on the fly. There is simply no excuse for an app that does 10 API calls a second to burn 100% CPU.
- deleted 2y ago[deleted]