7 ms·

Nvtop: Linux Task Monitor for Nvidia, AMD and Intel GPUs

453 points by alexzeitler 3y ago

hughesjj 3y ago
Nvtop+bottom are my favorite resource monitors for Linux today, but TIL nvtop also works for non-nvidia devices
- freedomben 3y ago
  wow, bottom[1] is awesome! This is now my favorite monitor It's not in the Fedora repos but there's a COPR for it. To install: sudo dnf copr enable atim/bottom sudo dnf install bottom [1] https://github.com/ClementTsang/bottom https://github.com/ClementTsang/bottom
  - petepete 3y ago
    btop++ is also worth a look if you like bottom, another take on a modern htop. https://github.com/aristocratos/btop https://github.com/aristocratos/btop
    - freedomben 3y ago
      Amazing, thank you!
    - Cu3PO42 3y ago
      And it also supports GPU usage monitoring!
- weinzierl 3y ago
  I use htop but often want to focus on a process, its children and all their threads. Is there a htop replacement that can; - Show a thread with all children an threads, but nothing else - Show the whole tree but keep the selected (Shift + Space) process in a fixed screen position. - Bubbles rows up and down into their new positions instead of having them jump around all over the place.
  - themoonisachees 3y ago
    Btop does all that and is available in your package manager. It also looks beautiful.
    - weinzierl 3y ago
      Has a nice 90s vibe. I have not figured out how to just expand the selected process. I could collapse everything except the selected process manually, but that'd be tedious.
      - DonHopkins 3y ago
        90's rave scene! ;)
      - themoonisachees 3y ago
        I have not explored how, but I'm pretty sure you can define custom templates for the layout. Maybe you may make a custom template with nothing but the selected process window?
collsni 3y ago
Amdgpu_top is another cool usage statistic/monitor for and GPUs
formalsystem 3y ago
nvtop or nvidia-smi gives you a good macro overview but I personally have found that utilization (EDIT: As reported by nvidia-smi) is actually a poor proxy for how fast your workload can be outside of just ensuring that a GPU is indeed being used If you're here because you're interested in AI performance I'd recommend instead https://docs.nvidia.com/nsight-compute/NsightComputeCli/index.html https://docs.nvidia.com/nsight-compute/NsightComputeCli/inde... to profile individual kernels. Nsight systems for a macro view https://developer.nvidia.com/nsight-systems https://developer.nvidia.com/nsight-systems and the PyTorch profiler if you're not authoring kernels directly but using something PyTorch https://pytorch.org/tutorials/recipes/recipes/profiler_recipe.html https://pytorch.org/tutorials/recipes/recipes/profiler_recip...
- refibrillator 3y ago
  FLOPs utilization is arguably the industry standard metric for efficiency right now and it should be a good first approximation of how much performance is left on the table. But if you mean the reported utilization in nvtop is misleading I completely agree (as someone who uses it daily). I’ve been meaning to dig into the source/docs to see what’s going on. The power usage seems to be a more reliable indicator of actual hardware utilization, at least on nvidia gear.
  - VHRanger 3y ago
    > FLOPs utilization is arguably the industry standard metric for efficiency right now I'd argue GB/s memory bandwidth is more worried about at the moment.
- ipsum2 3y ago
  I've been going off of power draw in nvidia-smi as a proxy of util, doesn't require additional setup or code changes.
  - KeplerBoy 3y ago
    That's hard to argue with. Of course power draw is a direct measure of hardware utilization, but it doesn't translate very well to a measure of GPU Code efficiency. Often you can squeeze out another order of magnitude of performance by rewriting the kernel and the power draw will always stay capped at whatever the maximum is. I'd say GPU power consumption is interesting if you're CPU bound and struggling to feed the GPU enough data and/or tasks.
cyberax 3y ago
Is there a description of the wire protocol between the driver and NVidia library? Context: I'd love to have a native Go-language library to read the GPU utilization for containerized workloads.
- 8organicbits 3y ago
  Yes, check out NVML. There's wrappers for a couple languages, not sure about golang. https://developer.nvidia.com/nvidia-management-library-nvml https://developer.nvidia.com/nvidia-management-library-nvml
malux85 3y ago
There is also nvitop, which I find better utilizes screen space when > 2 GPUS https://github.com/XuehaiPan/nvitop https://github.com/XuehaiPan/nvitop
- freedomben 3y ago
  Does nvitop support AMD cards?
  - malux85 3y ago
    Not sure sorry, I only have nvidia :<
- sa-code 3y ago
  The downside with nvitop is that it's written in python, which means having it in your environment can cause dependency conflicts. It's either that or you have a separate venv just for it. Maybe it's fine for personal use but sysadmins would prefer nvtop
  - cl3misch 3y ago
    That's why the authors recommend pipx for installing nvitop. I am not a sysadmin, but I prefer pipx over relying on the (often outdated) distro sources. https://github.com/XuehaiPan/nvitop?tab=readme-ov-file#installation https://github.com/XuehaiPan/nvitop?tab=readme-ov-file#insta...
brcmthrowaway 3y ago
Anything for MACOS?
- evilduck 3y ago
  nvtop supposedly builds for macOS, but https://lib.rs/crates/pumas https://lib.rs/crates/pumas is similar.
- kernelsanderz 3y ago
  There’s also asitop https://github.com/tlkh/asitop https://github.com/tlkh/asitop
- itsgrimetime 3y ago
  its not a terminal app like bottom or nvtop but I use https://github.com/exelban/stats https://github.com/exelban/stats and it has iGPU stats
pavelstoev 3y ago
you can also profile AI/ML performance without actually running it https://github.com/CentML/DeepView.Profile https://github.com/CentML/DeepView.Profile
superkuh 3y ago
radeontop is the same sort of thing if you live in amdgpu-ville and want something easy to compile. I was able to use it to show that with kernel 5.x admgpu vulkan when a process is pushed out of vram into gtt it'll never reload and get stuck in a 'slow' state.
shmerl 3y ago
May be you should use Nova instead of NVML for Nvidia? https://lists.freedesktop.org/archives/nouveau/2024-February/044078.html https://lists.freedesktop.org/archives/nouveau/2024-February... Other than that - a cool tool!
notorandit 3y ago
It is nice. But the status of GPU support in Linux is rather poor. I can see it working only with VLC. Firefox has some support while chromium based browsers have that only formally. In the real world you never see the video hw acceleration kicking in, neither with webrtc nor with videos. It is a pity.
- KeplerBoy 3y ago
  it's a valuable tool wherever GPU compute is used. Not sure if video decoding would even register as utilization since that is dedicated hardware.
  - usr1106 3y ago
    Yes, you can see video decoding, but it's indeed only one detail of the data presented, as it's only a part of the HW. Not at my computer now, can't tell what metric to watch.
    - KeplerBoy 3y ago
      ah yes, you can configure nvtop to display encode and decode loads. Interesting to watch how playback speed in VLC is directly reflected in the metric.
    - notorandit 3y ago
      Which software would use video decoding?
      - adrian_b 3y ago
        Pretty much everything except the browsers. I use several video players, including mpv, vlc and ffmpeg, and they have always used without problems the hardware video decoding and encoding on all kinds of GPUs. Only with Firefox and Chrome/Chromium in most versions the hardware acceleration is broken, even if there have been some versions where it worked fine (on NVIDIA), but at the next browser upgrade it was broken again. This does not bother me much, because I do not like to watch video files in a browser anyway. I always download them first and I play them locally.
distalx 3y ago
There is also Nvitop[1], which is more to my liking. [1] https://github.com/XuehaiPan/nvitop https://github.com/XuehaiPan/nvitop
- kiraaa 3y ago
  super easy to install and use
- cassianoleal 3y ago
  Seems to be for NVidia only, whereas OP claims: > Currently supported vendors are AMD (Linux amdgpu driver), Apple (limited M1 & M2 support), Huawei (Ascend), Intel (Linux i915 driver), NVIDIA (Linux proprietary divers), Qualcomm Adreno (Linux MSM driver).
blagie 3y ago
Now that I use Home Assistant, I want all my data sources to plug into there. It can handle the rendering for me as I see fit, and it's where data comes to integrate. It's one of those things which I wish existed, but I can't imagine anyone would have written. Until I do a web search. https://github.com/koriwi/sensors2mqtt/tree/main https://github.com/koriwi/sensors2mqtt/tree/main I have not used it yet, but that seems like how I'd want to do it.
- feitingen 3y ago
  Collectd can also output stats to mqtt, from sensors, disks, network and others.
Ycros 3y ago
I prefer btop, it does all the usual process monitoring as well as gpus in the latest versions.
- notorandit 3y ago
  Really? Mine is v1.3.2 and doesn't show Intel Iris Xe Graphics! {UPDATE} I see: no Intel GPU support yet!
stuaxo 3y ago
Its good to see linux graphics card utilities going multi platform, instead of the old way of being per-driver.
vilunov 3y ago
It doesn't work with mesa, does it? I'm using the new nvk driver and it shows me no GPU to monitor.
- thangngoc89 3y ago
  Of course it doesn't. `nvidia-smi` needs to be available
sylware 3y ago
cmake... erk... I would have prefered a simple and brutal shell script (not bash of course) to build it on elf/linux. It is worth trashing cmake, always, so I'll write it if I end up using that GPU monitoring tool.
thangngoc89 3y ago
My favorite would be gpustat [1]. This shows the bare minimum amount of information to let's me know that the training has problems/running well [1] https://github.com/wookayin/gpustat https://github.com/wookayin/gpustat
mrlonglong 3y ago
It'd be nice if it supported Nouveau.
visarga 3y ago
apt install is not working for me, is this by design? > nvtop : Depends: libnvidia-compute-418 but it is not going to be installed E: Unable to correct problems, you have held broken packages. <rant>I find broken installs a huge turnoff, especially those related to NVIDIA. With their 2.3T market cap they can't afford someone to write an universal point and click install script for ML usage? Every time I reinstall Linux I have to spend a whole day sorting NVIDIA out. Why do they have so many layers - driver, cuda, cuda toolkit, cudnn with conflicting versioning - it's a total mess. Instead of a nice install script we have a million install guides 10 pages long, all outdated.</>
- bayindirh 3y ago
  Because all these problems don’t hinder their bottom line. Cluster admins or Ph.D. students handle these problems, allowing people to work. All this infra is already buried under Conda, Jupyter, etc. for most people already. Sincerely, Your friendly HPC admin.
  - VHRanger 3y ago
    That's just because they don't have competition. Back in the dark days of 2015 we used to spend a day or two just getting tensorflow working on a GPU because of all the install issues, driver issues, etc. Theano was no better, but it was academic research code, we didn't expect better. Once pytorch started gaining ground, it forced to adapt - Keras was written to hide tensorflow's awfulness. Then Google realized it's an unrecoverable situation of technical debt and they started building JAX. With AMD, Intel, Tenstorrent, and several other AI chip specialists coming with pytorch compatibility, NVIDIA will eventually have to adapt. They still have the advantage of 15 years of CUDA code already written, but pytorch as ab abstraction layer can make the switch easier.
    - LoganDark 3y ago
      PyTorch and CUDA solve completely different problems. CUDA is a general purpose programming environment. PyTorch is for machine learning. PyTorch won't ever displace CUDA because there are things other than machine learning models that GPUs are good at accelerating.
KingLancelot 3y ago
[dead]
bvaisvil 3y ago
Zenith is my project which combines NVIDIA GPU monitoring with disk, CPU, and Top-like capabilities. https://github.com/bvaisvil/zenith https://github.com/bvaisvil/zenith

453 points by alexzeitler 3y ago

hughesjj 3y ago

Nvtop+bottom are my favorite resource monitors for Linux today, but TIL nvtop also works for non-nvidia devices

freedomben 3y ago
wow, bottom[1] is awesome! This is now my favorite monitor It's not in the Fedora repos but there's a COPR for it. To install: sudo dnf copr enable atim/bottom sudo dnf install bottom [1] https://github.com/ClementTsang/bottom https://github.com/ClementTsang/bottom
- petepete 3y ago
  btop++ is also worth a look if you like bottom, another take on a modern htop. https://github.com/aristocratos/btop https://github.com/aristocratos/btop
  - freedomben 3y ago
    Amazing, thank you!
  - Cu3PO42 3y ago
    And it also supports GPU usage monitoring!
weinzierl 3y ago
I use htop but often want to focus on a process, its children and all their threads. Is there a htop replacement that can; - Show a thread with all children an threads, but nothing else - Show the whole tree but keep the selected (Shift + Space) process in a fixed screen position. - Bubbles rows up and down into their new positions instead of having them jump around all over the place.
- themoonisachees 3y ago
  Btop does all that and is available in your package manager. It also looks beautiful.
  - weinzierl 3y ago
    Has a nice 90s vibe. I have not figured out how to just expand the selected process. I could collapse everything except the selected process manually, but that'd be tedious.
    - DonHopkins 3y ago
      90's rave scene! ;)
    - themoonisachees 3y ago
      I have not explored how, but I'm pretty sure you can define custom templates for the layout. Maybe you may make a custom template with nothing but the selected process window?

collsni 3y ago

Amdgpu_top is another cool usage statistic/monitor for and GPUs

formalsystem 3y ago

nvtop or nvidia-smi gives you a good macro overview but I personally have found that utilization (EDIT: As reported by nvidia-smi) is actually a poor proxy for how fast your workload can be outside of just ensuring that a GPU is indeed being used If you're here because you're interested in AI performance I'd recommend instead https://docs.nvidia.com/nsight-compute/NsightComputeCli/index.html https://docs.nvidia.com/nsight-compute/NsightComputeCli/inde... to profile individual kernels. Nsight systems for a macro view https://developer.nvidia.com/nsight-systems https://developer.nvidia.com/nsight-systems and the PyTorch profiler if you're not authoring kernels directly but using something PyTorch https://pytorch.org/tutorials/recipes/recipes/profiler_recipe.html https://pytorch.org/tutorials/recipes/recipes/profiler_recip...

refibrillator 3y ago
FLOPs utilization is arguably the industry standard metric for efficiency right now and it should be a good first approximation of how much performance is left on the table. But if you mean the reported utilization in nvtop is misleading I completely agree (as someone who uses it daily). I’ve been meaning to dig into the source/docs to see what’s going on. The power usage seems to be a more reliable indicator of actual hardware utilization, at least on nvidia gear.
- VHRanger 3y ago
  > FLOPs utilization is arguably the industry standard metric for efficiency right now I'd argue GB/s memory bandwidth is more worried about at the moment.
ipsum2 3y ago
I've been going off of power draw in nvidia-smi as a proxy of util, doesn't require additional setup or code changes.
- KeplerBoy 3y ago
  That's hard to argue with. Of course power draw is a direct measure of hardware utilization, but it doesn't translate very well to a measure of GPU Code efficiency. Often you can squeeze out another order of magnitude of performance by rewriting the kernel and the power draw will always stay capped at whatever the maximum is. I'd say GPU power consumption is interesting if you're CPU bound and struggling to feed the GPU enough data and/or tasks.

cyberax 3y ago

Is there a description of the wire protocol between the driver and NVidia library? Context: I'd love to have a native Go-language library to read the GPU utilization for containerized workloads.

8organicbits 3y ago
Yes, check out NVML. There's wrappers for a couple languages, not sure about golang. https://developer.nvidia.com/nvidia-management-library-nvml https://developer.nvidia.com/nvidia-management-library-nvml

malux85 3y ago

There is also nvitop, which I find better utilizes screen space when > 2 GPUS https://github.com/XuehaiPan/nvitop https://github.com/XuehaiPan/nvitop

freedomben 3y ago
Does nvitop support AMD cards?
- malux85 3y ago
  Not sure sorry, I only have nvidia :<
sa-code 3y ago
The downside with nvitop is that it's written in python, which means having it in your environment can cause dependency conflicts. It's either that or you have a separate venv just for it. Maybe it's fine for personal use but sysadmins would prefer nvtop
- cl3misch 3y ago
  That's why the authors recommend pipx for installing nvitop. I am not a sysadmin, but I prefer pipx over relying on the (often outdated) distro sources. https://github.com/XuehaiPan/nvitop?tab=readme-ov-file#installation https://github.com/XuehaiPan/nvitop?tab=readme-ov-file#insta...

brcmthrowaway 3y ago

Anything for MACOS?

evilduck 3y ago
nvtop supposedly builds for macOS, but https://lib.rs/crates/pumas https://lib.rs/crates/pumas is similar.
kernelsanderz 3y ago
There’s also asitop https://github.com/tlkh/asitop https://github.com/tlkh/asitop
itsgrimetime 3y ago
its not a terminal app like bottom or nvtop but I use https://github.com/exelban/stats https://github.com/exelban/stats and it has iGPU stats

pavelstoev 3y ago

you can also profile AI/ML performance without actually running it https://github.com/CentML/DeepView.Profile https://github.com/CentML/DeepView.Profile

superkuh 3y ago

radeontop is the same sort of thing if you live in amdgpu-ville and want something easy to compile. I was able to use it to show that with kernel 5.x admgpu vulkan when a process is pushed out of vram into gtt it'll never reload and get stuck in a 'slow' state.

shmerl 3y ago

May be you should use Nova instead of NVML for Nvidia? https://lists.freedesktop.org/archives/nouveau/2024-February/044078.html https://lists.freedesktop.org/archives/nouveau/2024-February... Other than that - a cool tool!

notorandit 3y ago

It is nice. But the status of GPU support in Linux is rather poor. I can see it working only with VLC. Firefox has some support while chromium based browsers have that only formally. In the real world you never see the video hw acceleration kicking in, neither with webrtc nor with videos. It is a pity.

KeplerBoy 3y ago
it's a valuable tool wherever GPU compute is used. Not sure if video decoding would even register as utilization since that is dedicated hardware.
- usr1106 3y ago
  Yes, you can see video decoding, but it's indeed only one detail of the data presented, as it's only a part of the HW. Not at my computer now, can't tell what metric to watch.
  - KeplerBoy 3y ago
    ah yes, you can configure nvtop to display encode and decode loads. Interesting to watch how playback speed in VLC is directly reflected in the metric.
  - notorandit 3y ago
    Which software would use video decoding?
    - adrian_b 3y ago
      Pretty much everything except the browsers. I use several video players, including mpv, vlc and ffmpeg, and they have always used without problems the hardware video decoding and encoding on all kinds of GPUs. Only with Firefox and Chrome/Chromium in most versions the hardware acceleration is broken, even if there have been some versions where it worked fine (on NVIDIA), but at the next browser upgrade it was broken again. This does not bother me much, because I do not like to watch video files in a browser anyway. I always download them first and I play them locally.

distalx 3y ago

There is also Nvitop[1], which is more to my liking. [1] https://github.com/XuehaiPan/nvitop https://github.com/XuehaiPan/nvitop

kiraaa 3y ago
super easy to install and use
cassianoleal 3y ago
Seems to be for NVidia only, whereas OP claims: > Currently supported vendors are AMD (Linux amdgpu driver), Apple (limited M1 & M2 support), Huawei (Ascend), Intel (Linux i915 driver), NVIDIA (Linux proprietary divers), Qualcomm Adreno (Linux MSM driver).

blagie 3y ago

Now that I use Home Assistant, I want all my data sources to plug into there. It can handle the rendering for me as I see fit, and it's where data comes to integrate. It's one of those things which I wish existed, but I can't imagine anyone would have written. Until I do a web search. https://github.com/koriwi/sensors2mqtt/tree/main https://github.com/koriwi/sensors2mqtt/tree/main I have not used it yet, but that seems like how I'd want to do it.

feitingen 3y ago
Collectd can also output stats to mqtt, from sensors, disks, network and others.

Ycros 3y ago

I prefer btop, it does all the usual process monitoring as well as gpus in the latest versions.

notorandit 3y ago
Really? Mine is v1.3.2 and doesn't show Intel Iris Xe Graphics! {UPDATE} I see: no Intel GPU support yet!

stuaxo 3y ago

Its good to see linux graphics card utilities going multi platform, instead of the old way of being per-driver.

vilunov 3y ago

It doesn't work with mesa, does it? I'm using the new nvk driver and it shows me no GPU to monitor.

thangngoc89 3y ago
Of course it doesn't. `nvidia-smi` needs to be available

sylware 3y ago

cmake... erk... I would have prefered a simple and brutal shell script (not bash of course) to build it on elf/linux. It is worth trashing cmake, always, so I'll write it if I end up using that GPU monitoring tool.

thangngoc89 3y ago

My favorite would be gpustat [1]. This shows the bare minimum amount of information to let's me know that the training has problems/running well [1] https://github.com/wookayin/gpustat https://github.com/wookayin/gpustat

mrlonglong 3y ago

It'd be nice if it supported Nouveau.

visarga 3y ago

apt install is not working for me, is this by design? > nvtop : Depends: libnvidia-compute-418 but it is not going to be installed E: Unable to correct problems, you have held broken packages. <rant>I find broken installs a huge turnoff, especially those related to NVIDIA. With their 2.3T market cap they can't afford someone to write an universal point and click install script for ML usage? Every time I reinstall Linux I have to spend a whole day sorting NVIDIA out. Why do they have so many layers - driver, cuda, cuda toolkit, cudnn with conflicting versioning - it's a total mess. Instead of a nice install script we have a million install guides 10 pages long, all outdated.</>

bayindirh 3y ago
Because all these problems don’t hinder their bottom line. Cluster admins or Ph.D. students handle these problems, allowing people to work. All this infra is already buried under Conda, Jupyter, etc. for most people already. Sincerely, Your friendly HPC admin.
- VHRanger 3y ago
  That's just because they don't have competition. Back in the dark days of 2015 we used to spend a day or two just getting tensorflow working on a GPU because of all the install issues, driver issues, etc. Theano was no better, but it was academic research code, we didn't expect better. Once pytorch started gaining ground, it forced to adapt - Keras was written to hide tensorflow's awfulness. Then Google realized it's an unrecoverable situation of technical debt and they started building JAX. With AMD, Intel, Tenstorrent, and several other AI chip specialists coming with pytorch compatibility, NVIDIA will eventually have to adapt. They still have the advantage of 15 years of CUDA code already written, but pytorch as ab abstraction layer can make the switch easier.
  - LoganDark 3y ago
    PyTorch and CUDA solve completely different problems. CUDA is a general purpose programming environment. PyTorch is for machine learning. PyTorch won't ever displace CUDA because there are things other than machine learning models that GPUs are good at accelerating.

KingLancelot 3y ago

[dead]

bvaisvil 3y ago

Zenith is my project which combines NVIDIA GPU monitoring with disk, CPU, and Top-like capabilities. https://github.com/bvaisvil/zenith https://github.com/bvaisvil/zenith