5 ms·

The problem with htop on Linux is that once there are 200+ processes running on the system htop takes significant portion of CPU time and utilization. This is b

The problem with htop on Linux is that once there are 200+ processes running on the system htop takes significant portion of CPU time and utilization. This is because htop has to go through each process entry in procfs (open, read, close) every second and parse the text context instead of just calling appropriate syscall like on the OpenBSD and friends to retrieve such information. It would help if kernel provided process information in binary form instead of serializing it into text. Or even better to provide specific syscalls for it like on macOS, Windows, OpenBSD, Solaris and others.

herpderperator 7y ago
Significant in what way? I created 400 processes + 328 threads on a 10-year-old CPU and htop is not using more than 1.3% CPU on a machine with 800% available CPU power (quad-core, 8-thread)[0]. That means 0.16% total CPU used. While I agree that it is _less_ efficient than some other ways, in what way is that _significant_? [0] https://i.imgur.com/onNSHQw.png https://i.imgur.com/onNSHQw.png
- kuschku 7y ago
  Try many small or tiny processes, and soon htop overhead becomes significant (up to 5% total utilization in my situation), that's not exactly efficient.
- sanjayts 7y ago
  On a 64 core / 512GB RAM server-class machine with 2K tasks, around 20K threads and 12% load, `htop` lags like crazy -- pretty much unusable.
- for_xyz 7y ago
  On my system there are 240 tasks running with ~1700 threads. Htop is using 6% of single core with cgroup column disabled and 9% with it enabled. It spends most of the time in kernel space so it's not htops fault. > While I agree that it is _less_ efficient than some other ways, in what way is that _significant_? This gets noticeable when every connected user/session runs the htop.
mhd 7y ago
I remember way back in the early years of the Third Age, I had to write a process accounting system that supported all kinds of Unices (HP-UX, AIX, Solaris, FreeBSD, Linux ...). And you're right, there's a plethora of other options other than procfs, although IIRC, Linux wasn't the only one I had to support with that variant. Wasn't Solaris one of them?
coleifer 7y ago
Lennart, we know it's you.
microcolonel 7y ago
I will say, htop is a lot more efficient than GNU top, despite its functionality. I do not use it in the (default?) mode where it lists all the threads, because that is nuts.
dfox 7y ago
Traditional Unix implementation of ps and similar tools work by directly reading the appropriate data structures from kernel (through /dev/kmem or something to that effect). Modern BSD implementations have libkvm.a, that abstracts that somehow, but still directly reads stuff from kernel space. I don't know, but I don't see doing random reads from kernel memory as particularly sane API to get list of processes and procfs is several orders of magnitude cleaner solution.
- twic 7y ago
  Getting a list of numbers from the kernel by serialising them as text in a pretend filesystem is a several orders of magnitude cleaner solution?
- luke0016 7y ago
  > Traditional Unix implementation of ps and similar tools work by directly reading the appropriate data structures from kernel (through /dev/kmem or something to that effect). This is not correct - /dev/kmem and similar are typically only readable by root. If what you say were correct, ps and friends wouldn't work for unprivileged users (unless they were setuid root, which they're not).
  - Erwin 7y ago
    Some version of *BSDs probably fixed that with some sysctl interface, but on older Unixes you would read the kernel memory to get that system information. You can see a ps.c implementation here: https://searchcode.com/codesearch/view/29853364/ https://searchcode.com/codesearch/view/29853364/ which uses kvm library: https://www.freebsd.org/cgi/man.cgi?query=kvm&sektion=3&apropos=0&manpath=FreeBSD+12.0-RELEASE+and+Ports https://www.freebsd.org/cgi/man.cgi?query=kvm&sektion=3&apro...
    - trasz 7y ago
      FWIW, in FreeBSD libkvm also uses the sysctl interface, it doesn’t read kernel memory directly.
seppel 7y ago
Well, you are right (and top has the same problem if not worse), but I'm always surprised what these tools are doing that they take significat CPU time. Parsing 200 virtual text files every second for sure should not. > Or even better to provide specific syscalls for it like on macOS, Windows, OpenBSD, Solaris and others. top manages to take 6% CPU on my Macbook.
dana321 7y ago
Its not meant to be used 24/7, its just used as a guide/diagnsostic to how the machine and its processes are performing! I use it for maybe about 10 seconds.
- for_xyz 7y ago
  Some people use tmux when connecting to their virtual machines on the server and forget to quit the htop instances. This adds to the total CPU utilization really quick.
zantana 7y ago
I have wondered if it made sense to re-write these top style tools using ebpf https://lwn.net/Articles/740157/ https://lwn.net/Articles/740157/ but I think ebpf requires root.

herpderperator 7y ago

Significant in what way? I created 400 processes + 328 threads on a 10-year-old CPU and htop is not using more than 1.3% CPU on a machine with 800% available CPU power (quad-core, 8-thread)[0]. That means 0.16% total CPU used. While I agree that it is _less_ efficient than some other ways, in what way is that _significant_? [0] https://i.imgur.com/onNSHQw.png https://i.imgur.com/onNSHQw.png

kuschku 7y ago
Try many small or tiny processes, and soon htop overhead becomes significant (up to 5% total utilization in my situation), that's not exactly efficient.
sanjayts 7y ago
On a 64 core / 512GB RAM server-class machine with 2K tasks, around 20K threads and 12% load, `htop` lags like crazy -- pretty much unusable.
for_xyz 7y ago
On my system there are 240 tasks running with ~1700 threads. Htop is using 6% of single core with cgroup column disabled and 9% with it enabled. It spends most of the time in kernel space so it's not htops fault. > While I agree that it is _less_ efficient than some other ways, in what way is that _significant_? This gets noticeable when every connected user/session runs the htop.

mhd 7y ago

I remember way back in the early years of the Third Age, I had to write a process accounting system that supported all kinds of Unices (HP-UX, AIX, Solaris, FreeBSD, Linux ...). And you're right, there's a plethora of other options other than procfs, although IIRC, Linux wasn't the only one I had to support with that variant. Wasn't Solaris one of them?

coleifer 7y ago

Lennart, we know it's you.

microcolonel 7y ago

I will say, htop is a lot more efficient than GNU top, despite its functionality. I do not use it in the (default?) mode where it lists all the threads, because that is nuts.

dfox 7y ago

Traditional Unix implementation of ps and similar tools work by directly reading the appropriate data structures from kernel (through /dev/kmem or something to that effect). Modern BSD implementations have libkvm.a, that abstracts that somehow, but still directly reads stuff from kernel space. I don't know, but I don't see doing random reads from kernel memory as particularly sane API to get list of processes and procfs is several orders of magnitude cleaner solution.

twic 7y ago
Getting a list of numbers from the kernel by serialising them as text in a pretend filesystem is a several orders of magnitude cleaner solution?
luke0016 7y ago
> Traditional Unix implementation of ps and similar tools work by directly reading the appropriate data structures from kernel (through /dev/kmem or something to that effect). This is not correct - /dev/kmem and similar are typically only readable by root. If what you say were correct, ps and friends wouldn't work for unprivileged users (unless they were setuid root, which they're not).
- Erwin 7y ago
  Some version of *BSDs probably fixed that with some sysctl interface, but on older Unixes you would read the kernel memory to get that system information. You can see a ps.c implementation here: https://searchcode.com/codesearch/view/29853364/ https://searchcode.com/codesearch/view/29853364/ which uses kvm library: https://www.freebsd.org/cgi/man.cgi?query=kvm&sektion=3&apropos=0&manpath=FreeBSD+12.0-RELEASE+and+Ports https://www.freebsd.org/cgi/man.cgi?query=kvm&sektion=3&apro...
  - trasz 7y ago
    FWIW, in FreeBSD libkvm also uses the sysctl interface, it doesn’t read kernel memory directly.

seppel 7y ago

Well, you are right (and top has the same problem if not worse), but I'm always surprised what these tools are doing that they take significat CPU time. Parsing 200 virtual text files every second for sure should not. > Or even better to provide specific syscalls for it like on macOS, Windows, OpenBSD, Solaris and others. top manages to take 6% CPU on my Macbook.

dana321 7y ago

Its not meant to be used 24/7, its just used as a guide/diagnsostic to how the machine and its processes are performing! I use it for maybe about 10 seconds.

for_xyz 7y ago
Some people use tmux when connecting to their virtual machines on the server and forget to quit the htop instances. This adds to the total CPU utilization really quick.

zantana 7y ago

I have wondered if it made sense to re-write these top style tools using ebpf https://lwn.net/Articles/740157/ https://lwn.net/Articles/740157/ but I think ebpf requires root.