26 ms·

ProcMon for Linux (Preview)

182 points by vmarsy 5y ago

jxf 5y ago
I'm not familiar with the Windows version of ProcMon, but judging by the Linux version, that looks like it's primarily replicating the function of `strace` in a CLI GUI. Worth reading Joe Damato's excellent article on strace: https://blog.packagecloud.io/eng/2016/02/29/how-does-strace-work/ https://blog.packagecloud.io/eng/2016/02/29/how-does-strace-...
- atomicnumber3 5y ago
  It's not quite strace, ProcMon also does network traffic stuff too. I used it quite a bit when my job was trying to shove proprietary (usually academic, usually engineering-related) software installers into a shape that we could make silently install. ProcMon was one of our first resorts whenever an installer would inevitably mysteriously crash when it ran inside an SCCM-managed installation of windows. Which. Was. OFTEN. Sorry, it's been over a decade now and I still have nightmares sometimes. But yeah procmon is cool.
  - ed_elliott_asc 5y ago
    I was in a similar role about that time, I bet you had fun with msvc and manifests :)
  - _carbyau_ 5y ago
    I do other sysadmin now but the job of getting whatever random vendor's installer into an SCCM deployable state required tools to see just WTF they were doing. I recall one .exe installer, unzipping to an MSI, which installed the program files but then ran .bat file to run three separate driver installers for the associated hardware. And I had to get it all working in AppV. I don't use Procmon much now as my work has changed, but I still consider it a good friend. I am glad linux has other ways to do it but as a linux noob, I'd try procmon first if I had to.
- Arnavion 5y ago
  Yes, ProcMon is the Windows equivalent of strace, with some filtering capability built-in. It also shows you the stack involved with a particular event, which can be useful for diagnosing the otherwise black box that is Windows. Eg [1] [1]: https://github.com/msys2/MSYS2-packages/issues/138#issuecomment-396447222 https://github.com/msys2/MSYS2-packages/issues/138#issuecomm...
  - eps 5y ago
    The "filtering capability" actually makes up a good half of its worth. Being able to very quickly drill down to an annotated call stack for a specific IRQ on a specific file is really handy.
    - Hello71 5y ago
      > call stack strace -k > specific IRQ IRQs are very low level and can't be traced by strace or procmon. both can easily log specific syscalls, e.g. strace -e. strace can't quite filter on ioctl, but apparently procmon can't trace ioctl at all: https://stackoverflow.com/questions/9947933/how-to-log-the-deviceiocontrol-calls-of-a-program-on-windows https://stackoverflow.com/questions/9947933/how-to-log-the-d..., so strace is ahead there. > specific file strace -P path
      - Arnavion 5y ago
        >> call stack >strace -k >> specific file >strace -P path That strace can do it too is fine and all, but the Windows ProcMon collects a trace that you can then filter, like Wireshark. So it enables a workflow where you have no idea what's going on, collect a trace, and discover what entry to focus on and what paths to filter for. (This is what I demonstrated in the msys issue link I posted.) An equivalent workflow is possible with strace too of course; pipe `strace` to a file and then discover what paths you want to grep for. It does become a bit noisy when the stack of every syscall is included, and requires more complicated grepping because it spans multiple lines, so you might have to first pre-process it into something structured like JSON.
- bboreham 5y ago
  Original ProcMon used ETW, Event Tracing for Windows; the analogous technology (although very different in style) on Linux is eBPF so that’s what this tool uses.
  - altano 5y ago
    I think you’re mistaken. ProcMon doesn’t use ETW on Windows and I don’t believe it ever did?
    - ale42 5y ago
      Indeed I don't think so. ProcMon uses a kernel driver for the event tracing.
    - bboreham 5y ago
      Sorry about that; I guess I misremembered? This file says it does, though only for network events: https://documentation.help/Process-Monitor/documentation.pdf https://documentation.help/Process-Monitor/documentation.pdf
stefan_ 5y ago
This seems to be using eBPF, so presumably your kernel will have to support that (only stating Ubuntu 18.04 is a bit unfortunate).
- unixhero 5y ago
  18 is already 3-4 years ago :)
  - nemetroid 5y ago
    The version itself is fine, but specifically listing Ubuntu as a system requirement is strange. If Ubuntu truly is a requirement, the name of the project should be "ProcMon for Ubuntu". Presumably it isn't, and the README should list the actual requirements (kernel versions, etc.), with a note to the effect of "packages are available for Ubuntu LTS releases since 18.04".
    - colejohnson66 5y ago
      It could also just mean it hasn’t been tested on anything else. But yes, the README should mention these things.
  - deleted 5y ago
    [deleted]
alpb 5y ago
Something that is really cool about Sysinternals tools are that they have a pretty usable GUI (granted, once you need to understand what does each UI icon do). Mimicking this experience in the TUI/terminal would certainly be a challenge. I would love to have a Procmon and ProcessExplorer equivalent with the GUI on Linux and macOS.
- a-dub 5y ago
  i always thought of the sysinternals tools as like the missing tools that just ship with unix systems. without them, there was no real visibility into open file descriptors, network ports, command lines for running processes, thread trees for processes or any of the rest. the gui was always just the windows way of presenting it. bringing a gui when the rich tools already exist on linux seems ... redundant? (seeing this brings back nightmares from windows admins who refused to install sysinternals tools on production machines)
  - themulticaster 5y ago
    This was my reaction as well. I haven't used Process Monitor/Process Explorer for a while, are they bringing anything to the table that's not covered by htop? With htop, I can even start tracing a process (using strace) or list the file opened by a process (using lsof) right inside htop. As might be obvious by now, I think htop has a fantastic TUI that also fits its use case very well. Of course there are other tools that allow deep introspection into what's really going on aside from htop. Another favorite of mine would be perf top, which uses the sampling profiling mechanism to profile the entire system (including the kernel) on a symbol/instruction level. In other words, you don't just see that 30% of CPU time is spent in /usr/bin/cpu_hog, you also see that the time is spent in some_symbol and you can jump right in and see disassemble some_symbol (as long as cpu_hog is not stripped).
    - oblio 5y ago
      > With htop, I can even start tracing a process (using strace) or list the file opened by a process (using lsof) right inside htop. Hmmm... didn't know it could do that.
unixhero 5y ago
Licensed under MIT. This looks like an amazing tool, can't wait to begin using it.
agucova 5y ago
What are the differences between this and htop, by example? I know htop can strace processes with the "s" key.
- snickerer 5y ago
  This was my first thought, too. Which useful feature does this tool have what htop does not have? Let's say there is such a feature. Why did they not implement it in htop? Made a patch and offered it to upstream? This is why we have FOSS. To benefit from each other's work.
  - tuwtuwtuwtuw 5y ago
    > Why did they not implement it in htop? Made a patch and offered it to upstream? This is why we have FOSS. To benefit from each other's work. This is why so much open source software is crap and without any structure. Htop is very different from proc on. Integrating proc on into Htop would make zero sense. Why would you suggest this would be a good idea without knowing what the software does?
  - fomine3 5y ago
    Finally Vi become emacs-evil.
rishabhd 5y ago
This will come in handy during DFIR, although we have different workflows for doing live analysis on machines for now.
mraza007 5y ago
Just out of curiosity, How is this tool different when compared to htop
Grayskull 5y ago
Even though I don't use procmon on windows basically never. Sysinternals on linux could be useful especially with same gui as on windows. It would be great to see autoruns and tcpview on linux.
boomskats 5y ago
I've been looking at a few eBPF tracing implementations around GitHub recently to try and get my head round it, and this codebase has to be one of the best documented (if not best written) examples out there. Kudos.
gtirloni 5y ago
Because Ubuntu 18.04 is used for the development environment, it relies on pretty old dependencies. Bug report: https://github.com/Sysinternals/ProcMon-for-Linux/issues/44 https://github.com/Sysinternals/ProcMon-for-Linux/issues/44
- JeremyNT 5y ago
  Yeah, if you have a modern distribution, look elsewhere. For a "preview" release it's interesting that they're only targeting older distros - maybe they're focused on enterprise use.
whalesalad 5y ago
Sorta related: If y’all haven’t used netdata - wow - it blew my socks off last night. I have it installed on all my hosts now and it’s already helping me diagnose a redis bgsave issue that I didn’t realize existed. Really well put together software.
- odyslam 5y ago
  Hey, I am odysseas, DevRel at Netdata. I would love to hear your story and what Netdata helped you diagnose in your systems. Hit me up (odyslam) in our forums: https://community.netdata.cloud https://community.netdata.cloud
- PanosJee 5y ago
  Why do you think most people don't know about netdata and stick with the "status quo" of gathering data -> timeseries db -> manually creating dashboards & configure alerts?
mnutt 5y ago
While I see a lot of comparisons to htop, I wonder how this compares to sysdig? I’ve found sysdig to be an invaluable swiss army knife of debugging performance issues; the only difficult part is finding the right documentation among the cloud offering of the same name.
aasasd 5y ago
Sysinternals is under Microsoft since 2006. Is this the ‘extend’ stage?
superkuh 5y ago
Why are libllvm6.0 llvm-6.0-dev and libclang-6.0-dev needed for this?
- nemetroid 5y ago
  Seemingly because they download and build bcc as part of the build process. As to why they do that... probably because the three year old distro they're targeting has a three year old version of bcc.
throwaway823882 5y ago
I never thought I'd say this, but: Thank you, Microsoft, for making useful, user-friendly software.

182 points by vmarsy 5y ago

jxf 5y ago

I'm not familiar with the Windows version of ProcMon, but judging by the Linux version, that looks like it's primarily replicating the function of `strace` in a CLI GUI. Worth reading Joe Damato's excellent article on strace: https://blog.packagecloud.io/eng/2016/02/29/how-does-strace-work/ https://blog.packagecloud.io/eng/2016/02/29/how-does-strace-...

atomicnumber3 5y ago
It's not quite strace, ProcMon also does network traffic stuff too. I used it quite a bit when my job was trying to shove proprietary (usually academic, usually engineering-related) software installers into a shape that we could make silently install. ProcMon was one of our first resorts whenever an installer would inevitably mysteriously crash when it ran inside an SCCM-managed installation of windows. Which. Was. OFTEN. Sorry, it's been over a decade now and I still have nightmares sometimes. But yeah procmon is cool.
- ed_elliott_asc 5y ago
  I was in a similar role about that time, I bet you had fun with msvc and manifests :)
- _carbyau_ 5y ago
  I do other sysadmin now but the job of getting whatever random vendor's installer into an SCCM deployable state required tools to see just WTF they were doing. I recall one .exe installer, unzipping to an MSI, which installed the program files but then ran .bat file to run three separate driver installers for the associated hardware. And I had to get it all working in AppV. I don't use Procmon much now as my work has changed, but I still consider it a good friend. I am glad linux has other ways to do it but as a linux noob, I'd try procmon first if I had to.
Arnavion 5y ago
Yes, ProcMon is the Windows equivalent of strace, with some filtering capability built-in. It also shows you the stack involved with a particular event, which can be useful for diagnosing the otherwise black box that is Windows. Eg [1] [1]: https://github.com/msys2/MSYS2-packages/issues/138#issuecomment-396447222 https://github.com/msys2/MSYS2-packages/issues/138#issuecomm...
- eps 5y ago
  The "filtering capability" actually makes up a good half of its worth. Being able to very quickly drill down to an annotated call stack for a specific IRQ on a specific file is really handy.
  - Hello71 5y ago
    > call stack strace -k > specific IRQ IRQs are very low level and can't be traced by strace or procmon. both can easily log specific syscalls, e.g. strace -e. strace can't quite filter on ioctl, but apparently procmon can't trace ioctl at all: https://stackoverflow.com/questions/9947933/how-to-log-the-deviceiocontrol-calls-of-a-program-on-windows https://stackoverflow.com/questions/9947933/how-to-log-the-d..., so strace is ahead there. > specific file strace -P path
    - Arnavion 5y ago
      >> call stack >strace -k >> specific file >strace -P path That strace can do it too is fine and all, but the Windows ProcMon collects a trace that you can then filter, like Wireshark. So it enables a workflow where you have no idea what's going on, collect a trace, and discover what entry to focus on and what paths to filter for. (This is what I demonstrated in the msys issue link I posted.) An equivalent workflow is possible with strace too of course; pipe `strace` to a file and then discover what paths you want to grep for. It does become a bit noisy when the stack of every syscall is included, and requires more complicated grepping because it spans multiple lines, so you might have to first pre-process it into something structured like JSON.
bboreham 5y ago
Original ProcMon used ETW, Event Tracing for Windows; the analogous technology (although very different in style) on Linux is eBPF so that’s what this tool uses.
- altano 5y ago
  I think you’re mistaken. ProcMon doesn’t use ETW on Windows and I don’t believe it ever did?
  - ale42 5y ago
    Indeed I don't think so. ProcMon uses a kernel driver for the event tracing.
  - bboreham 5y ago
    Sorry about that; I guess I misremembered? This file says it does, though only for network events: https://documentation.help/Process-Monitor/documentation.pdf https://documentation.help/Process-Monitor/documentation.pdf

stefan_ 5y ago

This seems to be using eBPF, so presumably your kernel will have to support that (only stating Ubuntu 18.04 is a bit unfortunate).

unixhero 5y ago
18 is already 3-4 years ago :)
- nemetroid 5y ago
  The version itself is fine, but specifically listing Ubuntu as a system requirement is strange. If Ubuntu truly is a requirement, the name of the project should be "ProcMon for Ubuntu". Presumably it isn't, and the README should list the actual requirements (kernel versions, etc.), with a note to the effect of "packages are available for Ubuntu LTS releases since 18.04".
  - colejohnson66 5y ago
    It could also just mean it hasn’t been tested on anything else. But yes, the README should mention these things.
- deleted 5y ago
  [deleted]

alpb 5y ago

Something that is really cool about Sysinternals tools are that they have a pretty usable GUI (granted, once you need to understand what does each UI icon do). Mimicking this experience in the TUI/terminal would certainly be a challenge. I would love to have a Procmon and ProcessExplorer equivalent with the GUI on Linux and macOS.

a-dub 5y ago
i always thought of the sysinternals tools as like the missing tools that just ship with unix systems. without them, there was no real visibility into open file descriptors, network ports, command lines for running processes, thread trees for processes or any of the rest. the gui was always just the windows way of presenting it. bringing a gui when the rich tools already exist on linux seems ... redundant? (seeing this brings back nightmares from windows admins who refused to install sysinternals tools on production machines)
- themulticaster 5y ago
  This was my reaction as well. I haven't used Process Monitor/Process Explorer for a while, are they bringing anything to the table that's not covered by htop? With htop, I can even start tracing a process (using strace) or list the file opened by a process (using lsof) right inside htop. As might be obvious by now, I think htop has a fantastic TUI that also fits its use case very well. Of course there are other tools that allow deep introspection into what's really going on aside from htop. Another favorite of mine would be perf top, which uses the sampling profiling mechanism to profile the entire system (including the kernel) on a symbol/instruction level. In other words, you don't just see that 30% of CPU time is spent in /usr/bin/cpu_hog, you also see that the time is spent in some_symbol and you can jump right in and see disassemble some_symbol (as long as cpu_hog is not stripped).
  - oblio 5y ago
    > With htop, I can even start tracing a process (using strace) or list the file opened by a process (using lsof) right inside htop. Hmmm... didn't know it could do that.

unixhero 5y ago

Licensed under MIT. This looks like an amazing tool, can't wait to begin using it.

agucova 5y ago

What are the differences between this and htop, by example? I know htop can strace processes with the "s" key.

snickerer 5y ago
This was my first thought, too. Which useful feature does this tool have what htop does not have? Let's say there is such a feature. Why did they not implement it in htop? Made a patch and offered it to upstream? This is why we have FOSS. To benefit from each other's work.
- tuwtuwtuwtuw 5y ago
  > Why did they not implement it in htop? Made a patch and offered it to upstream? This is why we have FOSS. To benefit from each other's work. This is why so much open source software is crap and without any structure. Htop is very different from proc on. Integrating proc on into Htop would make zero sense. Why would you suggest this would be a good idea without knowing what the software does?
- fomine3 5y ago
  Finally Vi become emacs-evil.

rishabhd 5y ago

This will come in handy during DFIR, although we have different workflows for doing live analysis on machines for now.

mraza007 5y ago

Just out of curiosity, How is this tool different when compared to htop

Grayskull 5y ago

Even though I don't use procmon on windows basically never. Sysinternals on linux could be useful especially with same gui as on windows. It would be great to see autoruns and tcpview on linux.

boomskats 5y ago

I've been looking at a few eBPF tracing implementations around GitHub recently to try and get my head round it, and this codebase has to be one of the best documented (if not best written) examples out there. Kudos.

gtirloni 5y ago

Because Ubuntu 18.04 is used for the development environment, it relies on pretty old dependencies. Bug report: https://github.com/Sysinternals/ProcMon-for-Linux/issues/44 https://github.com/Sysinternals/ProcMon-for-Linux/issues/44

JeremyNT 5y ago
Yeah, if you have a modern distribution, look elsewhere. For a "preview" release it's interesting that they're only targeting older distros - maybe they're focused on enterprise use.

whalesalad 5y ago

Sorta related: If y’all haven’t used netdata - wow - it blew my socks off last night. I have it installed on all my hosts now and it’s already helping me diagnose a redis bgsave issue that I didn’t realize existed. Really well put together software.

odyslam 5y ago
Hey, I am odysseas, DevRel at Netdata. I would love to hear your story and what Netdata helped you diagnose in your systems. Hit me up (odyslam) in our forums: https://community.netdata.cloud https://community.netdata.cloud
PanosJee 5y ago
Why do you think most people don't know about netdata and stick with the "status quo" of gathering data -> timeseries db -> manually creating dashboards & configure alerts?

mnutt 5y ago

While I see a lot of comparisons to htop, I wonder how this compares to sysdig? I’ve found sysdig to be an invaluable swiss army knife of debugging performance issues; the only difficult part is finding the right documentation among the cloud offering of the same name.

aasasd 5y ago

Sysinternals is under Microsoft since 2006. Is this the ‘extend’ stage?

superkuh 5y ago

Why are libllvm6.0 llvm-6.0-dev and libclang-6.0-dev needed for this?

nemetroid 5y ago
Seemingly because they download and build bcc as part of the build process. As to why they do that... probably because the three year old distro they're targeting has a three year old version of bcc.

throwaway823882 5y ago

I never thought I'd say this, but: Thank you, Microsoft, for making useful, user-friendly software.