4 ms·

I wrote some code to try and do this a while ago, 'exmap': https://github.com/jbert/exmap https://github.com/jbert/exmap The basic idea was a linux kernel modu

by jbert 14y ago

I wrote some code to try and do this a while ago, 'exmap': https://github.com/jbert/exmap https://github.com/jbert/exmap The basic idea was a linux kernel module which allowed you to get the underlying page id behind each VMA of any process. Putting this info together (and hoping it wasn't changing too much as you queried different processes) allowed me to calculate "effective resident" and "effective vm size", where I used the rule that a page shared across N processes contributed pagesize/N to each one. I could also give those same figures for each ELF section and ELF symbol (via some very fragile code which looked at the process mapping and read the ELF details to work out which bit of memory belonged to each section/symbol). The code has long since rotted and the architecture sucked (I tied the data gathering code into the UI). Would anyone be interested in reviving it/helping me revive it? I consider doing so from time to time, but I feel that codebase sucks now so avoid touching it.

antirez 14y ago
You can do this just via /proc using "maps" and "pagemap" files on recent Linux kernels.
- jbert 14y ago
  Nice! /proc/<pid>/pagemap is basically an alternate to my kernel module. So I could do a pure userspace solution now. Can anyone recommend visualisation tools using pagemap? I'd like to get the exmap functionality (per-process 'effective usage' and also ELF-section/symbol resolution. If that already exists great - if not maybe I can enhance an existing pagemap tool.
  - morsch 14y ago
    This is a visualisation tool by the guy who implemented pagemap: http://www.selenic.com/smem/ http://www.selenic.com/smem/ Unlike existing tools, smem can report proportional set size (PSS), which is a more meaningful representation of the amount of memory used by libraries and applications in a virtual memory system. Because large portions of physical memory are typically shared among multiple applications, the standard measure of memory usage known as resident set size (RSS) will significantly overestimate memory usage. PSS instead measures each application's "fair share" of each shared area to give a realistic measure. Here's some history behind it: http://lwn.net/Articles/329458/ http://lwn.net/Articles/329458/
primitur 14y ago
I was a big user of exmap a few years ago at a previous employer (THALES), where exmap was used to do a lot of things to systems being tested/certified for CENELEC SIL-4 (safety critical) usage. I wrote portions of the online testing systems, and the full system tests for these components used exmap in our test systems to corrupt text segments, copy stuff where it shouldn't be, etc. exmap was essential to testing. PM me privately your details and we can talk about what you want to do with exmap; at the very least I can explain to you how its being used in an industrial testing context, and there may be people still working there, maintaining exmap code, that would be interested in your situation, who nevertheless do not read n.yc.
- jbert 14y ago
  As far as I know, there isn't a pm facility on HN and I don't see any contact info in your profile. My email details are in my profile, if you have time to share some details of how exmap was/is used, I'd be grateful.

by jbert 14y ago

antirez 14y ago

You can do this just via /proc using "maps" and "pagemap" files on recent Linux kernels.

jbert 14y ago
Nice! /proc/<pid>/pagemap is basically an alternate to my kernel module. So I could do a pure userspace solution now. Can anyone recommend visualisation tools using pagemap? I'd like to get the exmap functionality (per-process 'effective usage' and also ELF-section/symbol resolution. If that already exists great - if not maybe I can enhance an existing pagemap tool.
- morsch 14y ago
  This is a visualisation tool by the guy who implemented pagemap: http://www.selenic.com/smem/ http://www.selenic.com/smem/ Unlike existing tools, smem can report proportional set size (PSS), which is a more meaningful representation of the amount of memory used by libraries and applications in a virtual memory system. Because large portions of physical memory are typically shared among multiple applications, the standard measure of memory usage known as resident set size (RSS) will significantly overestimate memory usage. PSS instead measures each application's "fair share" of each shared area to give a realistic measure. Here's some history behind it: http://lwn.net/Articles/329458/ http://lwn.net/Articles/329458/

primitur 14y ago

I was a big user of exmap a few years ago at a previous employer (THALES), where exmap was used to do a lot of things to systems being tested/certified for CENELEC SIL-4 (safety critical) usage. I wrote portions of the online testing systems, and the full system tests for these components used exmap in our test systems to corrupt text segments, copy stuff where it shouldn't be, etc. exmap was essential to testing. PM me privately your details and we can talk about what you want to do with exmap; at the very least I can explain to you how its being used in an industrial testing context, and there may be people still working there, maintaining exmap code, that would be interested in your situation, who nevertheless do not read n.yc.

jbert 14y ago
As far as I know, there isn't a pm facility on HN and I don't see any contact info in your profile. My email details are in my profile, if you have time to share some details of how exmap was/is used, I'd be grateful.