3 ms·

Measuring memory usage is hard on a modern OS. Not in the slightest. OS's have to allocate virtual memory to each process or that process with SIG-FAULT when i

Measuring memory usage is hard on a modern OS. Not in the slightest. OS's have to allocate virtual memory to each process or that process with SIG-FAULT when it touches invalid memory. Tracking per-process memory usage is one of the most trivial things OS/Pref tools do.

mikeash 10y ago
Measuring it is easy. Defining it is what's hard. If three processes each have the same 10MB executable mapped into memory, plus 10MB of heap data, how much memory is each process using? There's no easy answer to that question.
- valarauca1 10y ago
  Defining it is what's hard. how much memory is each process using? 20MB. Oh but they are sharing the pages the executable is mapped into. So how can you can you say they are each using that much RAM? Because that 10MB of RAM won't be freed until the last process exits. Each process is using 20MB of physical pages. If only 1 was running it would use 20MB. The semantics that each process is sharing 5 HUGEPAGES of heap isn't really important as even if 1 was running alone it uses that much memory. If you are sharing that much data with executables that you have this problem there is some low hanging fruit for memory optimizations you can do. Oh but you aren't *actually* using 60MB of Heap, only 40MB Indeed. And in cases where these 3 children have the same parent you look at the ParentMem + 40MB (child binary+individual heaps) to report a better picture. Shared/unshared semantics are not hard. Have you ever read /proc/$PID/maps ? Because it clearly labels what is/isn't shared physically. When you are optimizing for memory usage you rarely look at shared readonly pages as the kernel put them there for you. Defining it is what's hard. Injecting an interpretation before taking a measurement is not sound science or engineering. Measurements inform Interpretation. Not vice versa.
  - mikeash 10y ago
    Sure, shared/unshared semantics are not hard. How do you map that to a generic concept of "usage"? There's no right way to do it. It doesn't really make sense to ask "how much memory is this process using?" Instead, you want to ask questions like "how much memory are all of these processes using?" or "how much memory would be freed if I terminate these processes?" or "how much extra memory will be used if I start another process?" What's hard is understanding which question you want to ask in any given situation.
    - valarauca1 10y ago
      There's no right way to do it. What's hard is understanding which question you want to ask in any given situation. I don't see the issue. You admit that answering all of these questions is easy. The hard part is lawyering which question to ask. This is not a data reporting problem. This is a data interpretation problem. A good reporting tool should not be interpreting (or forcing an opinion) on the end user. You believe this problem is hard because you are forcing an interpretation to exist before you take a measurement. When you should not be interpreting until after measurement. Injecting an interpretation into a measurement process fundamentally changes that measurement. Measurements inform your interpretation. Not the opposite. TL;DR You put the cart in front of the horse.
      - mikeash 10y ago
        "This is a data interpretation problem." That's what I said. "Defining it is what's hard." You seem to understand what I'm saying, and agree with me, but insist on arguing for some reason I cannot comprehend.
        valarauca1 10y ago
        You seem to understand what I'm saying and agree with me Literally the opposite. Your initial statement was Measuring memory usage is hard on a modern OS. Which you've agreed was false as /proc/$PID/maps is easy to read. Then said: Defining it is what's hard But these statements are not mutually inclusive nor does the later imply the former. I stated measuring is literally the simplest thing. You said ah but what you measuring. That isn't dependent on the tools, or OS. So why did you state it was?
- TeMPOraL 10y ago
  Yeah, it depends on what exactly you're asking for. But one can provide different visualizations for different questions. As a typical computer user, this is what I care about: - who the frak is eating all my RAM? - which processes should I kill to get some memory back? Whatever the weird sharing things are going on, the memory is ultimately linear. So show me a large rectangle subdivided into sections, each one representing memory associated with a set of processes. One section for "this memory is used by Chrome.exe". One for "this memory is used by Chrome.exe, X.exe and Y.exe". One for "this memory is free". When I click on a section, it should highlight all the processes that share this memory. When I click on a process, it should highlight all the sections this process uses or shares with others. This way, I could quickly see who's wasting my RAM and how many processes do I need to kill to get some chunk of memory back.

mikeash 10y ago

Measuring it is easy. Defining it is what's hard. If three processes each have the same 10MB executable mapped into memory, plus 10MB of heap data, how much memory is each process using? There's no easy answer to that question.

valarauca1 10y ago
Defining it is what's hard. how much memory is each process using? 20MB. Oh but they are sharing the pages the executable is mapped into. So how can you can you say they are each using that much RAM? Because that 10MB of RAM won't be freed until the last process exits. Each process is using 20MB of physical pages. If only 1 was running it would use 20MB. The semantics that each process is sharing 5 HUGEPAGES of heap isn't really important as even if 1 was running alone it uses that much memory. If you are sharing that much data with executables that you have this problem there is some low hanging fruit for memory optimizations you can do. Oh but you aren't *actually* using 60MB of Heap, only 40MB Indeed. And in cases where these 3 children have the same parent you look at the ParentMem + 40MB (child binary+individual heaps) to report a better picture. Shared/unshared semantics are not hard. Have you ever read /proc/$PID/maps ? Because it clearly labels what is/isn't shared physically. When you are optimizing for memory usage you rarely look at shared readonly pages as the kernel put them there for you. Defining it is what's hard. Injecting an interpretation before taking a measurement is not sound science or engineering. Measurements inform Interpretation. Not vice versa.
- mikeash 10y ago
  Sure, shared/unshared semantics are not hard. How do you map that to a generic concept of "usage"? There's no right way to do it. It doesn't really make sense to ask "how much memory is this process using?" Instead, you want to ask questions like "how much memory are all of these processes using?" or "how much memory would be freed if I terminate these processes?" or "how much extra memory will be used if I start another process?" What's hard is understanding which question you want to ask in any given situation.
  - valarauca1 10y ago
    There's no right way to do it. What's hard is understanding which question you want to ask in any given situation. I don't see the issue. You admit that answering all of these questions is easy. The hard part is lawyering which question to ask. This is not a data reporting problem. This is a data interpretation problem. A good reporting tool should not be interpreting (or forcing an opinion) on the end user. You believe this problem is hard because you are forcing an interpretation to exist before you take a measurement. When you should not be interpreting until after measurement. Injecting an interpretation into a measurement process fundamentally changes that measurement. Measurements inform your interpretation. Not the opposite. TL;DR You put the cart in front of the horse.
    - mikeash 10y ago
      "This is a data interpretation problem." That's what I said. "Defining it is what's hard." You seem to understand what I'm saying, and agree with me, but insist on arguing for some reason I cannot comprehend.
      - valarauca1 10y ago
        You seem to understand what I'm saying and agree with me Literally the opposite. Your initial statement was Measuring memory usage is hard on a modern OS. Which you've agreed was false as /proc/$PID/maps is easy to read. Then said: Defining it is what's hard But these statements are not mutually inclusive nor does the later imply the former. I stated measuring is literally the simplest thing. You said ah but what you measuring. That isn't dependent on the tools, or OS. So why did you state it was?
TeMPOraL 10y ago
Yeah, it depends on what exactly you're asking for. But one can provide different visualizations for different questions. As a typical computer user, this is what I care about: - who the frak is eating all my RAM? - which processes should I kill to get some memory back? Whatever the weird sharing things are going on, the memory is ultimately linear. So show me a large rectangle subdivided into sections, each one representing memory associated with a set of processes. One section for "this memory is used by Chrome.exe". One for "this memory is used by Chrome.exe, X.exe and Y.exe". One for "this memory is free". When I click on a section, it should highlight all the processes that share this memory. When I click on a process, it should highlight all the sections this process uses or shares with others. This way, I could quickly see who's wasting my RAM and how many processes do I need to kill to get some chunk of memory back.