3 ms·

Ok, I made the comment in http://news.ycombinator.com/item?id=1138840 http://news.ycombinator.com/item?id=1138840, that "it's surprising that the CTO of a compa

by apower 17y ago

Ok, I made the comment in http://news.ycombinator.com/item?id=1138840 http://news.ycombinator.com/item?id=1138840, that "it's surprising that the CTO of a company specialized on system performance doesn't know how OS cache memory works." It turns out that he really has no clue. To the people who said the Windows source code is needed to understand how it works, hogwash. The file caching in Linux has been doing it this way for years. Windows does it for years as well. It's fairly standard OS caching technique.

barrkel 17y ago
The file system cache doesn't contribute to committed bytes, the metric that the Exo CTO is appealing to - in this respect the Ars Technica article is mistaken. (Actually, the fact that the Ars Technica article used Task Manager made me inclined to dismiss it as clueless.) Estimating real memory pressure is tough. There's the working set; but the working set may expand when you have sufficient physical memory, even though the marginal performance gains may be small. Similarly, the file system cache may expand when you have sufficient physical memory, but the marginal gains become small there too. Probably the best unified metric would be the amount of time applications spend blocked on I/O, where that I/O is either paging committed bytes in from disk, or data that would be cacheable - and would have been cached - on a system with more memory. But that's extremely hard to predict.
- sid0 17y ago
  If you've spent any amount of time at the Ars forums, you wouldn't refer to Peter as "clueless". Committed bytes is completely the wrong metric to appeal to, since it includes pages that have been committed but have never been touched. Windows reserves space for such pages in the page file (it never overcommits), but saying that it leads to "memory pressure" is ridiculous.
  - mhansen 17y ago
    If you've spent any amount of time at the Ars forums, you wouldn't refer to Peter as "clueless". Please explain, for the (majority?) of us who don't spend time at the Ars Forums
  - barrkel 17y ago
    I wrote off the article, rather than the author. If I indeed read more of the author, perhaps my opinion would be updated, but the article on its own doesn't add much weight of authority. The trouble with Task Manager is that it doesn't have sufficient per-process memory data to be useful. Process Explorer is better, because you can view Working Set (Private|Shareable|Shared) and Private Bytes. I'd view both private working set and private bytes as the most important metrics for a Windows app, but depending on the circumstances (such as how likely multiple instances are) other things may be considered. Committed Bytes would be a measure of potential maximum physical memory usage if every process were touching every page that it had available to it. Yes, no process ever does that, but statistically there will be an average fraction of that memory which is touched, so it can be interpreted as a proxy for physical memory pressure. A poor one though, as different processes have different patterns. As to the Ars forums, with a brief perusal it seems similar to the programming Reddit - i.e. a lot of opinionated and frequently misinformed noise, interleaved with a small amount of signal. It looks like a time sink, to be frank. But to give Peter his due, I registered with the forums and scanned through his past posts (DrPizza), and I can't find much that I disagree with; I particularly agree with him on the consolification of PC games, and the lack of MW2's appeal to a mature gamer, etc.
    - sid0 17y ago
      Peter did admit his mistake about superfetch (he thought the xpnet thing was counting that too as used memory), but he's still right about the overall metric being wrong -- see the comments to http://exo-blog.blogspot.com/2010/02/editorial-what-took-you-so-long.html http://exo-blog.blogspot.com/2010/02/editorial-what-took-you... . edit: and do you think he doesn't know about procexp/perfmon? :)
      - barrkel 17y ago
        Why do you insist on making this personal? I discounted an article, not a person. By bringing the person into this, you turn it into an irrational, emotional thing, where facts don't matter. That's not where you want to be. And at the risk of extending this thing far beyond its worthy lifetime, that comment you link to proves very little. As I already said: > Committed Bytes would be a measure of potential maximum physical memory usage if every process were touching every page that it had available to it. Yes, no process ever does that, but statistically there will be an average fraction of that memory which is touched, so it can be interpreted as a proxy for physical memory pressure. A poor one though, as different processes have different patterns. Allocating a huge bunch of memory and not touching it is one statistical pattern, but it's not particularly common, which is key. It's an outlier, so it isn't strong evidence. On average, committed bytes can be interpreted as a proxy for memory pressure, but like I said, there's likely to be a fair amount of variance there, so it's a poor proxy. There certainly wouldn't be a 1:1 relationship between committed memory and physical memory pressure; if I had to guess, I would guess somewhere like 60% of committed memory should ideally be in physical ram. For example, my machine has 12GB of RAM, 3.5GB committed bytes total and 2.5GB working set total. But that working set includes both private and shared working set pages - calculating the active working set properly would require dividing shared page sizes by the number of copies in different processes. But just for fun, I plotted the log of working set values in MB along the X axis, and the log of private bytes in MB along the Y axis, for each process, and this is what I got: http://imgurl.filetac.com/img/68343980.png http://imgurl.filetac.com/img/68343980.png Looks like a pretty close correlation to me - yes, there are some spikes in private bytes, but on average, private bytes is a proxy for committed bytes (I excluded shared pages for confounding reasons, as they are usually things like EXEs and DLLs - not counted against commit charge), committed bytes is a proxy for physical memory usage, which in turn is a proxy for physical memory pressure.
  - barrkel 17y ago
    It looks like Peter was right on the money, in terms of the guy being a sham. Very interesting examination here: http://blogs.zdnet.com/BTL/?p=31024&page=1 http://blogs.zdnet.com/BTL/?p=31024&page=1 FWIW, I never downvoted you - I actually upvoted you when I saw you dropped to 0.
sid0 17y ago
No, a large part of file caching in Vista and 7 is preemptive -- Linux doesn't do it by default (I think there's a utility called preload that does it).

by apower 17y ago

barrkel 17y ago

The file system cache doesn't contribute to committed bytes, the metric that the Exo CTO is appealing to - in this respect the Ars Technica article is mistaken. (Actually, the fact that the Ars Technica article used Task Manager made me inclined to dismiss it as clueless.) Estimating real memory pressure is tough. There's the working set; but the working set may expand when you have sufficient physical memory, even though the marginal performance gains may be small. Similarly, the file system cache may expand when you have sufficient physical memory, but the marginal gains become small there too. Probably the best unified metric would be the amount of time applications spend blocked on I/O, where that I/O is either paging committed bytes in from disk, or data that would be cacheable - and would have been cached - on a system with more memory. But that's extremely hard to predict.

sid0 17y ago
If you've spent any amount of time at the Ars forums, you wouldn't refer to Peter as "clueless". Committed bytes is completely the wrong metric to appeal to, since it includes pages that have been committed but have never been touched. Windows reserves space for such pages in the page file (it never overcommits), but saying that it leads to "memory pressure" is ridiculous.
- mhansen 17y ago
  If you've spent any amount of time at the Ars forums, you wouldn't refer to Peter as "clueless". Please explain, for the (majority?) of us who don't spend time at the Ars Forums
- barrkel 17y ago
  I wrote off the article, rather than the author. If I indeed read more of the author, perhaps my opinion would be updated, but the article on its own doesn't add much weight of authority. The trouble with Task Manager is that it doesn't have sufficient per-process memory data to be useful. Process Explorer is better, because you can view Working Set (Private|Shareable|Shared) and Private Bytes. I'd view both private working set and private bytes as the most important metrics for a Windows app, but depending on the circumstances (such as how likely multiple instances are) other things may be considered. Committed Bytes would be a measure of potential maximum physical memory usage if every process were touching every page that it had available to it. Yes, no process ever does that, but statistically there will be an average fraction of that memory which is touched, so it can be interpreted as a proxy for physical memory pressure. A poor one though, as different processes have different patterns. As to the Ars forums, with a brief perusal it seems similar to the programming Reddit - i.e. a lot of opinionated and frequently misinformed noise, interleaved with a small amount of signal. It looks like a time sink, to be frank. But to give Peter his due, I registered with the forums and scanned through his past posts (DrPizza), and I can't find much that I disagree with; I particularly agree with him on the consolification of PC games, and the lack of MW2's appeal to a mature gamer, etc.
  - sid0 17y ago
    Peter did admit his mistake about superfetch (he thought the xpnet thing was counting that too as used memory), but he's still right about the overall metric being wrong -- see the comments to http://exo-blog.blogspot.com/2010/02/editorial-what-took-you-so-long.html http://exo-blog.blogspot.com/2010/02/editorial-what-took-you... . edit: and do you think he doesn't know about procexp/perfmon? :)
    - barrkel 17y ago
      Why do you insist on making this personal? I discounted an article, not a person. By bringing the person into this, you turn it into an irrational, emotional thing, where facts don't matter. That's not where you want to be. And at the risk of extending this thing far beyond its worthy lifetime, that comment you link to proves very little. As I already said: > Committed Bytes would be a measure of potential maximum physical memory usage if every process were touching every page that it had available to it. Yes, no process ever does that, but statistically there will be an average fraction of that memory which is touched, so it can be interpreted as a proxy for physical memory pressure. A poor one though, as different processes have different patterns. Allocating a huge bunch of memory and not touching it is one statistical pattern, but it's not particularly common, which is key. It's an outlier, so it isn't strong evidence. On average, committed bytes can be interpreted as a proxy for memory pressure, but like I said, there's likely to be a fair amount of variance there, so it's a poor proxy. There certainly wouldn't be a 1:1 relationship between committed memory and physical memory pressure; if I had to guess, I would guess somewhere like 60% of committed memory should ideally be in physical ram. For example, my machine has 12GB of RAM, 3.5GB committed bytes total and 2.5GB working set total. But that working set includes both private and shared working set pages - calculating the active working set properly would require dividing shared page sizes by the number of copies in different processes. But just for fun, I plotted the log of working set values in MB along the X axis, and the log of private bytes in MB along the Y axis, for each process, and this is what I got: http://imgurl.filetac.com/img/68343980.png http://imgurl.filetac.com/img/68343980.png Looks like a pretty close correlation to me - yes, there are some spikes in private bytes, but on average, private bytes is a proxy for committed bytes (I excluded shared pages for confounding reasons, as they are usually things like EXEs and DLLs - not counted against commit charge), committed bytes is a proxy for physical memory usage, which in turn is a proxy for physical memory pressure.
- barrkel 17y ago
  It looks like Peter was right on the money, in terms of the guy being a sham. Very interesting examination here: http://blogs.zdnet.com/BTL/?p=31024&page=1 http://blogs.zdnet.com/BTL/?p=31024&page=1 FWIW, I never downvoted you - I actually upvoted you when I saw you dropped to 0.

sid0 17y ago

No, a large part of file caching in Vista and 7 is preemptive -- Linux doesn't do it by default (I think there's a utility called preload that does it).