11 ms·
32 MiB Working Sets on a 64 GiB machine
- Varriount 3y agoI found this paragraph particularly interesting: > Trimming the working set of a process doesn’t actually save memory. It just moves the memory from the working set of the process to the standby list. Then, if the system is under memory pressure the pages in the standby list are eligible to be compressed, or discarded (if unmodified and backed by a file), or written to the page file. But “eligible” is doing a lot of heavy lifting in that sentence. The OS doesn’t immediately do anything with the page, generally speaking. And, if the system has gobs of free and available memory then it may never do anything with the page, making the trimming pointless. The memory isn’t “saved”, it’s just moved from one list to another. It’s the digital equivalent of paper shuffling. I'd always been under the impression that as soon as memory was trimmed from the working set. Perhaps this was the case at some point, and was a reason for the PROCESS_MODE_BACKGROUND_BEGIN priority? As the blog mentions, the SetPriorityClass call has had this behavior since at least 2015, though I wouldn't be surprised if this behavior has existed for much longer. As for why this "bug" hasn't been fixed, my guess is that it's due to a couple of factors: - Windows has become fairly good over the years at keeping the core UI responsive even when the system is under heavy load. - There are plenty of ways to reduce memory/CPU usage that don't involve a call to SetPriorityClass. I'd wager that setting a process's priority class is not the first thing that would come to mind. - As a result of the previous two points, the actual number of programs using that call is quite small. I'd actually be interested in knowing what, if any, parts of Windows use it. (As a side note, if there was a bug in a Windows API function, how would you even report it?
- Bjartr 3y ago> Windows has become fairly good over the years at keeping the core UI responsive even when the system is under heavy load. I would say that's debatable. There are fewer complete freezes that require a restart, but things like the task manager (can't get much more core than that), which used to be instantly available and responsive if the machine was recoverable at all, now can take tens of seconds to show up and and respond to interactions under heavy load.
- ComputerGuru 3y agoLaunching it no longer uses a software interrupt request.
- mrguyorama 3y agoI'm not sure I understand, why not just manage your own memory instead of asking the OS to magically understand your memory use-case and properly manage/limit? If the memory isn't owned by the OS, that's probably the wrong actor to manage it. They say that, instead, they used an undocumented API they obviously didn't understand. Why? If you are concerned about using up too much memory, then do something about it
- remram 3y agoWhat's undocumented?
- mrguyorama 3y ago>They didn’t document this (!) Is the fourth line of the article linked.
- tiagod 3y agoThe article makes it clear that the API is documented. The 32MiB limit isn't.
- fluoridation 3y agoTo be fair, the limit is part of the API's semantics. If the limit is undocumented, the API is at least partly undocumented. It's as if a car had a headlights button that the manual had this to say about it: The headlights button turns on the headlights and the headlights indicator light on the dashboard. However, nothing in the manual states that the headlights indicator shares a circuit with a valve that drains the fuel tank, so in effect when you turn the headlights on, you're causing the fuel to leak. I would say the button is not properly documented in that case.
- Dylan16807 3y agoYou're stretching too far in your attempt to be fair. When people talk about using undocumented APIs in windows, that means secret APIs, not badly documented APIs. That is the only meaning that fits the accusatory tone of the comment.
- tedunangst 3y agoI feel like there must be another part to the story. How did only one chrome user notice this?
- remram 3y agoI mean, it's only the installer. I know I only run it once per machine, usually right in the middle of installing all the rest of my usual utilities. It would be easy to miss that it's much too slow. It's not like I have a good sense of what it does anyway, in addition to copying files for a fraction of a second.
- mook 3y agoOnly one Chrome user successfully reached a developer in a position to do something about it. Who knows how many encountered it but didn't notice, and how many more noticed it but couldn't find a way to file a bug that would get noticed.
- brucedawson 3y agoIt's a good question. It could be that most people don't notice this things, those who don't don't report it, etc. Or maybe it only affects some people, but I can't think how that could be true. More mysteries
- brucedawson 3y agoI'm running some monitoring that will tell me how setup.exe behaves on my machines. I tested it by forcing an update from Chrome. The private commit and private working set were 66.76 and 61.70 MiB, so no capping, but that makes sense because a user-initiated update isn't running at low priority, so we don't trigger background mode. So, the only time the bug happens is for background updates that may be happening when the user isn't even present. And, since the CPU priority is low they won't harm responsiveness of foreground applications. But they will waste CPU time and electricity and battery life (if on battery). So, a silent waste.
- john567 3y agoI can tell you exactly how I noticed it because it was on my workstation this reproduced originally. I have a Ryzen 9 5950X CPU and it has a crazy fan ramp. Anytime a process maxes out the CPU it will be audible to me when I'm working. Now, to put this in perspective. It doesn't happen when I'm gaming. I can play Diablo 4 and Cyberpunk 2077 without this noticable fan noise but when setup.exe loses it, the fan noise is how I notice it. It will max out one core and I'm going to assume "never" complete. The longest I waited was 78 CPU minutes before killing the process. This would happen now and then but it would not prevent Chrome from successfully updating. So, it was bizarre to begin with.
- Borg3 3y agoHaha :) I just checked MSDN docs and vioala: PROCESS_MODE_BACKGROUND_BEGIN - Not supported under Win2003 or Windows XP. Another improvement from Microsoft without really understanding their own OS.
- jacobgorm 3y agoI think they understood their OS and memory model better than most commenters in this thread do. I was once paid to read most of Windows Internals 5 and try to understand their memory management concepts, some of which date back to VMS, and most of them actually make a lot of sense. The Background mode has its merits too, but I guess they never imagined that a background process would need to spend huge amounts of memory just to do some background work.
- Filligree 3y agoHuge? The limit is 32MiB. That isn’t large by any measure.
- Borg3 3y agoYeah, especially it was intruduced with Win7 AFAIR. So processes were already pretty inflated ;)
- jacobgorm 3y agoWin7 was supposed to be usable with 1GiB of RAM, so 32MiB for some low priority background task seems quite reasonable. https://support.microsoft.com/en-us/windows/windows-7-system-requirements-df0900f2-3513-a851-13e7-0d50bc24e15f https://support.microsoft.com/en-us/windows/windows-7-system...
- Dylan16807 3y agoI wouldn't really count the 32 bit version, but either way windows 10 has the exact same requirements as windows 7. Actually 7 was higher by a gigabyte if you wanted XP mode! So if you agree this would have been a bad limit in 2020, then that suggests it also would have been a bad limit in 2010.
- chrisbolt 3y agoPrevious discussion (2 days ago): https://news.ycombinator.com/item?id=37734685 https://news.ycombinator.com/item?id=37734685
- dang 3y agoThat one never made the front page so maybe we'll move the comments hither and let this one run. (It's on my list to do more for submissions like that which accrue a bunch of upvotes slowly but never break the front page...)
- remram 3y agoLooks like you did the move. It's really weird because it changed the timestamps on all the comments... They are even different between my "threads" page and what I see when I click through.
- dang 3y agoYes, sorry, I know it's confusing. I relativized the timestamps for 2 reasons: (1) if we didn't do that, commenters would start asking "why are there a bunch of comments here older than the OP they're commenting on?" - and our experience is that this leads to more confusion than the other way around; plus (2) it seems unfair to have all the comments on the earlier thread plummet to the bottom of the new thread because they're so much older. This topic comes up regularly in a slightly different context, which is re-upping stories that make it into the second-chance pool (https://news.ycombinator.com/item?id=26998308 https://news.ycombinator.com/item?id=26998308). There are a bunch of past explanations here in case helpful: https://hn.algolia.com/?dateRange=all&page=0&prefix=true&query=by%3Adang%20timestamps%20re-up&sort=byDate&type=comment https://hn.algolia.com/?dateRange=all&page=0&prefix=true&que....
- jrmg 3y agoTrimming the working set of a process doesn’t actually save memory. It just moves the memory from the working set of the process to the standby list. Then, if the system is under memory pressure the pages in the standby list are eligible to be compressed, or discarded (if unmodified and backed by a file), or written to the page file. [… If] the system has gobs of free and available memory then it may never do anything with the page, making the trimming pointless. The memory isn’t “saved”, it’s just moved from one list to another. It’s the digital equivalent of paper shuffling. This sounds like it should be basically free in a memory-unconstrained system - is the system really just spending all its time managing these lists? Why is it so expensive?
- tetha 3y agoI don't know about Windows, but the Linux kernel is entirely fine to just keep rarely used memory pages around as long as it doesn't need the memory. If you don't need the memory otherwise, having the page loaded and cached is better than needing to load it. Even though, then you end up with confused reports about "Linux eating all the memory" and "an idle system using all the memory (for caching)". AFAIK, newer kernels also have a preemptive swap-out of idle pages so they can evict the pages quicker, so an idle system might even be swapping (out). This results in even more confusion.
- zamalek 3y ago> "Linux eating all the memory" and "an idle system using all the memory (for caching)". I find these reports bewildering (alongside CPU usage). You would hope that something would be exploiting your expensive machine to the fullest (so long as the work being done is useful/intentional/desired - obviously not wasteful page thrashing).
- andrepd 3y agoThere's the power consumption issue, of course.
- 3y ago
- leptons 3y ago..
- HideousKojima 3y ago> don't know of any computers that have 64 GiB of RAM. It's the other way around, most computers have ram in gibibytes but list it as gigabytes. It's mostly hard drive and other storage manufacturers that treat gigabytes as actual gigabytes as a way to skimp out
- Dylan16807 3y agoIf you want a powershell command to check your memory capacity, try (Get-CimInstance Win32_PhysicalMemory).Capacity My RAM sticks each have 8589934592 = 2^33 bytes.
- jacobgorm 3y agoBut why would Chrome's setup.exe need a 32MiB working set? If all it does is download and unpack some files in the background, it would seem like something that is doable in 32MiB or less. The Windows BACKGROUND mode is useful for stuff like virus scanning or database log compaction, where you want to make some progress, but only if you can be certain not to hurt any foreground workloads. Back when we had hard drives, this wasn't actually trivial to achieve, because it is very easy to interfere with the foreground workload just by doing a few disk seeks every now and then. I agree the 32MiB working setlimit is somewhat arbitrary and should be documented, but Windows is full of these arbitrary constants, like the 32kiB paging chunk for instance. My recommendation for Chrome would be to stick with the background mode, and fix whatever problem is causing the working set to exceed 32MiB.
- toast0 3y agoThe offline installer reports as 103 mb, depending on the compression settings, it might need a significant amount of that mapped in at once.
- jacobgorm 3y agoProbably because the files to be binary patched are now quite large, and are loaded into memory verbatim. Perhaps they could memory-map them instead, and ensure that their binary diffing tool works from left to right to avoid thrashing?
- Dylan16807 3y agoHow do you know it's not memory mapped? Working left to right means you have to force the compiler to always align the code in the same order, which sounds difficult and like a waste of time.
- jacobgorm 3y agoNo.
- criddell 3y agoBruce - any chance you could make your ETW videos available again? I see they used to be available through a Microsoft partnership of some type but that seems to have ended a few years ago. I’d love to learn more about ETW and your videos seem like a good place to start. If anybody else has other recommendations, please share!
- a1o 3y agoETW is Event Tracing for Windows if someone reads the above and is wondering.
- criddell 3y agoThanks. And the videos I’m talking about are linked from here: https://randomascii.wordpress.com/2014/08/19/etw-training-videos-available-now/ https://randomascii.wordpress.com/2014/08/19/etw-training-vi...
- brucedawson 3y agoSorry, I don't own them so there's nothing I can do, unless I can buy them back
- criddell 3y agoMakes sense. Thanks for sharing all that you have. It's an incredible resource.
- Majromax 3y agoI wonder if this is some cross-pollination legacy of Windows Phone. On such a device, background processes are still necessary, memory is a much tighter constraint, and enough pages might be mapped directly from device flash for auto-eviction to make sense.
- toast0 3y agoAs a Windows Phone fan, the timing doesn't quite match up. They reported this happens as far back as Windows 7, but Windows Phone 7 was based on Windows CE; WP 8 was NT based, I would have expected cross-pollination to start showing up in Windows 8.
- sznio 3y agoI remember using some software to increase the pagefile on my Windows Phone. Multitasking worked so much better after that. Apps loaded faster since it would just take the image of an app suspended in the pagefile rather than starting it over.
- amluto 3y agoSeriously? Windows has had this bizarre “working set” concept effectively forever. I thought it was weird when Windows 2000 was new, and I still think it’s weird. In the land of page replacement algorithms, there are many of various degrees of cleverness, and then there’s “we exceeded an arbitrarily limit — nothing else needed that memory, but we’re going to zap it out of the page tables anyway!” (x86 has had “accessed” tracking for a long long time. What would be wrong with keeping over-the-working-set pages present but not “accessed” and tracking accesses without causing page faults? Or at least only trying to expire pages when there’s some degree of memory pressure.)
- deleted 3y ago[deleted]
- NavinF 3y agoWow, PROCESS_MODE_BACKGROUND_BEGIN is quite the footgun. I wonder if this started out as a hack to reduce the impact of OEM bloatware on benchmarks. Is there an easy way to list all processes that use this priority class?
- brucedawson 3y agoThere probably is a way to find what processes have this mode enabled, but it's probably more productive to search github for references to the problematic constant.
- fluoridation 3y agoRun Process Explorer and enable the Memory Priority column. In the Select Columns dialog, its checkbox is on the Process Memory tab. Priority 5 is normal; priority 1 is background.
- kristianp 3y agoSo he didn't report the bug to Microsoft? His final comments were just to say: "This issue has been known for eight years, on many versions of Windows, and it still hasn’t been corrected or even documented. I hope that changes now."
- brucedawson 3y agoI reported the documentation bug. I don't use Feedback Hub because it never works for this type of issue. I know that some Microsoft developers will see this blog post and I hope that internal bugs get filed. But ultimately as a non-paid tester of Windows I'm under no obligation to report non-security bugs in any particular way. I like reporting them through twitter and blog posts.
- theolivenbaum 3y agoYour blog is the most effective bug reporting tool for Windows issues. Thanks for keeping it going!
- _a_a_a_ 3y agoI reported a probably Windows perf bug. Nothing happened. I reported it again. Basically they weren't interested despite my careful spelling it out. Time wasted. Fuck them. Maybe it wasn't a bug (hah!) but they could have come back to me and at least told me they'd looked at it and it wasn't. I'd have appreciated that. But no.
- dabbz 3y agoBack when I worked on Windows, we would triage Windows Feedback hub reported issues once a sprint (~2 weeks). But only those which had at least 2-5 upvotes on them. One-off bugs usually meant they weren't being widespread impactful and weren't as higher priority on our list (even if that isn't the case). But the feedback hub is an abyss of random complaints to filter through. Sometimes we'd see items like "Windows wouldn't save my word document and now I hate windows" and being on the networking team were like "uhhh thanks"
- 3y ago