4 ms·
Finding Windows HANDLE leaks, in Chromium and others
- aneutron 5y agoIt's always a great day where I'm guaranteed to learn something new when randomascii posts something. Such non trivial bugs and debug methods.
- jve 5y agoI appreciate how humble he is: This I understand, this I don't, here I have experience, but not here. Example: "It took months to understand the bug, but then a fix was created fairly promptly (not by me – I still don’t understand that part of the code)" Also nice how the issues are handled by team members, that know more about stuff in their field. Like, maybe other people wouldn't be skilled enough to find the issue, but he wasn't skilled enough to fix it. Learning along the way. When tackling issues, there are always some knowns and unknowns and the experience is gained. Thanks to the way randomascii is written, I'v had some success using WPA.
- Someone1234 5y agoJust enabling the Handles column made me realize that the "Asus Update Service" SysTray Icon/application had a handle leak: as it had 64K handles open. There's also an Asus Lighting Service that still has 10K handles open.
- pixl97 5y agoThis doesn't surprise me at all. Vendor tools, like the ASUS utilities, or Dells update application, and many others are commonly outsourced from different companies and seem to import a string of security flaws. Providing the program, not providing a secure program tends to be the metric for delivery on these applications.
- goldenkey 5y agoJust for others who may not see where the handles column is: "Open task manager, go to details tab, right click on one of the already listed columns, click select columns, then scroll down to check handles." After sorting by handles, I see similar results. Logitech Hub Agent has 13,000 handles. It seems these gaming companies hire the worst coders without care for actual performance. Yet...they advertise their products as cutting edge performance. Quite the irony.
- fullstop 5y agoI get the feeling that gaming PCs are restarted often.
- Arrath 5y agoNot remotely true, in my personal case. I might change that stance when I get home tonight and check for leaks like this, though.
- sumtechguy 5y agoAnother two good utilities to use is performance monitor, and process explorer. Perf mon is good for showing usage over time with its built in graphing and recording.
- taneq 5y ago> It seems these gaming companies hire the worst coders without care for actual performance. Yet...they advertise their products as cutting edge performance. Quite the irony. Unfortunately gaming companies routinely hire inexperienced coders because they're cheap. By the time those coders have skilled up, they've been burned by the gaming industry in one of the many ways that the gaming industry burns coders, and so they no longer work in gamedev.
- bluedino 5y agoIt seemed like in the old days you had to read the Petzold book to learn Windows programming. It actually explained what was going on underneath the hood. And if you wanted more, you'd read something by Jeffrey Richter, David Solomon, or Mark Russinovich. Now? You watch a couple YouTube videos, or perhaps a terrible Apress book. You get something to appear on the page, and viola!, you're a programmer.
- ripley12 5y agoThere’s a real problem here: those are still the books to read if you really want to understand Windows. They’re great books but they’re old, often out of print, and the onus is on the reader to distinguish outdated info from still-relevant info. Yet as far as I can tell, nobody’s publishing better books about Windows programming today. It’s remarkably hard in 2021 for a newbie to start learning Windows programming “properly”.
- no_time 5y agoIt's because MS treats win32 (or how ever you want to call the "legacy" api) like the plague despite it being the past,present and arguably the future. I still can't name a single UWP app that isnt a crude mockery of some win32 app that runs circles around it in both performance and UI design. My favorite example of how this new generation of MS devs handle performance: https://github.com/microsoft/terminal/issues/10362 https://github.com/microsoft/terminal/issues/10362
- ripley12 5y agoI think that’s an unfair characterization of the Terminal issue you linked, and the Terminal developers in general. If you follow the Terminal repo it is clear that the devs are not fans of the “modern” Windows technologies they are forced to work with.
- po1nter 5y agoI just did this and I'm surprised to see DropBox with 19K handles. Even more than Visual Studio at 4.5K handles. Handle type summary: <Unknown type> : 4 <Unknown type> : 2 <Unknown type> : 332 ALPC Port : 19 Desktop : 2 Directory : 4 Event : 395 File : 131 IoCompletion : 55 IRTimer : 11 Job : 3 Key : 104 Mutant : 21 Process : 66 Section : 45 Semaphore : 18109 Thread : 154 TpWorkerFactory : 5 WaitCompletionPacket: 70 WindowStation : 3 Total handles: 19535
- mook 5y ago"Process Hacker" can sometimes display more information as well; for example, I see that igfxEM.exe on my machine (Intel graphics driver) is leaking around 5000 handles to the registry key HLKM\system\ControlSet001\Control\Class\{5c4c3332-344d-483c-8739-259e934c9cc8}\0001 over a week of uptime…
- bigbizisverywyz 5y agoOh I used to love leaky HANDLE hunts. These things could bring windows 3.n (and maybe NT 4.v too?) to its knees pretty quickly so I was always really careful about diligently freeing them. And, like many commenters below constantly surprised that so many apps still ship with these problems, since it's so easy to spot them. Windows perfmon also gives you a nice graph, so you can correlate e.g. GUI behaviour with a jump in handles being created that then never get freed.
- tialaramex 5y agoWhilst it's easy to spot the leak, the fact that you can leak handles by entirely forgetting they exist can make it a little harder for the programmers to find where their code leaked the handle and so fix it. If you leak 6GB of RAM, there's 6GB of evidence about what was leaked exactly. If it's full of terrible love poetry you can rule out "FootBallScores" and focus on the "TeenagePoems" data structure and related code. But if you leak 50 000 HANDLEs then er... oops? The post shows you can narrow it down to say, Event HANDLES but after that it gets increasingly sticky. Hopefully somewhere there's a C++ object that owns the handle and that has leaked which you can trace, but as I understand it, the handles themselves might be all that leaked, leaving you to instrument software so that you can find out which code made all the handles and then trace back those that seem leaked.
- sumtechguy 5y agoOne good rule I used to use and make everyone in my team use was when you get a handle write the 'anti' call at the same time. Good portions of the older windows API is allocate functions and destroy functions. If you write both at the same time your mental overhead is less. many start using 'auto managed' style languages (c++, java, etc) where the life cycle is not as clear. The life cycle is the same though but you own it in an indirect way. Who makes it. Who uses it. Who destroys it. In some languages that is easier to do, others you own it front to back. When doing this I try to start with create/destroy then usage. It is a style that helps remove leaks before they happen. They can still slip in there... I have used the common string thing a few times to help narrow leaks (does not work in all cases :( ). You also can use tools like valgrind, boundchecker, purify, etc. Think MS has a couple that I can not remember off the top of my head.
- nyanpasu64 5y agoOn the topic of leaking HANDLEs, Synergy/Barrier's service leaks zombie processes and causes systems to malfunction over time, and this hasn't been fixed in years. https://github.com/debauchee/barrier/issues/567 https://github.com/debauchee/barrier/issues/567
- brucedawson 5y agoOuch. Zombie handles are so much worse. Now we're talking serious memory, and since it isn't attributed to the process that leaks the handles it is easy to not realize what is happening. More details: as I explain in the first link in the blog post, if a process leaks process handles then roughly 64 KB of memory will be used for each leaked handle (each zombie process) but that memory will not be attributed to any process. If you have the Handles column open you will see a large count in one process, but that process will not have a large memory footprint. If you kill that process you will reclaim the memory from all of the zombies. Good times.
- fassssst 5y agoWe use this internally to help prevent leaks: https://github.com/Microsoft/wil/wiki/RAII-resource-wrappers https://github.com/Microsoft/wil/wiki/RAII-resource-wrappers
- ddulaney 5y agoThat's an awesome set of tools! I've written much jankier versions of these myself, and I'll definitely look into using the official ones. That said, I don't know that they would have helped in the article's case. It looks like there was a higher-level resource getting leaked, which would also most likely leak these.
- mad_vill 5y agoah I didn't know we opensourced wil. sweet.
- minedwiz 5y agoThink it was around the same time as terminal
- tgtweak 5y agoFirst thing I do on a new Windows setup is turn on "handles", "threads", "Commit size", "NP pool" columns in task manager details... If you want to see some real offenders... have a quick look at Asus's "LightingService.exe" (the daemon that controls their rgb LED coloring suite). Gets up to 2m+ handles after a day or two of running on my system. The next biggest offender on my system? Their ez updater "EzUpdt.exe" at ~60,000 handles. Haven't bothered looking at it with process explorer/procmon yet, but I'm sure it's a quick fix.
- Scene_Cast2 5y agoIn my experience, removing that software is tricky as well. There is OpenRGB (among others) that should be able to control lighting on multiple peripherals. And FanControl (among others) for fan curves.
- tgtweak 5y agoIt's a shit-show also, with every vendor having their own solution and 0 interop. RGB addressing isn't rocket science I'm not sure why every app needs its own proprietary daemon to handle this - mercy on your soul if you have corsair + asus + gigabyte in your system as I do.
- ziml77 5y agoI think it's because no one standardized on protocols for addressing RGB lighting on RAM, GPUs, and motherboards. We at least have those ARGB headers fairly standardized now, but wiring up even things that are slotted into the motherboard would suck (having just a few fans with those is already a mess). Also the manufacturers probably love it because it pushes people towards not mixing products from different companies.
- ziml77 5y agoI didn't even check the handles on LightingService. That shit got uninstalled fast after I saw it eating an entire CPU core and then some. I honestly didn't even want any lighting control software but all the RAM that had the speed and latency that I wanted were decorated with them so I needed software to set them to a static color. Thankfully I learned about OpenRGB.
- tyingq 5y agoWas not surprised to see MS Teams with 12k+ open handles after only running for less than an hour. Click the details tab in task manager, right click on any column, select "add columns".
- pixl97 5y agoHeh, if Teams was a linux application the first line of the documentation would be about editing sysctl.config to allow far more open files by default.
- dafelst 5y agoIf I recall, Teams is an electron app, which means it uses Chromium under the hood - possibly the same bug? I'm not sure that Electron apps have the same local storage characteristics as the full browser, but it seems feasible.
- herf 5y agoYou can track handles within your process using GetProcessHandleCount. We have discovered some leaks in Windows APIs this way (which were fixed pretty fast).
- Arnavion 5y agoI wonder if these problems would happen less on Linux. On the one hand, most distros default to 1024 for the (soft) limit on open fds, which would be easier to hit if you were leaking fds and thus easier to detect. On the other hand, a lot of things that HANDLEs get used for might end up being userspace pointers from userspace libraries instead, so the leak would be of memory instead of fds and hard to detect again.