6 ms·
What is Windows doing while hogging that lock?
- SeanBoocock 9y agoGood followup to Bruce's previous blog, discussed here: https://randomascii.wordpress.com/2017/07/09/24-core-cpu-and-i-cant-move-my-mouse/ https://randomascii.wordpress.com/2017/07/09/24-core-cpu-and... Surprised at how much you can divine from the ETW traces with enough persistence.
- nitwit005 9y agoMight be worth looking to see if the processes are creating hidden windows or other UI objects. The previous post references NtGdiCloseProcess, and it'd make sense that some of those handles would require a lock to clean up safely.
- rootw0rm 9y agoaccording to his code, he's tested hidden windows and regular ones.
- brucedawson 9y agoThe test code is up there. The master process loads user32.dll and makes a couple of window-type calls, but the other processes do nothing but wait on semaphores and then die.
- revelation 9y agoWell.. what is it doing? Analyzing the assembly is neat but this doesn't seem like a CPU performance problem, it's a high-level design issue. It's a bit like analyzing sleep sort.
- brucedawson 9y agoAgreed. I was just curious, and it seemed like a fun way to demonstrate the technique, which can be useful when profiling CPU intensive parts of your own code.
- rootw0rm 9y agomeh, i'm an idiot.
- jsnell 9y agoThe question isn't about three instructions being executed by three separate OS threads. It's about three instructions being executed at the same time on a single thread on a single core. The interrupt context will only contain a single program counter value; which of the instructions that could currently be considered to be executing on that thread will it actually point to?
- rootw0rm 9y agoi figured i misunderstood. i guess if you run it long enough that statistically it doesn't matter? i still think the culprit is going to turn out to be something more mundane.
- tedunangst 9y agoI would guess PC is most recently retired instruction? Or would it be not retired? Although that is an interesting scenario. slow eax, ebx fast ecx, ecx Assuming slow and fast always finish at the same time, will the profiler ever see the CPU executing slow?
- comex 9y agoThe PC would be recorded from an interrupt handler as the address to resume execution at; I don’t think there’s any other way to record PCs on x86 (there’s Last Branch Record but that’s only for branch instructions). So, I’d expect it would be the first not-yet-retired instruction, and you’d always see ‘slow’, not ‘fast’.
- brucedawson 9y ago> you’d always see ‘slow’, not ‘fast’. And yet - line 34 of the spreadsheet shows "xor r8d,r8d" as being quite expensive. My theory is that the branch which targets it must be mispredicted and it gets blamed. Similarly, line 24 of the spreadsheet shows "cmp ebx,eax" as being quite expensive. The only explanation I can come up with is that the previous instruction was blocked by a cache miss and then the "cmp" must have been folded in with the add so that mumble mumble nope I don't have a good mental model. So, there are quite a few examples of instructions which by themselves are obviously always fast being hit by a huge portion of the CPU samples.
- pbsd 9y agoHmgNextOwned doesn't do, by itself, all that much. There is some mild pointer-chasing, but not too horrible. All it does is to go through the global GDI handle table and look for handles owned by the terminating process. It is used loosely as DWORD index = 0; while(true) { SOME_GDI_OBJECT_TYPE obj; index = HmgNextOwned(index, pid, &obj); if(index == 0) break; // done / nothing found // ... do some clean up of the object here } The thing is, this loop is executed 6 (12, if counting the instances inlined into NtGdiCloseProcess) times when terminating a process, for various object types, and for each call of HmgNextOwned it will lock and unlock the handle manager lock.
- brucedawson 9y agoThanks for the explanation. If it scans the entire GDI handle table then that suggests that its runtime will be proportional to the system's total number of GDI objects, correct? That sort of design can easily get expensive. I'm surprised it can't iterate through its own GDI objects without having to traverse everybody else's as well.
- pbsd 9y agoYeah, it's proportional to the total number of GDI objects. Approximately: DWORD HmgNextOwned(DWORD index, DWORD pid, HANDLE * handle) { GDI_TABLE_ENTRY entry; GreAcquireHmgrSemaphore(); // GetNextEntryIndex is inlined in reality, this is the main loop you see in the disassembly while(index = gpHandleManager->GetNextEntryIndex(index, &entry)) { if(entry.Type != 0 && ((pid & 0xFFFFFFFD) ^ entry.ProcessId) & 0xFFFFFFFE) == 0) { *handle = index | (entry.Upper << 16); break; } } GreReleaseHmgrSemaphore(); return index; } Best information I could find on the relevant structures is [1], which gives an idea of what you're traversing. I don't know why this is designed this way; it does seem suboptimal. [1] https://cvr-data.blogspot.com/2016/11/windows-10-anniversary-update-gdi.html https://cvr-data.blogspot.com/2016/11/windows-10-anniversary...
- brucedawson 9y agoThat link was extremely helpful! My unsubstantiated belief is that the process-shutdown bug was new in Windows 10 Anniversary Edition, and that link supports that idea. It sounds like Windows 10 Anniversary Edition changed the management of GDI objects to avoid leaking kernel addresses and that made HmgNextOwned more expensive, thus triggering the visible serialization of process destruction. Cool! I really wish I could upvote the parent post more times. Failing that I've updated the latest post to link to it as an explanation for how the whole problem got started.
- brucedawson 9y agoIt's great to see all this discussion. If anyone has a link to any Intel papers or systematic analysis of how sampling profilers interact with Intel super-scalar out-of-order processors that would be appreciated.
- ThrowingPotato 9y agoWhat are the full specs on your 24 core machine?
- brucedawson 9y agoHP Z840, two sockets, 64 GB of RAM, SSD. I don't remember the details and I'm at home now. The most recent tests were done on my four-core/eight-thread laptop. It's running a more recent version of Windows so it's a "fairer" test in that way, and it reproduces the problem just fun with my artificial test program, CreateProcessTests.
- Symmetry 9y ago>If there are three instructions executing simultaneously when the interrupt fires then which one “gets the blame”? If a load instruction misses in the cache and forces the CPU to wait then will the samples show up on the load instruction, or on the first use of the data? Both seem to happen. Specifically in order to handle interrupts a CPU may execute instructions out of order but there's a final retire step where the results of all instructions are made 'official' in order. So from outside it looks like the CPU was executing in order. The upshot is that, except in crazy optimized code that might sometimes overwhelm the retire unit, the last instruction you see as not having finished hasn't finished but there might be other instructions being worked on further along as well.