In another life I worked on Windows Error Reporting (WER) - the part of the stack that was responsible for collecting, analyzing and sending reports to Microsof

In another life I worked on Windows Error Reporting (WER) - the part of the stack that was responsible for collecting, analyzing and sending reports to Microsoft and even provide solutions to the user based on the analysis. For usermode problems WER would do a first level analysis on the client by looking at the crashing thread's exception record (1) or in the case of a user mode hang it would do something much more sophisticated called wait chain traversal (2). It would then generate a signature for the issue and report it to the backend via a single (and cheap) http/https request. The backend would then increment a count for that particular signature and if needed, would ask the client for more information about the issue. Among other things it could ask the client to capture a dump of the crashing process and even a multiprocess dump in the case of hang that involved multiple processes (process A waiting for process B waiting for process C etc). The client would do the necessary collection, prompt the user if necessary (depending on the user's consent settings) and eventually send it off. The 2nd level collection was configurable via a internal web portal - teams could look up their top signatures and request for more information about it. They'd eventually get notified once the extra information was collected - for high frequency issues, the turnaround was very quick. Kernel mode issues aka bluescreens were treated slightly differently. The client wasn't able to do first-level analysis on kernel mode crashes. Instead it would always send at atleast a minidump of the bluescreen to a backend service where a pool of servers would run the !analyze windbg extension in realtime on the minidump. (1) https://msdn.microsoft.com/en-us/library/aa260334(v=vs.60).aspx https://msdn.microsoft.com/en-us/library/aa260334(v=vs.60).a... (2) https://msdn.microsoft.com/en-us/library/windows/desktop/ms681622(v=vs.85).aspx https://msdn.microsoft.com/en-us/library/windows/desktop/ms6...