3 ms·

Since the root problem seems to have been inside gRPC, the stacks may or may not point to an actual cause, depending on underlying architecture. For example, i

Since the root problem seems to have been inside gRPC, the stacks may or may not point to an actual cause, depending on underlying architecture. For example, if there is some sort of asynchronous queue for message send/receive, the stacks might just show threads waiting. I'm speculating obviously. But for any complex application you usually are looking at a mix of verbose logs and live debugging when troubleshooting. The advantage of good logs is that you can follow the control/data flow in the application, whereas attaching a debugger gives you current state. Figuring out why you're in that state may be quite difficult, depending on the application.

anyfoo 8y ago
> For example, if there is some sort of asynchronous queue for message send/receive, the stacks might just show threads waiting. Yeah, that's the point, it will show the threads waiting, but it will normally also exactly show where, with what call stack, the threads are waiting. The full call stack of several stuck processes usually gives you a pretty good idea what is being waited on, and why. I say "usually" because things get murkier with e.g. interpreted languages, where what the program is actually doing is on a different abstraction layer than the presented stack traces, but even then you have a good chance of at least figuring out what the stalling resource is, and what processes are involved. > Since the root problem seems to have been inside gRPC, the stacks may or may not point to an actual cause I'd go so far to say that they are unlikely to point at the actual cause (as said, you still need to continue debugging, and good logs always help), but at least you now know what code to look at without having to guess reproduction steps. It's how we debug hangs in modern complex operating systems, where there often is a lot of IPC involved. The very first thing to ask for and look at is stack shots comprising all processes and threads in the system (either in kernel land or user land or both, depending on the problem), preferably many of those taken in close-ish succession.

anyfoo 8y ago

> For example, if there is some sort of asynchronous queue for message send/receive, the stacks might just show threads waiting. Yeah, that's the point, it will show the threads waiting, but it will normally also exactly show where, with what call stack, the threads are waiting. The full call stack of several stuck processes usually gives you a pretty good idea what is being waited on, and why. I say "usually" because things get murkier with e.g. interpreted languages, where what the program is actually doing is on a different abstraction layer than the presented stack traces, but even then you have a good chance of at least figuring out what the stalling resource is, and what processes are involved. > Since the root problem seems to have been inside gRPC, the stacks may or may not point to an actual cause I'd go so far to say that they are unlikely to point at the actual cause (as said, you still need to continue debugging, and good logs always help), but at least you now know what code to look at without having to guess reproduction steps. It's how we debug hangs in modern complex operating systems, where there often is a lot of IPC involved. The very first thing to ask for and look at is stack shots comprising all processes and threads in the system (either in kernel land or user land or both, depending on the problem), preferably many of those taken in close-ish succession.