3 ms·

Good war story. One thing I was wondering: > It essentially showed our layer0 service freezing, and even stop reporting its own internal metrics, only to reco

by anyfoo 8y ago

Good war story. One thing I was wondering: > It essentially showed our layer0 service freezing, and even stop reporting its own internal metrics, only to recover a few minutes later. As we didn’t know how to reproduce the problem, it was hard to make progress. I don’t know the environment (and not too much about the current affairs of e.g. web development in general), but since the problem was noticeable and lasted for minutes, up to almost an hour according to the graph, is there anything that would prevent them from logging into an affected machine and taking multiple stack shots of the relevant processes in close succession? That would immediately show which threads are hanging where. The rest of debugging is still required to figure out why it hangs there (especially if you are not familiar with the internals of the code at the top of the obtained stack shots), but at least you know where to look.

muststopmyths 8y ago
Since the root problem seems to have been inside gRPC, the stacks may or may not point to an actual cause, depending on underlying architecture. For example, if there is some sort of asynchronous queue for message send/receive, the stacks might just show threads waiting. I'm speculating obviously. But for any complex application you usually are looking at a mix of verbose logs and live debugging when troubleshooting. The advantage of good logs is that you can follow the control/data flow in the application, whereas attaching a debugger gives you current state. Figuring out why you're in that state may be quite difficult, depending on the application.
- anyfoo 8y ago
  > For example, if there is some sort of asynchronous queue for message send/receive, the stacks might just show threads waiting. Yeah, that's the point, it will show the threads waiting, but it will normally also exactly show where, with what call stack, the threads are waiting. The full call stack of several stuck processes usually gives you a pretty good idea what is being waited on, and why. I say "usually" because things get murkier with e.g. interpreted languages, where what the program is actually doing is on a different abstraction layer than the presented stack traces, but even then you have a good chance of at least figuring out what the stalling resource is, and what processes are involved. > Since the root problem seems to have been inside gRPC, the stacks may or may not point to an actual cause I'd go so far to say that they are unlikely to point at the actual cause (as said, you still need to continue debugging, and good logs always help), but at least you now know what code to look at without having to guess reproduction steps. It's how we debug hangs in modern complex operating systems, where there often is a lot of IPC involved. The very first thing to ask for and look at is stack shots comprising all processes and threads in the system (either in kernel land or user land or both, depending on the problem), preferably many of those taken in close-ish succession.

by anyfoo 8y ago

muststopmyths 8y ago

Since the root problem seems to have been inside gRPC, the stacks may or may not point to an actual cause, depending on underlying architecture. For example, if there is some sort of asynchronous queue for message send/receive, the stacks might just show threads waiting. I'm speculating obviously. But for any complex application you usually are looking at a mix of verbose logs and live debugging when troubleshooting. The advantage of good logs is that you can follow the control/data flow in the application, whereas attaching a debugger gives you current state. Figuring out why you're in that state may be quite difficult, depending on the application.

anyfoo 8y ago
> For example, if there is some sort of asynchronous queue for message send/receive, the stacks might just show threads waiting. Yeah, that's the point, it will show the threads waiting, but it will normally also exactly show where, with what call stack, the threads are waiting. The full call stack of several stuck processes usually gives you a pretty good idea what is being waited on, and why. I say "usually" because things get murkier with e.g. interpreted languages, where what the program is actually doing is on a different abstraction layer than the presented stack traces, but even then you have a good chance of at least figuring out what the stalling resource is, and what processes are involved. > Since the root problem seems to have been inside gRPC, the stacks may or may not point to an actual cause I'd go so far to say that they are unlikely to point at the actual cause (as said, you still need to continue debugging, and good logs always help), but at least you now know what code to look at without having to guess reproduction steps. It's how we debug hangs in modern complex operating systems, where there often is a lot of IPC involved. The very first thing to ask for and look at is stack shots comprising all processes and threads in the system (either in kernel land or user land or both, depending on the problem), preferably many of those taken in close-ish succession.