I have use gdb and various debuggers for a long time. They are very useful but there are limitations. "gdb rr" - tried it with demo helloworld. That works.

I have use gdb and various debuggers for a long time. They are very useful but there are limitations. "gdb rr" - tried it with demo helloworld. That works. But when tried it with a more complex program (FBOSS - Faceboss' Switch/Router open source program), gdb rr function core dump immediately. Also there are certain categories of bugs what are very difficult to use gdb with: * multi threads / race condition bugs - gdb break point in affect the flow of the program and cause different behaviors. * Bugs can not be reliably reproduced. * Server programs or embedded system programs where the code can not be stopped. * This problem get much worst when combine with performance related issue - I work on one system design requirement that would switch over to backup system when system missing a heart beat for 50 milliseconds. Any gdb breakpoint in that app would automatically trigger a hard system fail over to backup. BTW, I do use gdb a lot and can script gdb to do conditional breakpoints with script that auto dump variables, info etc.