Disclaimer (and a bit of context): I work for a company that builds a special type of debugger, aimed at debugging live applications in production. As one woul

Disclaimer (and a bit of context): I work for a company that builds a special type of debugger, aimed at debugging live applications in production. As one would expect, we've spent an enormous amount of time thinking about debuggers, how they work from the inside and what they contribute to the general experience of developing software. Debuggers are, for developers well-versed in working with them, priceless tools (see [1] for an example from this very thread). There's so, so much value in walking through a program not only in one's head (as well-put by pg at one time[2]), but by using a tool that enforces a more structured, "formal" process for program exploration. I use it when trying to understand how code I write actually looks like IRL, but also - and maybe more so - for understanding code OTHER PEOPLE wrote with better context than any piece of documentation or simply reading the code would ever give me. But - as the article and some[3] of[4] the[5] comments[6] here point out - it has warts (quick plug - I recently wrote a blog post[7] that details my perspective on remote debuggers and how they can be better). One thing that I didn't see mentioned in this thread or in the article, and is a real PITA, is performance. Debuggers are not built to deal with performance problems - or collection of temporal data, for that matter - at all. Consider how one might go about debugging a real-life performance issue, especially in the context of modern applications - many threads, many hosts, many replicas of the same service, etc. You probably have a tracer - something like Zipkin[8] or Jaeger[9] - that helps you to break apart latency issues in your endpoints. You also probably have a set of instrumented metrics (graphed inside your APM, perhaps) for quick observation when something is not reacting as fast as you expect it to. But sometimes they don't tell the full story - there's just too many possible edge cases. Performance issues can be caused by data structures exploding in size, causing degraded performance and OOMs. They can be caused by one method, in a chain of dozens of different, inter-connected ones, that is especially slow. It can be related to non-optimized SQL queries (maybe hidden behind an ORM). It can be a weird code path - one the developer did not expect, and did not test for - that gets triggered by users with very specific characteristics. Hard. [1] https://news.ycombinator.com/item?id=25960923#unv_25960923:~:text=The%20debugger%20is%20the%20main%20reason,for%20JVM%20or%20dynamic%20language%20stuff https://news.ycombinator.com/item?id=25960923#unv_25960923:~...... [2] http://www.paulgraham.com/head.html http://www.paulgraham.com/head.html [3] https://news.ycombinator.com/item?id=25940064#unv_25957905:~:text=gdb%20and%20lldb%20don't%20work%20great,where%20multiple%20threads%20are%20executing%20concurrently https://news.ycombinator.com/item?id=25940064#unv_25957905:~... [4] https://news.ycombinator.com/item?id=25940064#unv_25957481:~:text=use.-,Hell%2C%20in%20my%20main%20project%20I've%20been%20unable%20to%20use%20any%20version%20of%20gdb%20%26%20lldb%20for%20something%20like%20two%20months%20as%20both%20segfault%20a%20couple%20seconds%20after%20reaching%20a%20breakpoint,for%20some%20reason.%20It's%20a%20shitshow https://news.ycombinator.com/item?id=25940064#unv_25957481:~.... [5] https://news.ycombinator.com/item?id=25940064#unv_25957428:~:text=The%20problem%20of%20debugging%20has%20become,have%20some%20kind%20of%20async%20mechanism https://news.ycombinator.com/item?id=25940064#unv_25957428:~.... [6] https://news.ycombinator.com/item?id=25940064#unv_25957481:~:text=if%20debuggers%20took%20less%20time%20to%20start%20than%20a%20full%20%22add%20print%20%2D%3E%20recompile%20%2D%3E%20relaunch%20app%22%20cycle%20as%20is%20my%20experience%20with%20C%2B%2B%2C,they%20would%20certainly%20get%20more%20use.,-Hell%2C https://news.ycombinator.com/item?id=25940064#unv_25957481:~... [7] https://lightrun.com/debugging/remote-debugging-re-examined-for-the-wfh-era/ https://lightrun.com/debugging/remote-debugging-re-examined-... [8] https://zipkin.io/ https://zipkin.io/ [9] https://www.jaegertracing.io/ https://www.jaegertracing.io/