6 ms·

Omniscient Debugging (2007)

41 points by eitland 4y ago

eitland 4y ago
Came across a reference to this in my bookmarks. The original was gone and before I go reliable snapshots of my bookmarks but I found at least a couple of interesting open source projects, like the linked one about and also https://pleiad.cl/tod/ https://pleiad.cl/tod/ which also has videos: https://pleiad.cl/tod/screenshots.html https://pleiad.cl/tod/screenshots.html Does anyone on HN have experience with any of these or other similar tools? Why don't we hear about this anymore? Edit: Probably bookmarked it based on this old HN thread: https://news.ycombinator.com/item?id=4801710 https://news.ycombinator.com/item?id=4801710
- Bjartr 4y ago
  The term you want to search for is "time travel debugger", there's plenty to choose from
- Veserv 4y ago
  This is a class of tools called time travel debuggers. Here is a little history of them: http://jakob.engbloms.se/archives/1564 http://jakob.engbloms.se/archives/1564 Generally speaking we do not see much mention of them because the technology involved is reasonably tricky to develop and software developers and companies are as a rule penny wise, but pound foolish with respect to tooling. So, there is very little money in actually expending the development effort to make a polished, accessible tool and any such tool will see relatively limited mass market uptake unless it is free as in beer. This largely precludes the existing relatively polished commercial offerings from achieving ubiquity and thus why it stays relatively niche and underused. That is also why despite there being so many examples over time none of them have been able to achieve escape velocity.
  - timmisiak 4y ago
    I worked on the one that ships with WinDbg: https://learn.microsoft.com/en-us/windows-hardware/drivers/debugger/time-travel-debugging-overview https://learn.microsoft.com/en-us/windows-hardware/drivers/d... Literally every person who uses it for debugging a hard problem absolutely raves about it. Debugging something like a stack corruption or a heap corruption is trivial. And every other class of bugs are also incredibly easy, as long as it doesn't rely on very precise timing where the tracing changes the behavior. So why don't more people use it? Why haven't more people heard about it? I'm not entirely sure, but I do have a guess. Most bugs are shallow. We tend to think about the really hard, really deep bugs, but the vast majority of devs are working on bugs where it's easier to just add some logging to figure out a logic error.
    - moremetadata 4y ago
      If an app is set up to output all variables, doesn't debugview or debugview++ effectively give you the same ability to step back in time? Of course, it means adding code to the app to output everything to debugview where as the debugger can use the debug symbols and then log everything which is alot less work, but some languages dont come with a debugger. Stepping back in time is useful though!
      - HelloNurse 4y ago
        > Of course, it means adding code to the app to output everything to debugview It's a big deal, for example when the "app" is a third party library that cannot be altered.
      - mark_undoio 4y ago
        The answer, as always with engineering is a "yes but" It's pretty appealing. I also believe MS Visual Studio had something that effectively worked like that albeit without changing the coffee manually. But - if you've got bugs due to memory corruption, race conditions, etc then the logging doesn't necessarily tell you what you need as you don't know when the change actually happened. The other issue is that once you've started logging everywhere the string formatting is likely going to be a significant slowdown - at which point you could have used a time travel debugger and probably get better performance. (disclaimer: I work on a time travel debugger)
  - roca 4y ago
    Time-travel debugging and omniscient debugging are different. For example, https://rr-project.org https://rr-project.org is a time-travel debugger --- it can simulate executing backwards through time --- but it is not an omnisicient debugger because it doesn't provide fast access to all program states. https://pernos.co/about https://pernos.co/about is an omniscient debugger built on top of rr: it provides fast access to all program states and visualizations built on top of that.
- modeless 4y ago
  https://pernos.co/ https://pernos.co/ is available today for debugging real C++ programs. It looks awesome. I write GPU code so I don't get to use nice tools like that.
- exDM69 4y ago
  GDB supports reverse execution out of the box (not on all targets / remote), but it can be useful in some cases [0]. The tricky part is I/O and all the other side effects, which can not be undone as easily as memory / register use. It can still be useful for complex algorithms and other side-effect free code. For more practical debugging, record/replay based systems such as radare2 [1] can be more useful. [0] https://sourceware.org/gdb/onlinedocs/gdb/Reverse-Execution.html https://sourceware.org/gdb/onlinedocs/gdb/Reverse-Execution.... [1] https://rada.re/n/ https://rada.re/n/
- acemarke 4y ago
  Yep, my day job is working on the Replay time-traveling debugger for JS ( https://replay.io/ https://replay.io/ ). Also saw someone post an indie gaming company's TTD development environment yesterday: https://www.reddit.com/r/programming/comments/11a2meo/tomorrow_corporations_time_travelling_debugger/ https://www.reddit.com/r/programming/comments/11a2meo/tomorr...
mark_undoio 4y ago
Nice quote on one big benefit of such systems: "there are no non-deterministic problems" Edit: (disclaimer: I work on one such system)
jackcviers3 4y ago
I've worked on a lot of sophisticated data pipelines, in robotics, built customized wysiwyg editors, distributed systems, append-only logs with replicated data types, in around a dozen languages. While I have found occasionally remote profiling useful for optimizing hot code paths, I have always found that trace-level logs and structured logging with aggregation and indexing is superior to debugging in most applications. Essentially - println debugging is superior to interactive debugging, especially for intermittent bugs, data races, deadlocks, resource leakage, modeling and serialization bugs, binary or source code incompatibility bugs, and bugs that are catastrophic and rare or pathological in nature. The problem with interactive debugging is that you typically only have a view into not only a single process of a system, but also only a single concurrent execution thread of the single process. When you have structured logging and aggregation and indexed log search, you have a much broader view of the systems under investigation and can see the conditions causing the bug with a much higher frequency than with interactive debugging. For any non-trivial bug in any sufficiently large input or program you have to have insight into the root cause via intuition to cause it to occur quickly enough to inspect the program via the debugger. This obviously doesn't scale well. It's much better to have a detailed, structured log mode built into your systems that is capable of being enabled on demand - via environment variable, remote procedure call, request header, data packet, or command flag during deployment. This allows you to observe the system behavior and incur the cost of the detailed logging on demand, much like you would when connecting via remote debugging, but on a system of systems scale, increasing the probability of capturing the bug's root cause over stepping forward and backward through interactive execution. Additionally, care should be taken to program in a value-centric way when you can get away with it performance or space-wise. Essentially - use FP principles of immutability and programs as values and composition to build your programs where you can. Values have a unique property in that they are serialization state that can be inspected or substituted in place of references to the value without changing the behavior of a program at runtime. This often simplifies debugging even in complex side-effecting concurrent algorithms. When you can't, comprehensive argument and shared state change log events that can be shared amongst many members of the development, stakeholder, and operations teams asynchronously helps to narrow the surface area scan of the system you are debugging. Formal methods during modeling, design, and development can help to increase constraints and enable more comprehensive test suites, but property-based testing with randomized generated valid inputs also serve as a sort of automated preemptive debugger. Things like chance in the js world, quickcheck/hedgehog in haskell and their ports or siblings in other languages, like hypothesis in Python, fall into this category of testing tools. Interactive debugging is the tool of last resort, and still serves a useful purpose when all other mitigation and system state capture methods have failed, but I personally find myself believing that if a bug's root cause hasn't been discovered by this stage I have made some set of mistakes in the design of the system, either in the application of its logical rules or in the basic assumptions fom interpretation of business requirements to the choices of abstractions/tradeoffs made in development. It's usually a bad day when I have to resort to interactive debugging.

41 points by eitland 4y ago

Came across a reference to this in my bookmarks. The original was gone and before I go reliable snapshots of my bookmarks but I found at least a couple of interesting open source projects, like the linked one about and also https://pleiad.cl/tod/ https://pleiad.cl/tod/ which also has videos: https://pleiad.cl/tod/screenshots.html https://pleiad.cl/tod/screenshots.html Does anyone on HN have experience with any of these or other similar tools? Why don't we hear about this anymore? Edit: Probably bookmarked it based on this old HN thread: https://news.ycombinator.com/item?id=4801710 https://news.ycombinator.com/item?id=4801710

Bjartr 4y ago
The term you want to search for is "time travel debugger", there's plenty to choose from
Veserv 4y ago
This is a class of tools called time travel debuggers. Here is a little history of them: http://jakob.engbloms.se/archives/1564 http://jakob.engbloms.se/archives/1564 Generally speaking we do not see much mention of them because the technology involved is reasonably tricky to develop and software developers and companies are as a rule penny wise, but pound foolish with respect to tooling. So, there is very little money in actually expending the development effort to make a polished, accessible tool and any such tool will see relatively limited mass market uptake unless it is free as in beer. This largely precludes the existing relatively polished commercial offerings from achieving ubiquity and thus why it stays relatively niche and underused. That is also why despite there being so many examples over time none of them have been able to achieve escape velocity.
- timmisiak 4y ago
  I worked on the one that ships with WinDbg: https://learn.microsoft.com/en-us/windows-hardware/drivers/debugger/time-travel-debugging-overview https://learn.microsoft.com/en-us/windows-hardware/drivers/d... Literally every person who uses it for debugging a hard problem absolutely raves about it. Debugging something like a stack corruption or a heap corruption is trivial. And every other class of bugs are also incredibly easy, as long as it doesn't rely on very precise timing where the tracing changes the behavior. So why don't more people use it? Why haven't more people heard about it? I'm not entirely sure, but I do have a guess. Most bugs are shallow. We tend to think about the really hard, really deep bugs, but the vast majority of devs are working on bugs where it's easier to just add some logging to figure out a logic error.
  - moremetadata 4y ago
    If an app is set up to output all variables, doesn't debugview or debugview++ effectively give you the same ability to step back in time? Of course, it means adding code to the app to output everything to debugview where as the debugger can use the debug symbols and then log everything which is alot less work, but some languages dont come with a debugger. Stepping back in time is useful though!
    - HelloNurse 4y ago
      > Of course, it means adding code to the app to output everything to debugview It's a big deal, for example when the "app" is a third party library that cannot be altered.
    - mark_undoio 4y ago
      The answer, as always with engineering is a "yes but" It's pretty appealing. I also believe MS Visual Studio had something that effectively worked like that albeit without changing the coffee manually. But - if you've got bugs due to memory corruption, race conditions, etc then the logging doesn't necessarily tell you what you need as you don't know when the change actually happened. The other issue is that once you've started logging everywhere the string formatting is likely going to be a significant slowdown - at which point you could have used a time travel debugger and probably get better performance. (disclaimer: I work on a time travel debugger)
- roca 4y ago
  Time-travel debugging and omniscient debugging are different. For example, https://rr-project.org https://rr-project.org is a time-travel debugger --- it can simulate executing backwards through time --- but it is not an omnisicient debugger because it doesn't provide fast access to all program states. https://pernos.co/about https://pernos.co/about is an omniscient debugger built on top of rr: it provides fast access to all program states and visualizations built on top of that.
modeless 4y ago
https://pernos.co/ https://pernos.co/ is available today for debugging real C++ programs. It looks awesome. I write GPU code so I don't get to use nice tools like that.
exDM69 4y ago
GDB supports reverse execution out of the box (not on all targets / remote), but it can be useful in some cases [0]. The tricky part is I/O and all the other side effects, which can not be undone as easily as memory / register use. It can still be useful for complex algorithms and other side-effect free code. For more practical debugging, record/replay based systems such as radare2 [1] can be more useful. [0] https://sourceware.org/gdb/onlinedocs/gdb/Reverse-Execution.html https://sourceware.org/gdb/onlinedocs/gdb/Reverse-Execution.... [1] https://rada.re/n/ https://rada.re/n/
acemarke 4y ago
Yep, my day job is working on the Replay time-traveling debugger for JS ( https://replay.io/ https://replay.io/ ). Also saw someone post an indie gaming company's TTD development environment yesterday: https://www.reddit.com/r/programming/comments/11a2meo/tomorrow_corporations_time_travelling_debugger/ https://www.reddit.com/r/programming/comments/11a2meo/tomorr...

mark_undoio 4y ago

Nice quote on one big benefit of such systems: "there are no non-deterministic problems" Edit: (disclaimer: I work on one such system)

jackcviers3 4y ago

I've worked on a lot of sophisticated data pipelines, in robotics, built customized wysiwyg editors, distributed systems, append-only logs with replicated data types, in around a dozen languages. While I have found occasionally remote profiling useful for optimizing hot code paths, I have always found that trace-level logs and structured logging with aggregation and indexing is superior to debugging in most applications. Essentially - println debugging is superior to interactive debugging, especially for intermittent bugs, data races, deadlocks, resource leakage, modeling and serialization bugs, binary or source code incompatibility bugs, and bugs that are catastrophic and rare or pathological in nature. The problem with interactive debugging is that you typically only have a view into not only a single process of a system, but also only a single concurrent execution thread of the single process. When you have structured logging and aggregation and indexed log search, you have a much broader view of the systems under investigation and can see the conditions causing the bug with a much higher frequency than with interactive debugging. For any non-trivial bug in any sufficiently large input or program you have to have insight into the root cause via intuition to cause it to occur quickly enough to inspect the program via the debugger. This obviously doesn't scale well. It's much better to have a detailed, structured log mode built into your systems that is capable of being enabled on demand - via environment variable, remote procedure call, request header, data packet, or command flag during deployment. This allows you to observe the system behavior and incur the cost of the detailed logging on demand, much like you would when connecting via remote debugging, but on a system of systems scale, increasing the probability of capturing the bug's root cause over stepping forward and backward through interactive execution. Additionally, care should be taken to program in a value-centric way when you can get away with it performance or space-wise. Essentially - use FP principles of immutability and programs as values and composition to build your programs where you can. Values have a unique property in that they are serialization state that can be inspected or substituted in place of references to the value without changing the behavior of a program at runtime. This often simplifies debugging even in complex side-effecting concurrent algorithms. When you can't, comprehensive argument and shared state change log events that can be shared amongst many members of the development, stakeholder, and operations teams asynchronously helps to narrow the surface area scan of the system you are debugging. Formal methods during modeling, design, and development can help to increase constraints and enable more comprehensive test suites, but property-based testing with randomized generated valid inputs also serve as a sort of automated preemptive debugger. Things like chance in the js world, quickcheck/hedgehog in haskell and their ports or siblings in other languages, like hypothesis in Python, fall into this category of testing tools. Interactive debugging is the tool of last resort, and still serves a useful purpose when all other mitigation and system state capture methods have failed, but I personally find myself believing that if a bug's root cause hasn't been discovered by this stage I have made some set of mistakes in the design of the system, either in the application of its logical rules or in the basic assumptions fom interpretation of business requirements to the choices of abstractions/tradeoffs made in development. It's usually a bad day when I have to resort to interactive debugging.