6 ms·
Time Travel Debugging
- timmisiak 9y agoI'm the dev lead for WinDbg and the windows debugging platform, and I'm a former dev working on time travel debugging (I wrote the cpu emulation for the project). Happy to answer any questions that anyone has. Since I'm "posting too fast", let me put my answers here for now: qod: Right now you can attach TTD to a process after initialization (or any point in time prior to the repro), but you can't trace starting from a breakpoint. We'd love to add that functionality though, so I'm sure you'll see it in a future version of the tool. Glad you like the new WinDbg! It's been very polarizing (which you can see just reading the comments on this post!), so it's good to hear the positive feedback sometimes :) We've got a ton of plans to make debugging even faster and more effective in WinDbg, so hopefully we'll win more people over as we make our tools easier to use and more powerful.
- qod 9y agoFirst I would just like to say how impressed I am with the new WinDbg and TTD, this is fantastic work and a huge improvement on the existing debugging tools. My question is: if I want to start taking a TTD trace after hitting a particular breakpoint in WinDbg, can this be done with the current tooling? Use case would be if there is a large amount of initialization code that doesn't need to be traced because the bug or crash is known to occur at a later stage in the program.
- timmisiak 9y agoRight now you can attach TTD to a process after initialization (or any point in time prior to the repro), but you can't trace starting from a breakpoint. We'd love to add that functionality though, so I'm sure you'll see it in a future version of the tool. Glad you like the new WinDbg! It's been very polarizing (which you can see just reading the comments on this post!), so it's good to hear the positive feedback sometimes :) We've got a ton of plans to make debugging even faster and more effective in WinDbg, so hopefully we'll win more people over as we make our tools easier to use and more powerful.
- jasonwelk 9y agoThis is cool stuff to see in the world of C++. It's great to see language communities feed off each other and drive innovation like this.
- cma 9y agoI think it has been in gdb before Swift.
- userbinator 9y agoYes: https://www.gnu.org/s/gdb/news/reversible.html https://www.gnu.org/s/gdb/news/reversible.html This has approximately nothing to do with C++ or even "language communities".
- jasonwelk 9y agoStrange, the impression I got from MS devs on Twitter is that this is something they have been working on for a while that they just finished and premiered at a CPP conference.
- SteveJS 9y agoThe core technology was born out of a group in MS research long ago. Moving from research code to production is highly non-trivial. I first heard how time travel debugging was implemented before i knew about Facebook. I suspect SSDs are part of what made this more practical as it used to simply saturate disk IO, even on specialized systems. With a pci express SSD this becomes much more pragmatic then when the initial idea was implemented. If the overhead doesn’t prevent capturing a trace reproducing the symptom, there is zero reason you cannot find the root cause. But that actually pales in comparison with treating the execution traces as data.
- xms 9y agoYes I believe TTT, or iDNA as it was known, was first released internally around 15 years ago. Or maybe that was just the Nirvana process instrumentation framework it is based on. Either way it's a long time. A modified version of the tracer is used sometimes to help with support cases, but this is the first public release of the WinDbg replay bit as well.
- userbinator 9y agoTTD is efficient and works to add as little as possible overhead as it captures code execution in trace files. ...and then in the table below the screenshot... Large overhead at record time. May collect more data that is needed. Data files can become large. Based on past experiences with debugging using tracepoints etc., I'm more inclined to believe the latter. Also, as a side-note, the "more modern visuals" of "WinDbg Preview" look horrible. It's a debugger, not a toy for the barely-computer-literate. Those who want the "friendly experience" will use Visual Studio instead, which amusingly enough continues to have menus instead of the disgusting ribbons as of the latest 2017 version. The screenshot also shows two lines of slightly-misaligned "Command" "Memory" "Source". Yuck. (For those who don't understand, this what the original WinDbg looks like --- simple and functional: http://sandsprite.com/blogs/images/main_ui.png http://sandsprite.com/blogs/images/main_ui.png )
- deleted 9y ago[deleted]
- timmisiak 9y agoThe overhead varies depending on how cpu/IO bound the application is. IO isn't really affected, so IO bound applications tend to not see a big slowdown. In theory, you could see a very large slowdown in the worst case, but in the average case for a "medium sized" process the slowdown would be noticeable but not affect the usability. This technology isn't based on tracepoints, it's based on in-process cpu emulation. The emulation overhead is on the order of 10-20x in many cases, whereas tracepoint overhead would be on the order of 1000x I believe (maybe worse). If there is something specific you dislike about the visuals of WinDbg Preview, let us know through the feedback hub or emailing windbgfb@microsoft.com. We realize that folks that have been using WinDbg for 20 years are likely to not be interested in a new UI, but we face 20 years of legacy code every time we want to add a new feature to the UI. As an example, the new WinDbg UI has a javascript window for writing scripts that extend and automate the debugger. It took us approximately 8x less time to implement in WinDbg Preview than what we estimated it would cost in the legacy UI (and a much more junior dev was able to do it as well). We want to innovate without disrupting folks that have effective workflows in WinDbg, so we really want to hear feedback on the new UI. If there are specific things that we can change to make you more efficient in the new UI, please let us know.
- arbesfeld 9y agoVery cool stuff! Microsoft also developed a TTD system for JavaScript/Node.js: https://www.microsoft.com/en-us/research/publication/time-travel-debugging-javascriptnode-js/ https://www.microsoft.com/en-us/research/publication/time-tr... At LogRocket (https://logrocket.com https://logrocket.com) we're building a TTD of sorts for recording production users in your web app. While we can't replay exact code, we can get you most of the way there with DOM recording/logs/network waterfall.
- JoshTriplett 9y agoHere's a review from the Mozilla folks: http://robert.ocallahan.org/2017/10/thoughts-on-microsofts-time-travel.html http://robert.ocallahan.org/2017/10/thoughts-on-microsofts-t...
- alexashka 9y agoLooks very nifty. Is this being developed for Windows only?
- kuwze 9y agoThere is also undodb[0] and gdb supports reversing[1]. There is also the rr[2] project from Mozilla. All of those support Linux. I also recently discovered WDD[3] (which is for Windows) as well. [0]: https://undo.io/ https://undo.io/ [1]: https://www.gnu.org/software/gdb/news/reversible.html https://www.gnu.org/software/gdb/news/reversible.html [2]: http://rr-project.org/ http://rr-project.org/ [3]: https://github.com/ipkn/wdd https://github.com/ipkn/wdd
- nn3 9y agoThere is also PinPlay / DrDebug https://software.intel.com/en-us/articles/program-recordreplay-toolkit https://software.intel.com/en-us/articles/program-recordrepl... https://software.intel.com/en-us/articles/pintool-drdebug https://software.intel.com/en-us/articles/pintool-drdebug
- timmisiak 9y agoI believe all of those projects are based on single-core debugging. Restricting the trace to a single core is one way of introducing determinism in the process as a way of allowing replay later. Microsoft time travel debugging is a multi-core technology, which allows you to capture interactions between threads in multi-threaded processes. Edit: Since I'm still "posting too fast", let me respond here to clarify. Both rr and undodbg support multiple threads, but do not record them simultaneously. They restrict execution to a single core. That's what I mean by recording interactions between threads. For instance, if you have shared memory between processes, that won't work with rr/undodb generally (although I know at least rr can work around this by recording both processes). I'm not saying one is intrinsically better than the other since there are tradeoffs with both approaches. There is a high constant overhead for recording all cores, but it does have the advantage of scaling well to large numbers of cores.
- kuwze 9y agorr does support multithreading[0]. "rr doesn't record shared-memory multithreading, so it forces your application's threads to execute serially. Your application can see an additional slowdown if it takes advantage of multicore parallelism." [0]: https://github.com/mozilla/rr/wiki/Usage#recording-an-execution https://github.com/mozilla/rr/wiki/Usage#recording-an-execut...
- dmitrygr 9y agoWe at VMware had this working and released in 2008. We supported windows and Linux. We even supported things like network applications that had lots of outside state. It was called replay debugging and besides Mozilla, few used it. Sadly it was killed when workstation 8.0 came out.
- timmisiak 9y agoThat really was a cool feature of VMware (and I used to work on the vmware hypervisor briefly), although it used very different techniques and was limited to single core record/replay. Microsoft TTD is based on a different technology, and captures traces of individual processes. More useful for usermode folks, less useful for kernel mode folks compared to the vmware technology.
- AlphaSite 9y agoWas it an open source project? If not do you know where it lives inside the company (I’m curious about how it works).
- pmoriarty 9y ago"Sadly it was killed when workstation 8.0 came out." What ever happened to the intellctual property for that? Any chance it could be open sourced?
- dmitrygr 9y agoInternal politics.
- RyanRies 9y agoHere is a recent CppCon video: https://www.youtube.com/watch?time_continue=3389&v=l1YJTg_A914 https://www.youtube.com/watch?time_continue=3389&v=l1YJTg_A9... I'm an escalation engineer at Microsoft, and we have been using this for many, many years to solve complex customer cases. Having it go public is a pretty big deal for us.
- lj3 9y agoReminds me of Qira. http://qira.me/ http://qira.me/
- mncharity 9y agoFor more than two decades now, I've been using the limited availability of reversible debugging as an example of just how dysfunctional software engineering is as a field. The shoemaker's family's tattered feet. Reversible debugging been around for six decades (eg EXDAMS on Multics), been generally practical for two or three, been available in gdb for one, and now, finally, here we are. It's taken two bloody human generations to get this close to widely deploying TTD. So yay. Maybe I can now retire TTD as an example of glacial progress. There's no shortage of others. Sigh.
- csours 9y agoI'm curious to hear some. Are you thinking of stuff like Plan 9 files/services?
- mncharity 9y agoGP> example[s] of glacial progress > I'm curious to hear some. How many keyboard keys can you hold down without interference effects? Some keyboards offer 10-ish. Or keycaps which are touch pads. Or change color. Or are little active displays. And touch pads which are pressure sensitive. Or detect proximity. Or have a nice stylus. But not mine. It's not space-cadet, or old thinkpad, but chicklets. One nifty aspect of VR is that a lot of things get unstuck. Even if you're legally blind, VR brings you hand tracking, and vibrational haptics, and spatial audio. You might stroke home-row keys, and make little 3D gestures with fingers. But current Leap Motion hand-tracking software doesn't do linux, so you have to run it in a VM, and point a camera at the keyboard, and... Here we are on HN. USENET had kill files - people you didn't want to see again. Recommendation systems track preferences and predict what might be of interest. Filters and hints - at lunch I only want to see high value content, but after work, I don't might conversational humor, but that varies by thread topic. But here we are on HN. I was recently playing python 2 vs 3 games again. So that's an easy "from __past__ import python2", right? Not without scoped method dispatch. Like Ruby's refinements. Here we are using browsers. Self language prototypical programming now available to the masses? Smalltalk-like live coding? Well, sort of. But try inheriting from a built-in "object" - "error: magical method X only works on magical pseudo-object type Y". Browser implementations have sometimes written the fast path, skipped implementing the general case, and considered themselves done. So does the dearth of prototypical-style code mean it's not a useful style, or is inaccessible, or still hasn't been widely tried? Shrug. We increasingly attempt to decompose our big math, algorithm, and data structure libraries along orthogonal and algebraic lines. Sketching lattices of theories, with types and operators and laws. "1 is not an object; + is not a message" (1980's Smalltalk critique - does anyone remember where to find this? a technical memo?). If an algorithm only depends on + and a partial order, it says that. But it's not yet common. How often do you use units, instead of a primary school "what's the resistance? 3! Three what? 3!... volts!". And how many languages let you be explicit about laws? Or help you avoid "well, in wetware I model the problem domain, and in wetware I design the solution, and in wetware I compile the solution to an intermediate form... which is at long last understood by the idiot software compiler of narrowly circumscribed helpfulness". And how many such lattices support pushouts? Add a law, add an operator, and you get the same thing as if done in the other order. Haskell spent what, a half decade?, with lots of commitee and community discussion, and broken code, just to insert Applicative in its type hierarchy. ARPA folks were once considering adding lots of little "lubricant" grants, analogous to the WWII troubleshooters which scurried around industry asking "what are you bottlenecked on? what might speed your work?" A new adminstration, and ARPA had more existential worries. But one strawman was something like "much of our knowledge about how to write good VM's is in person X's head; they've been meaning to write a book, but aren't getting to it; instead of waiting years for that knowledge to slowly diffuse, it would be inexpensive to pay someone to help, sitting outside their office, and stand on their desk". Instead we did the wait for years thing.
- DSingularity 9y agoMozilla has a similar tool, check it out: https://github.com/mozilla/rr https://github.com/mozilla/rr
- libeclipse 9y agoJust waiting for the rust/lisp squad to come along and tell us that we wouldn't need all this if we just joined them.
- steveklabnik 9y agorr works well with Rust code.
- roca 9y agoI'm a notorious Rust fan and also the lead author of rr :-).
- revelation 9y agoOh my god they redesigned WinDbg with a Microsoft Office UI lookalike starter kit including ribbons. And at least one of the screenshots showed JavaScript. So many fingers crossed hoping they don't ruin this indispensable basic tool. Every platform needs to have a no-nonsense debugger that doesn't fall over immediately when it has no symbols and source. WinDbg (was?) that.