4 ms·

So it's said to take days of stress testing for the crash to occur each time. I've had to trace bugs under emulation that took substantial amounts of time to t

by byuu 12y ago

So it's said to take days of stress testing for the crash to occur each time. I've had to trace bugs under emulation that took substantial amounts of time to trigger. Thankfully in my case, I was able to serialize the entire system into snapshots. And so by just taking a snapshot every few minutes, the one right before a crash allowed me to be able to quickly reproduce the problem. Does this crash event occur inside of VMs as well, and if so, could VM snapshots be used to accelerate the bug hunt here?

xuhu 12y ago
Honest question: what guarantees that if the bug reproduced 10 minutes after a snapshot, it will reproduce again if you resume running nondeterminstically after that snapshot ?
- deleted 12y ago
  [deleted]
- panzi 12y ago
  Probably nothing, but it's worth a try.
- washadjeffmad 12y ago
  Nothing, but it reduces the conditions of the variables, which is handier than nothing.
- alexhill 12y ago
  Well, nothing, but that's bug hunting: you try it and see. Whether or not this helps to recreate the problem on demand, you know more than you did before.
- byuu 12y ago
  That's always an issue, but in my testing with emulators, once you rule out external inputs (eg keyboard input), it tends to become predictable and reproducible. And using a VM should help dramatically in reducing randomness. And you keep capturing states and narrowing the window until you can reproduce it in eg ten seconds; as too much can happen in the span of ten minutes. The trick is that you have to get a state right before the problem actually begins, but you don't yet know where that is. So it's possible your state capture may be a bit too late, after the issue already eg corrupted memory somewhere. I know that there are certainly bugs that this kind of technique would never work on. I've hit a few bugs that could only be triggered on live hardware before. But I'm curious if they've tried this kind of approach for this bug yet or not.
  - rwmj 12y ago
    Could you post more about your technique? Like, what emulator are you using (qemu)? How do you trigger the snapshots? Are you snapshotting memory or disk or both? How much disk space is consumed by all these snapshots? Do you discard snapshots? How do you know when the bug has been triggered?
    - CJefferson 12y ago
      Byuu is the author of the excellent SNES (Super Nintendo) emulator bsnes. I would guess he is referring to emulation errors in emulators, where snapshot support is common, and usually of good quality.
    - byuu 12y ago
      Sure, it's my own software. I hook up saving and loading states to key presses, so eg F5 would save, F7 would load. And then F6/F8 would increment or decrement the save slot number. You basically have to capture everything possible: all memory, the state of all the CPU registers, the state of all hardware registers, etc. Obviously disk would be a real challenge, where you'd have to keep a delta list of disk changes since program start, or simply not serialize that state. If you miss anything, you can have problems loading states correctly. However, there is quite a bit of tolerance between theory and practice, so if there is something that you really can't capture the state of for some reason (like a hardware write-only register when you weren't logging what was previously written to it), a lot of times you can get away with it anyway. Because the system I am using is so old, snapshots are only 300KB each. Sometimes I dump them to disk, sometimes I just keep them in RAM. I know that a PC would be much more challenging, given how much more hardware is at play, and because VMs aren't quite the same as pure software emulation like qemu (though you could potentially use qemu for this too), but VM software does implement this snapshot system, so clearly it's possible. You know when the bug is triggered through the visual output. And what's cool is that by saving periodic snapshots automatically to a ring buffer, you can code a special keypress to "rewind" the program. So it crashes, then you go back a bit, and save a disk snapshot there, and wait to see if the bug repeats. If it does, then you turn on trace logging and dump all of the CPU instructions between your save point and the crash. Then you go to the crash point, and slowly work your way back to try and find out where things went wrong.
- rhelmer 12y ago
  Reversible debugging might help more here than snapshots, since it allows you to record a failure and then debug it: http://rr-project.org/ http://rr-project.org/
heywire 12y ago
This is an approach I use for reproducing issues on userspace applications as well. I happen to maintain a fairly large Windows-based application at my workplace, and sometimes the steps to reproduce an issue could take minutes to hours just to get you to the point of being able to try and replicate the issue. I personally use VMware Workstation at work, and since I'm on a quick SSD, taking and reverting multiple snapshots has little performance cost.
- byuu 12y ago
  I've always thought it would be nice if the kernel exposed this ability to regular userland applications. On Unix systems, Ctrl+Z of a terminal application will suspend it, but not across reboots. Core files are very close, yet typically they are only produced in response to a crash, and aren't really meant for resuming a process. Obviously there's lots of limitations (eg shared memory could cause issues, hardware states could change), but for a lot of cases it would probably work pretty well. Not only is it great for triggering crashes, but if you then broke out into a debugger, you could toy with toggling register and memory contents and then single-step through the instructions to see what happens. It's really substantially easier to debug software under emulation because of features like these.
  - sweettea 12y ago
    You might enjoy rr, 'record-replay', which works kind of like gdb (and I believe is based on it) after you record a program execution --- you can then use gdb multiple times on the stream of execution. rr-project.org.
  - amaranth 12y ago
    Sounds like you're looking for the checkpoint/restore functionality that has been in the works for a few years now. Check out http://en.wikipedia.org/wiki/CRIU http://en.wikipedia.org/wiki/CRIU for more details, seems they have things mostly working now.
  - ams6110 12y ago
    DragonFly BSD does this. They call it process checkpointing. Processes under DragonFly may be "checkpointed" or suspended to disk at any time. They may later be resumed on the originating system, or another system by "thawing" them. http://www.dragonflybsd.org/features/#index8h2 http://www.dragonflybsd.org/features/#index8h2

by byuu 12y ago

xuhu 12y ago

Honest question: what guarantees that if the bug reproduced 10 minutes after a snapshot, it will reproduce again if you resume running nondeterminstically after that snapshot ?

deleted 12y ago
[deleted]
panzi 12y ago
Probably nothing, but it's worth a try.
washadjeffmad 12y ago
Nothing, but it reduces the conditions of the variables, which is handier than nothing.
alexhill 12y ago
Well, nothing, but that's bug hunting: you try it and see. Whether or not this helps to recreate the problem on demand, you know more than you did before.
byuu 12y ago
That's always an issue, but in my testing with emulators, once you rule out external inputs (eg keyboard input), it tends to become predictable and reproducible. And using a VM should help dramatically in reducing randomness. And you keep capturing states and narrowing the window until you can reproduce it in eg ten seconds; as too much can happen in the span of ten minutes. The trick is that you have to get a state right before the problem actually begins, but you don't yet know where that is. So it's possible your state capture may be a bit too late, after the issue already eg corrupted memory somewhere. I know that there are certainly bugs that this kind of technique would never work on. I've hit a few bugs that could only be triggered on live hardware before. But I'm curious if they've tried this kind of approach for this bug yet or not.
- rwmj 12y ago
  Could you post more about your technique? Like, what emulator are you using (qemu)? How do you trigger the snapshots? Are you snapshotting memory or disk or both? How much disk space is consumed by all these snapshots? Do you discard snapshots? How do you know when the bug has been triggered?
  - CJefferson 12y ago
    Byuu is the author of the excellent SNES (Super Nintendo) emulator bsnes. I would guess he is referring to emulation errors in emulators, where snapshot support is common, and usually of good quality.
  - byuu 12y ago
    Sure, it's my own software. I hook up saving and loading states to key presses, so eg F5 would save, F7 would load. And then F6/F8 would increment or decrement the save slot number. You basically have to capture everything possible: all memory, the state of all the CPU registers, the state of all hardware registers, etc. Obviously disk would be a real challenge, where you'd have to keep a delta list of disk changes since program start, or simply not serialize that state. If you miss anything, you can have problems loading states correctly. However, there is quite a bit of tolerance between theory and practice, so if there is something that you really can't capture the state of for some reason (like a hardware write-only register when you weren't logging what was previously written to it), a lot of times you can get away with it anyway. Because the system I am using is so old, snapshots are only 300KB each. Sometimes I dump them to disk, sometimes I just keep them in RAM. I know that a PC would be much more challenging, given how much more hardware is at play, and because VMs aren't quite the same as pure software emulation like qemu (though you could potentially use qemu for this too), but VM software does implement this snapshot system, so clearly it's possible. You know when the bug is triggered through the visual output. And what's cool is that by saving periodic snapshots automatically to a ring buffer, you can code a special keypress to "rewind" the program. So it crashes, then you go back a bit, and save a disk snapshot there, and wait to see if the bug repeats. If it does, then you turn on trace logging and dump all of the CPU instructions between your save point and the crash. Then you go to the crash point, and slowly work your way back to try and find out where things went wrong.
rhelmer 12y ago
Reversible debugging might help more here than snapshots, since it allows you to record a failure and then debug it: http://rr-project.org/ http://rr-project.org/

heywire 12y ago

This is an approach I use for reproducing issues on userspace applications as well. I happen to maintain a fairly large Windows-based application at my workplace, and sometimes the steps to reproduce an issue could take minutes to hours just to get you to the point of being able to try and replicate the issue. I personally use VMware Workstation at work, and since I'm on a quick SSD, taking and reverting multiple snapshots has little performance cost.

byuu 12y ago
I've always thought it would be nice if the kernel exposed this ability to regular userland applications. On Unix systems, Ctrl+Z of a terminal application will suspend it, but not across reboots. Core files are very close, yet typically they are only produced in response to a crash, and aren't really meant for resuming a process. Obviously there's lots of limitations (eg shared memory could cause issues, hardware states could change), but for a lot of cases it would probably work pretty well. Not only is it great for triggering crashes, but if you then broke out into a debugger, you could toy with toggling register and memory contents and then single-step through the instructions to see what happens. It's really substantially easier to debug software under emulation because of features like these.
- sweettea 12y ago
  You might enjoy rr, 'record-replay', which works kind of like gdb (and I believe is based on it) after you record a program execution --- you can then use gdb multiple times on the stream of execution. rr-project.org.
- amaranth 12y ago
  Sounds like you're looking for the checkpoint/restore functionality that has been in the works for a few years now. Check out http://en.wikipedia.org/wiki/CRIU http://en.wikipedia.org/wiki/CRIU for more details, seems they have things mostly working now.
- ams6110 12y ago
  DragonFly BSD does this. They call it process checkpointing. Processes under DragonFly may be "checkpointed" or suspended to disk at any time. They may later be resumed on the originating system, or another system by "thawing" them. http://www.dragonflybsd.org/features/#index8h2 http://www.dragonflybsd.org/features/#index8h2