7 ms·

The mystery of the fifteen-millisecond breakpoint instruction

148 points by jsnell 11y ago

caf 11y ago
I'm surprised the author didn't submit this patch back to the Linux ARM architecture maintainers - it seems entirely apropos.
- raverbashing 11y ago
  Yes, or at least a discussion on why this instruction is not supported (might be just a slip-up or have a bigger underlying reason)
notacoward 11y ago
There's something extra-satisfying about debugging a debug facility. Nice detective work.
userbinator 11y ago
What exactly is the purpose of logging all unhandled breakpoint instructions encountered in user-mode code to the syslog, and then continuing...? On Windows, a breakpoint instruction encountered outside of a debugger causes the standard error dialog to appear, and if you choose not to attach a debugger, the process terminates. IMHO that's more reasonable behaviour, as breakpoint instructions shouldn't appear in code that's not being debugged (either by a debugger or itself, as is sometimes the case like the author does here.) because unprivileged user-space code is able to force the kernel to waste a lot of time printing messages, slowing the whole system down User code can do that already, or even just run in an infinite loop.
- dezgeg 11y ago
  >>because unprivileged user-space code is able to force >>the kernel to waste a lot of time printing messages, >>slowing the whole system down >User code can do that already[...] I'm pretty sure that'd be considered a kernel bug. After all, there already is a a function (printk_ratelimited) which is supposed to be used in situations where userspace can trigger a kernel print.
  - userbinator 11y ago
    I meant that you can already do these if you want to consume a lot of CPU time: for(;;) printf("Hello world!"); for(;;) ;
    - seba_dos1 11y ago
      Consuming a lot of CPU time and forcing kernel to do that are separate things.
    - madez 11y ago
      But that won't slow down the whole system, if you didn't mess with the process priorities.
      - lultimouomo 11y ago
        sync() will most definitely do.
- caf 11y ago
  It doesn't continue, it delivers a SIGTRAP to the process and the default disposition of that signal (if the process isn't being debugged or has otherwise arranged to handle the signal differently) is to have the process dump core and exit.
TheLoneWolfling 11y ago
Wait... It takes ~1ms to write a line to a log file?
- anonyfox 11y ago
  disk is slow, seems reasonable.
  - TheLoneWolfling 11y ago
    Don't forget the Pi (generally) uses flash. So seek speeds are irrelevant. If you're writing the following in 1ms: Apr 4 22:50:59 heating kernel: [14572.940927] Unhandled prefetch abort: breakpoint debug exception (0x002) at 0x00008438 By my count that's ~1 kilobit. That means that you're writing to flash at ~1 Mb/s. (~125KB/s) That's slow, to put it mildly. (~5% of what it should be, according to http://ozzmaker.com/2012/12/27/how-to-test-the-sd-card-speed-on-your-raspberry-pi/ http://ozzmaker.com/2012/12/27/how-to-test-the-sd-card-speed... ) I could see that kind of speed if it was causing a disk seek every write (although if it was doing that you're probably using the wrong file system), but again, the Pi uses flash. So what gives? Is there braindead sync behavior that's causing it? Is it hitting a pathological case in the flash controller? Or what? (For that matter: is there a file system that is optimized for log files?)
    - tjoff 11y ago
      Just read the second comment on the link you provided and you have the answer.
      - TheLoneWolfling 11y ago
        > Nowaday’s fast Class 10 cards achieve their high read and write speeds only by buffering in advance, assuming that the subsequent blocks will be read or written next. That means they inevitably sacrifice random read and write speeds. This effect is in the order of several magnitudes. For instance, a card with sequential read and write speeds of 10 MB/s might collapse to just 0.01 MB/s for random access. Yow. Yet another example of over-optimization to benchmarks. ...Except that, even then, how exactly is appending to a log file repeatedly random access? Bad journaling?
glandium 11y ago
This reminds me of one thing I hit 2 years ago, where several android mobile devices would be slow to call signal handlers, because the kernel was booted with user_debug=31(on production devices), which printk stack traces before calling signal handlers. The kernel command line argument was hardcoded on some code aurora branches for Qualcomm boards, iirc.
deleted 11y ago
[deleted]
- RobotCaleb 11y ago
  Is that necessary? Do try less hard, please.
makomk 11y ago
There's apparently an even worse usermode-triggerable denial of service in the Pi related to breakpoints, one that allows usermode code to completely hang it: http://hackaday.com/2015/08/25/raspberry-pi-halt-and-catch-well-halt/ http://hackaday.com/2015/08/25/raspberry-pi-halt-and-catch-w...
trendandbeyond 11y ago
Testing
evmar 11y ago
He wrote: > Think about the common debugging scenario where the user sets a conditional breakpoint: "break if x > y". Testing that condition is going to take 15ms each time. Wouldn't the implementation of a software breakpoint look like: if x > y: bkpt so it'd only be slow on the case where it needed to break? I guess I can imagine you could implement it the other way (always break, check the condition after breaking, resume if the condition isn't met) but it's not obvious to me why you'd do it that way. Just guessing, but if you're just patching the binary to insert the breakpoint you might not have enough space? But you need space for the "bkpt" instruction too...
- derf_ 11y ago
  I don't know what gdb does, but even on x86 a conditional breakpoint can be ridiculously slow. On a loop that executes a few hundred thousand times before meeting the condition, it can take minutes to evaluate something that executed instantly before. I often go back, edit the code to add something like if (condition) { foo = foo; } recompile, restart the debugger, and break on the foo = foo line. Even with all of those steps, that's still faster than using a conditional breakpoint most of the time.
  - revelation 11y ago
    For next time, Windows has the DebugBreak function which will stop the program and (if you have a debugger attached) act the same way as stopping on that line. On Linux, I've seen some people use asm("int 3") (which is what a debugger would use for software breakpoints).
  - comex 11y ago
    Incidentally, GDB can sometimes evaluate breakpoint conditions on the target: https://sourceware.org/gdb/onlinedocs/gdb/Conditions.html https://sourceware.org/gdb/onlinedocs/gdb/Conditions.html (search for "target's side") I don't know what targets support it, though. (Pretty sure OS X isn't one of them.)
  - spc476 11y ago
    Worse case scenario---the CPU only supports a breakpoint instruction. Doing a conditional breakpoint involves the debugger setting a breakpoint instruction at the next instruction, resuming, then resetting the breakpoint and setting a breakpoint at the next instruction (with special care taken for calls, jumps and conditional jumps) and doing this, for every instruction until the condition is met. Second worst case---the CPU has both breakpoints and single stepping mode (the 8086 supports this). In this case, the debugger no longer needs to set a breakpoint on every instruction, but the CPU still endures an interrupt per instruction until the condition is met. Best case---the CPU has a way to specify a condition in hardware (the 80386 and above can do this to varying degrees) but often you have a limited number of conditions that can be handled and if you exceed this, then you either have to disallow it, or fall back to one of the previous methods (or, perhaps depending upon the condition, you can play around with page settings to capture reads or writes but that has its own overhead).
AceJohnny2 11y ago
Very tangential to the topic at hand, it makes me happy to see a somewhat lucid comment on that article by Terry A Davis. (For those who lack context, Davis is a brilliant but schizophrenic programmer who has some notoriety on HN: https://news.ycombinator.com/item?id=7818823 https://news.ycombinator.com/item?id=7818823 More info at http://motherboard.vice.com/read/gods-lonely-programmer http://motherboard.vice.com/read/gods-lonely-programmer )

148 points by jsnell 11y ago

caf 11y ago

I'm surprised the author didn't submit this patch back to the Linux ARM architecture maintainers - it seems entirely apropos.

raverbashing 11y ago
Yes, or at least a discussion on why this instruction is not supported (might be just a slip-up or have a bigger underlying reason)

notacoward 11y ago

There's something extra-satisfying about debugging a debug facility. Nice detective work.

userbinator 11y ago

What exactly is the purpose of logging all unhandled breakpoint instructions encountered in user-mode code to the syslog, and then continuing...? On Windows, a breakpoint instruction encountered outside of a debugger causes the standard error dialog to appear, and if you choose not to attach a debugger, the process terminates. IMHO that's more reasonable behaviour, as breakpoint instructions shouldn't appear in code that's not being debugged (either by a debugger or itself, as is sometimes the case like the author does here.) because unprivileged user-space code is able to force the kernel to waste a lot of time printing messages, slowing the whole system down User code can do that already, or even just run in an infinite loop.

dezgeg 11y ago
>>because unprivileged user-space code is able to force >>the kernel to waste a lot of time printing messages, >>slowing the whole system down >User code can do that already[...] I'm pretty sure that'd be considered a kernel bug. After all, there already is a a function (printk_ratelimited) which is supposed to be used in situations where userspace can trigger a kernel print.
- userbinator 11y ago
  I meant that you can already do these if you want to consume a lot of CPU time: for(;;) printf("Hello world!"); for(;;) ;
  - seba_dos1 11y ago
    Consuming a lot of CPU time and forcing kernel to do that are separate things.
  - madez 11y ago
    But that won't slow down the whole system, if you didn't mess with the process priorities.
    - lultimouomo 11y ago
      sync() will most definitely do.
caf 11y ago
It doesn't continue, it delivers a SIGTRAP to the process and the default disposition of that signal (if the process isn't being debugged or has otherwise arranged to handle the signal differently) is to have the process dump core and exit.

TheLoneWolfling 11y ago

Wait... It takes ~1ms to write a line to a log file?

anonyfox 11y ago
disk is slow, seems reasonable.
- TheLoneWolfling 11y ago
  Don't forget the Pi (generally) uses flash. So seek speeds are irrelevant. If you're writing the following in 1ms: Apr 4 22:50:59 heating kernel: [14572.940927] Unhandled prefetch abort: breakpoint debug exception (0x002) at 0x00008438 By my count that's ~1 kilobit. That means that you're writing to flash at ~1 Mb/s. (~125KB/s) That's slow, to put it mildly. (~5% of what it should be, according to http://ozzmaker.com/2012/12/27/how-to-test-the-sd-card-speed-on-your-raspberry-pi/ http://ozzmaker.com/2012/12/27/how-to-test-the-sd-card-speed... ) I could see that kind of speed if it was causing a disk seek every write (although if it was doing that you're probably using the wrong file system), but again, the Pi uses flash. So what gives? Is there braindead sync behavior that's causing it? Is it hitting a pathological case in the flash controller? Or what? (For that matter: is there a file system that is optimized for log files?)
  - tjoff 11y ago
    Just read the second comment on the link you provided and you have the answer.
    - TheLoneWolfling 11y ago
      > Nowaday’s fast Class 10 cards achieve their high read and write speeds only by buffering in advance, assuming that the subsequent blocks will be read or written next. That means they inevitably sacrifice random read and write speeds. This effect is in the order of several magnitudes. For instance, a card with sequential read and write speeds of 10 MB/s might collapse to just 0.01 MB/s for random access. Yow. Yet another example of over-optimization to benchmarks. ...Except that, even then, how exactly is appending to a log file repeatedly random access? Bad journaling?

glandium 11y ago

This reminds me of one thing I hit 2 years ago, where several android mobile devices would be slow to call signal handlers, because the kernel was booted with user_debug=31(on production devices), which printk stack traces before calling signal handlers. The kernel command line argument was hardcoded on some code aurora branches for Qualcomm boards, iirc.

deleted 11y ago

[deleted]

RobotCaleb 11y ago
Is that necessary? Do try less hard, please.

makomk 11y ago

There's apparently an even worse usermode-triggerable denial of service in the Pi related to breakpoints, one that allows usermode code to completely hang it: http://hackaday.com/2015/08/25/raspberry-pi-halt-and-catch-well-halt/ http://hackaday.com/2015/08/25/raspberry-pi-halt-and-catch-w...

trendandbeyond 11y ago

Testing

evmar 11y ago

He wrote: > Think about the common debugging scenario where the user sets a conditional breakpoint: "break if x > y". Testing that condition is going to take 15ms each time. Wouldn't the implementation of a software breakpoint look like: if x > y: bkpt so it'd only be slow on the case where it needed to break? I guess I can imagine you could implement it the other way (always break, check the condition after breaking, resume if the condition isn't met) but it's not obvious to me why you'd do it that way. Just guessing, but if you're just patching the binary to insert the breakpoint you might not have enough space? But you need space for the "bkpt" instruction too...

derf_ 11y ago
I don't know what gdb does, but even on x86 a conditional breakpoint can be ridiculously slow. On a loop that executes a few hundred thousand times before meeting the condition, it can take minutes to evaluate something that executed instantly before. I often go back, edit the code to add something like if (condition) { foo = foo; } recompile, restart the debugger, and break on the foo = foo line. Even with all of those steps, that's still faster than using a conditional breakpoint most of the time.
- revelation 11y ago
  For next time, Windows has the DebugBreak function which will stop the program and (if you have a debugger attached) act the same way as stopping on that line. On Linux, I've seen some people use asm("int 3") (which is what a debugger would use for software breakpoints).
- comex 11y ago
  Incidentally, GDB can sometimes evaluate breakpoint conditions on the target: https://sourceware.org/gdb/onlinedocs/gdb/Conditions.html https://sourceware.org/gdb/onlinedocs/gdb/Conditions.html (search for "target's side") I don't know what targets support it, though. (Pretty sure OS X isn't one of them.)
- spc476 11y ago
  Worse case scenario---the CPU only supports a breakpoint instruction. Doing a conditional breakpoint involves the debugger setting a breakpoint instruction at the next instruction, resuming, then resetting the breakpoint and setting a breakpoint at the next instruction (with special care taken for calls, jumps and conditional jumps) and doing this, for every instruction until the condition is met. Second worst case---the CPU has both breakpoints and single stepping mode (the 8086 supports this). In this case, the debugger no longer needs to set a breakpoint on every instruction, but the CPU still endures an interrupt per instruction until the condition is met. Best case---the CPU has a way to specify a condition in hardware (the 80386 and above can do this to varying degrees) but often you have a limited number of conditions that can be handled and if you exceed this, then you either have to disallow it, or fall back to one of the previous methods (or, perhaps depending upon the condition, you can play around with page settings to capture reads or writes but that has its own overhead).

AceJohnny2 11y ago

Very tangential to the topic at hand, it makes me happy to see a somewhat lucid comment on that article by Terry A Davis. (For those who lack context, Davis is a brilliant but schizophrenic programmer who has some notoriety on HN: https://news.ycombinator.com/item?id=7818823 https://news.ycombinator.com/item?id=7818823 More info at http://motherboard.vice.com/read/gods-lonely-programmer http://motherboard.vice.com/read/gods-lonely-programmer )