6 ms·

The Greatest Bug of All

54 points by naish 18y ago

DenisM 18y ago
Remember that the very first thing you do, when looking at any bug, before you even start thinking about it, and long before you look at your code, is replicate it. You can't debug what you can't replicate, and user reports are usually lacking in some details that your trained eye will catch. So if you don't have a repro, you don't investigate the bug? Not a good way to treat your customers.
- nertzy 18y ago
  I think you're reading it a bit too extremely. His point is that you always try to reproduce the bug before you assume you understand it. His comments don't say anything about what to do if you're unable to reproduce. He's just suggesting in which order he thinks you should take certain actions. It's altogether too easy to jump straight to a patch, submit it, and miss the forest for the trees because you never actually went through the use flow and realized you were patching a symptom, not the underlying problem.
  - DenisM 18y ago
    It's still a dangerous path to go down, at leat in systems software. If your system collapses after 48 hours of stress load, what do you do then? How about 48 days? Still looking for repro? What if customer can't or won't hand over the data? One will do well to learn debugging things from least amount of information - log files, crash dumps etc. Doing this consistently even for the easy bugs with repro will teach developers to put more information into log files and make data structures easier to discover within dumps. Then once hard problems come you will be ready.
    - nertzy 18y ago
      I understand your sentiment but frankly I disagree. Your first paragraph presents a straw man. Sure, in extreme cases, reproducing a bug is not mandatory. Again, the author does not claim that reproducing a bug is mandatory, but rather that it is a useful practice. It is not a virtue to try to base your work on the least amount of information. You say that "when the hard problems come you will be ready". But really when the hard problems come you will look at them assuming that the logged information is enough to solve them. No programmer can know ahead of time what information to log, so by purposefully blindfolding yourself from experiencing the bug directly, you might miss the bigger picture. Indeed, I can't think of a reasonable logging mechanism that the author could have thought of ahead of time that would have helped with this particular bug. Emphasis on "reasonable".
      - DenisM 18y ago
        Rather, when the hard problems come the logged information will be the only thing you will have. Eventually you wil solve all the easy problems with repro, and you will only have hard problems left. It's not a hipotethical, that is actually one step in evolution of systems software.
icey 18y ago
That interlude was great.
- biohacker42 18y ago
  Interludes like that should be part of everything. I want Steve Yegge to have a Scarlett Johansson mini rant in the middle of his blog posts.
  - icey 18y ago
    I'm not gonna lie, it would make me read the whole thing.
- bgutierrez 18y ago
  That interlude completely disrupted my presentation today when someone noticed that my search box had "Jessica Alba" in it. I had no idea who she was, and now the entire engineering deptartment thinks I'm nuts for her.
tptacek 18y ago
Memory mapping is not faster than read(2) for typical access patterns.
- calambrac 18y ago
  So don't use memory mapping unless you know why you're using it, and you know that the code you're writing won't be used to access anything other than local files on disk.
- ajross 18y ago
  More to the point: if the data is really on the disk, an API change isn't going to make the hardware run faster. But a mapped region certainly can be faster for random access reads from a file. Doing a seek() followed by a read() takes two system calls (and thus four context switches) in addition to any I/O overhead. Faulting the page in from an existing mapping instead takes only one interrupt handler at worst, and potentially completes without kernel intervention at all if the page is already mapped.
  - dhess 18y ago
    Typically a system call only leads to a context switch if the call will block, or if the caller has used up its timeslice; otherwise, it's just a couple of relatively cheap protection domain crossings.
    - tptacek 18y ago
      A read from the disk is always going to sleep the process.
      - dhess 18y ago
        Yes, regardless of the mechanism that causes the disk read. Given a particular access pattern, whether the disk gets read or not is independent of the choice to use seek+read or mmap. System calls that don't block don't cause a context switch (1). Therefore, the occurrence of context switches due to disk reads is also independent of the choice to use seek+read or mmap, so trying to avoid context switches isn't a reason to use one or the other. (1) modulo the timeslice expiring during the system call, which could actually help the system call case since there's no need to take an interrupt from user space when the timeslice expires due to a timer: the system call can be predicted since it's in the code stream, but the interrupt can't be. Anyway, that's all in the noise.)
nadim 18y ago
Wil Shipley's blog is recommended reading, all the archives, all of it... great stuff. Well written, relevant and insightful.
culley 18y ago
"Now, since you're not a PhD student, you like money, so this is bad." - great!
ConradHex 18y ago
>The old adage (not mine) is that 99% of the time operating system bugs are actually bugs in your program, and the other 1% of the time they are still bugs in your program, so look harder, dammit. I agree in general, but... I have run into a few platform bugs in my time. For example, at one point realloc in the standard C library that came with Visual Studio would break if you (IIRC) allocated a bunch of blocks that were over 16k, and realloced them to be smaller than 16k. And I had to figure that out the hard way, because the bug was making our code crash. Also, on most game consoles, I'd say about a quarter of the time you think something is a platform bug, it really is. Seriously. Compiler bugs and SDK bugs just aren't all that uncommon there.

54 points by naish 18y ago

Remember that the very first thing you do, when looking at any bug, before you even start thinking about it, and long before you look at your code, is replicate it. You can't debug what you can't replicate, and user reports are usually lacking in some details that your trained eye will catch. So if you don't have a repro, you don't investigate the bug? Not a good way to treat your customers.

nertzy 18y ago
I think you're reading it a bit too extremely. His point is that you always try to reproduce the bug before you assume you understand it. His comments don't say anything about what to do if you're unable to reproduce. He's just suggesting in which order he thinks you should take certain actions. It's altogether too easy to jump straight to a patch, submit it, and miss the forest for the trees because you never actually went through the use flow and realized you were patching a symptom, not the underlying problem.
- DenisM 18y ago
  It's still a dangerous path to go down, at leat in systems software. If your system collapses after 48 hours of stress load, what do you do then? How about 48 days? Still looking for repro? What if customer can't or won't hand over the data? One will do well to learn debugging things from least amount of information - log files, crash dumps etc. Doing this consistently even for the easy bugs with repro will teach developers to put more information into log files and make data structures easier to discover within dumps. Then once hard problems come you will be ready.
  - nertzy 18y ago
    I understand your sentiment but frankly I disagree. Your first paragraph presents a straw man. Sure, in extreme cases, reproducing a bug is not mandatory. Again, the author does not claim that reproducing a bug is mandatory, but rather that it is a useful practice. It is not a virtue to try to base your work on the least amount of information. You say that "when the hard problems come you will be ready". But really when the hard problems come you will look at them assuming that the logged information is enough to solve them. No programmer can know ahead of time what information to log, so by purposefully blindfolding yourself from experiencing the bug directly, you might miss the bigger picture. Indeed, I can't think of a reasonable logging mechanism that the author could have thought of ahead of time that would have helped with this particular bug. Emphasis on "reasonable".
    - DenisM 18y ago
      Rather, when the hard problems come the logged information will be the only thing you will have. Eventually you wil solve all the easy problems with repro, and you will only have hard problems left. It's not a hipotethical, that is actually one step in evolution of systems software.

icey 18y ago

That interlude was great.

biohacker42 18y ago
Interludes like that should be part of everything. I want Steve Yegge to have a Scarlett Johansson mini rant in the middle of his blog posts.
- icey 18y ago
  I'm not gonna lie, it would make me read the whole thing.
bgutierrez 18y ago
That interlude completely disrupted my presentation today when someone noticed that my search box had "Jessica Alba" in it. I had no idea who she was, and now the entire engineering deptartment thinks I'm nuts for her.

tptacek 18y ago

Memory mapping is not faster than read(2) for typical access patterns.

calambrac 18y ago
So don't use memory mapping unless you know why you're using it, and you know that the code you're writing won't be used to access anything other than local files on disk.
ajross 18y ago
More to the point: if the data is really on the disk, an API change isn't going to make the hardware run faster. But a mapped region certainly can be faster for random access reads from a file. Doing a seek() followed by a read() takes two system calls (and thus four context switches) in addition to any I/O overhead. Faulting the page in from an existing mapping instead takes only one interrupt handler at worst, and potentially completes without kernel intervention at all if the page is already mapped.
- dhess 18y ago
  Typically a system call only leads to a context switch if the call will block, or if the caller has used up its timeslice; otherwise, it's just a couple of relatively cheap protection domain crossings.
  - tptacek 18y ago
    A read from the disk is always going to sleep the process.
    - dhess 18y ago
      Yes, regardless of the mechanism that causes the disk read. Given a particular access pattern, whether the disk gets read or not is independent of the choice to use seek+read or mmap. System calls that don't block don't cause a context switch (1). Therefore, the occurrence of context switches due to disk reads is also independent of the choice to use seek+read or mmap, so trying to avoid context switches isn't a reason to use one or the other. (1) modulo the timeslice expiring during the system call, which could actually help the system call case since there's no need to take an interrupt from user space when the timeslice expires due to a timer: the system call can be predicted since it's in the code stream, but the interrupt can't be. Anyway, that's all in the noise.)

nadim 18y ago

Wil Shipley's blog is recommended reading, all the archives, all of it... great stuff. Well written, relevant and insightful.

culley 18y ago

"Now, since you're not a PhD student, you like money, so this is bad." - great!

ConradHex 18y ago

>The old adage (not mine) is that 99% of the time operating system bugs are actually bugs in your program, and the other 1% of the time they are still bugs in your program, so look harder, dammit. I agree in general, but... I have run into a few platform bugs in my time. For example, at one point realloc in the standard C library that came with Visual Studio would break if you (IIRC) allocated a bunch of blocks that were over 16k, and realloced them to be smaller than 16k. And I had to figure that out the hard way, because the bug was making our code crash. Also, on most game consoles, I'd say about a quarter of the time you think something is a platform bug, it really is. Seriously. Compiler bugs and SDK bugs just aren't all that uncommon there.