8 ms·

Sometimes the bug isn't in your code, it's in the CPU

303 points by there 15y ago

daenz 15y ago
Amazing. I'm happy his sanity survived!
jdfreefly 15y ago
First off, I would say that is some pretty awesome work by this guy to chase this down. Including his work with the manufacturer to help them reliably recreate the issue. Second, I would say that over the course of my 10 year career in managing developers, I've heard many, many times that the bug was in the kernel, or in the hardware, or in the complier, or in the other lower level thing the developer had no control over. This has been the correct diagnosis exactly once. If I had to guess, I would say about 5%.
- huhtenberg 15y ago
  I had few bugs traced down to the kernel, on Windows. ALL of them were in 3rd party antivirus packages. In fact, it a machine blue-screened after installing our stuff, you can rest assured it had an antivirus and it was a Kaspersky.
  - wolf550e 15y ago
    Didn't Raymond Chen write once how most Windows crashes (of some version, during some year) were due to beta nvidia drivers installed by gamers to increase FPS?
- warmfuzzykitten 15y ago
  I've been a developer for over 30 years, from mainframes to micros, and a hardware problem has been responsible for a bug in my code exactly zero times.
  - forgotusername 15y ago
    It's interesting how with significantly worse (or at the very least, comparable) complexity to manage, and uniformly horrific costs for repairing production bugs, the ASIC design industry and Intel/AMD in particular have managed to scrape by with something like <20 bugs between them in the past decade. Perhaps we need to incentivize software developers with fear of execution, or something.
    - wmf 15y ago
      A recent x86 processor model has at least 20 "errata" that they'll tell you about; in the last decade they must have had hundreds. But most of them are just worked around so they don't affect you.
    - sdbbp 15y ago
      The amount of effort spent on what is generally called "functional verification" is much higher for hardware than for software. Also, the specifications tend to be clearer and the source code size is smaller than you might imagine.
    - Tuna-Fish 15y ago
      Modern x86 CPUs can trap arbitrary instructions to microcode. This means that most hardware bugs can be fixed with a firmware update that just slows down the cpu somewhat when it encounters the offending instruction. There certainly are a lot of hardware bugs in cpus -- it's just that most of them get fixed before anyone outside the cpu company ever sees them.
  - ryanmolden 15y ago
    Yeah, when I find myself drifting towards the 'maybe it is a bug in the compiler/OS/debugger' territory I know it is time to take a break from the debugging as it is rarely true[1]. Nice to see it occasionally happens :) [1] I work on Visual Studio so I have found compiler/debugger bugs as I am generally using 'in development' bits, but far more often than not the bug turns out to be mine and mine alone :)
  - tomjen3 15y ago
- deleted 15y ago
  [deleted]
- finnw 15y ago
  I have only twice thought one of my problems was due to a compiler bug, and I was right one of those times (and that was because my company was stuck with a 4-year old version of the compiler; The bug had already been fixed in the latest version.)
- VBprogrammer 15y ago
  I would go as far as to say 'finding' bugs in the Compiler / Optimizer / OS / Hardware is a warning signal of a poor programmer. Always expect you are doing it wrong. It will so rarely be the case that this expectation is wrong that you can discount it as insignificant.
  - JOnAgain 15y ago
    Disagree. In fact, were I to come up with a rule of thumb, I'd say the opposite is true. Want to find bugs in Sun's Java 6 compiler for X64 Linux , use annotations (yeah, I found one in their V30 release last week). Want to find bugs in MS' C++ compiler, write your own templates (this was a few years ago, maybe it's better?). The best programmers push the limit of their tools because they know what's "supposed to happen". Poor programmers hit something that doesn't work, and just try something else, cause, well they're just trying shit. I would go so far to say that poor programmers, in fact, are unable to find compiler, optimizer, OS, or hardware bugs because, by definition, they probably don't have a firm handle on what's "supposed to happen".
    - sparsevector 15y ago
      I think what VBprogrammer meant is that thinking you've found a bug in a compiler / OS / CPU is often a warning sign you're a poor programmer. Often times a beginner will have a bug in their code that is too subtle for them to identify, so they end up attributing it to some external factor. Actually finding a bug in a compiler / OS / CPU is as you suggest likely a sign you're doing something advanced or unusual and therefore are perhaps more knowledgeable than most.
      - VBprogrammer 15y ago
        Yeah, that's exactly what I meant. Sorry if the sarcasm didn't quite carry. I know these things can and do happen. I've come across one or two of these strange ones before, but too often I've seen people jump to the conclusion that someone / something else was to blame. Without any other real evidence other than that they have exhausted their shallow back of talent.
- mansr 15y ago
  I have been the first to trigger two CPU bugs and came across a third a few days after it was discovered, before it was published. Once errata are published, software workarounds are usually put in place quickly, and tripping over them is rare. Compiler bugs are another story entirely. I have found dozens of them (confirmed), and I can find more whenever I feel like it.
  - haberman 15y ago
    Out of curiosity, if someone paid you to find compiler bugs for a day, how would you go about it? (I've found several missed-optimization bugs in gcc, but I found them while working on a project where I examine assembly frequently; I have no idea how I'd go about looking for a compiler bug).
    - mansr 15y ago
      One way of actively looking for compiler bugs is using a tool like Csmith[1]. Another is to compile some known-difficult code (e.g. Libav[2]) with various combinations of (optimisation) flags until the test suite fails. Most of the bugs I've found were during routine testing of Libav. While I don't consider missed optimisations bugs as such, they are easy to find. Simply compile some non-trivial function and look at the output. There's usually something that could be done better, especially if some exotic instruction can be used. [1] http://embed.cs.utah.edu/csmith/ http://embed.cs.utah.edu/csmith/ [2] http://libav.org/ http://libav.org/
      - haberman 15y ago
        > While I don't consider missed optimisations bugs as such, they are easy to find. Simply compile some non-trivial function and look at the output. Perhaps you'll give me a little credit :) if I mention that I found missed optimization bugs in extremely trivial functions. One of them involved gcc generating several completely useless stores even at -O3: http://gcc.gnu.org/bugzilla/show_bug.cgi?id=44194 http://gcc.gnu.org/bugzilla/show_bug.cgi?id=44194
        15y ago
- ahi 15y ago
  When I'm working, it's always a bug in the compiler, kernel or hardware. The semicolon was implied. Just give me a minute to work around the compiler bug.
bebop 15y ago
Great job tracking down a hardware bug! That must be really exciting, and you get your name in the AMD errata I assume? One of my comp sci professors found a bug in an Intel chip and got his name in the errata. I think that gives you +100 to nerd credibility :)
- methoddk 15y ago
  +100 nerd cred indeed. That's something that trumps any normal bug.
  - rhizome 15y ago
    He already had it. Matt Dillon is the business, and that's not a fake name of his. He quit FreeBSD-core because they chafed at his awesomeness, from which he went to start a FreeBSD fork, Dragonfly BSD, one of whose goals is process and state portability across CPUs and machines. He was also the technical stud behind Best Internet, one of the earliest and largest and highest-performance ISPs of the Mom&Pop era of the Internet (~93-97).
etrain 15y ago
My hat's off to this guy for the work he did, and indeed, finding a CPU is quite the accomplishment. That said - what is it about the hardware manufacturers that makes them relatively immune to this sort of thing? Is it formal verification and rigid engineering process? Is it that they spend so much money developing these things that they better do them right, god dammit? Sometimes I think that the whole industry would be much better off if everyone up the stack was held to these kinds of standards. If that were the case though, where would we be? We'd have rock solid systems, but how sophisticated would they be? Would UNIX exist? What about (a more bulletproof and less feature complete) Java?
- boyter 15y ago
  Considering it takes years to design a CPU I don't think it would be worth it. The point of software is that you can change it quickly and so it evolves faster then the hardware that runs it. IMO that's an advantage that easily overcomes any instability that new software brings. That said there was an interesting article/interview (cant find it sorry maybe someone else can) with one of the creators of hotmail. Off the top of my head he said that because he came from the hardware side when creating the hotmail software it never broke due to the processes and practices he followed. Perhaps there is a middle ground that we can take and get benefits all around.
  - etrain 15y ago
    On the other hand, you've got FPGAs, which bring the software stuff to the hardware guys - and based on my (limited) understanding of the hardware industry, they've improved certain prototyping exercises by an order of magnitude. Agreed that the rapid development cycles and malleability of the product is part of what's made software great - but my feeling is that maybe we've let that slip a little too much - leading to the bloated, slow, buggy software that everyone runs all the time.
- amackera 15y ago
  Formal verification goes a long way.
- sliverstorm 15y ago
gue5t 15y ago
Here are some more details about this particular bug: http://leaf.dragonflybsd.org/mailarchive/commits/2011-12/msg00259.html http://leaf.dragonflybsd.org/mailarchive/commits/2011-12/msg...
- there 15y ago
  Such fun work to be doing on Christmas day...
jaylevitt 15y ago
As someone who found four compiler bugs in three weeks - in a five-nines fault-tolerant OS, yet! - and who found a PostgreSQL optimizer bug within weeks of learning SQL, I think the key to being "that guy" is playing five-whys with every single bug you encounter. I work with some very talented developers who, when they try something and it doesn't work, try something else. I am fundamentally incapable of that. If it doesn't work, I MUST KNOW WHY. Even if that requires building a debug version of my entire stack, adding all sorts of traces, and wolf-fence debugging until I have a minimal fail case. It's a real limitation; if I hit an undebuggable brick wall, I have no ability to attack the problem from a different angle. Luckily, there are few things that are fundamentally undebuggable.
- jacques_chester 15y ago
  I had to google "wolf-fence debugging". I found I knew of it under a different name: binary search debugging. Git includes built-in support under the bisect command.
  - mbateman 15y ago
    I also looked this up, and found the original paper that coined the term, which starts: > The "Wolf Fence" method of debugging time-sharing programs in higher languages evolved from the "Lions in South Africa" method that I have taught since the vacuum-tube machine language days. It is a quickly converging iteration that serves to catch run-time errors. http://dl.acm.org/citation.cfm?id=358695 http://dl.acm.org/citation.cfm?id=358695 (if you have access) Anyone know what the "Lions in South Africa" method is? I couldn't find it via Google, it just kept turning up references to the same paper.
    - prolepunk 15y ago
      After a quick googling I found this explanation of wolf-fence debugging: http://coreygoldberg.blogspot.com/2008/12/wolf-fence-debugging.html http://coreygoldberg.blogspot.com/2008/12/wolf-fence-debuggi... It stipulates that the state of Alaska has got exactly one wolf, so you build a fence across the middle of the state to find on which side wolf would howl, then subdivide the problem, etc... I'm assuming in your case the wolf got replace with a lion and Alaska with South Africa.
comice 15y ago
Next time my code isn't working as expected, I'm going to shout "cpu bug!" and cite this article.
sjwright 15y ago
When a CPU bug is discovered, what options are available for remedying the situation?
- forgotusername 15y ago
  Patch around it in microcode (applied by the OS or BIOS on every boot; releasable as an OS update), disable the related CPU feature if possible (twiddling bits during OS initialization), or trap any related exception the CPU throws, detect the bug's condition, and patch up the running task's state from the exception handler (again, another OS update). If all else fails, issue a product recall or downplay the bug's severity.
  - troymc 15y ago
    or recall all CPUs and replace them with new repaired ones. That's what they do in Wonderland. :D On a more serious note, I wonder what auto manufacturers do. (There are many CPUs in modern automobiles, and auto manufacturers are often compelled to do recalls.)
    - marshray 15y ago
      Traditional embedded devices (i.e. not smartphones) tend to use very mature and well understood CPUs. Moreso for things holding life and propery like cars. If a bug does occur and cause a crash, normally a watchdog timer will reset the CPU quickly enough to avoid unrecoverable problems. They're certainly not pushing the bleeding edge at all like the 3 GHz desktop/laptop processors.
throwawayderp 15y ago
Nice catch. It would be interesting if he has accidentally triggered a backdoor, such as mentioned in this post. http://theinvisiblethings.blogspot.com.au/2009/03/trusting-hardware.html http://theinvisiblethings.blogspot.com.au/2009/03/trusting-h...
dhruvbird 15y ago
wow! this is quite a rare thing...
16s 15y ago
Please stop referring to him as "this guy", he's well known in the BSD and Linux worlds. He had commit access to FreeBSD before many things we take for granted today even existed. His name is Matt Dillon and he's one hell of a hardware/OS hacker. http://en.wikipedia.org/wiki/Matt_Dillon_%28computer_scientist%29 http://en.wikipedia.org/wiki/Matt_Dillon_%28computer_scienti...
- etrain 15y ago
  I'm an offender of the 'this guy' thing. I have heard of Matt before, but come on, there is almost no context as to who he is given in the posting, and do you really expect everyone in this community to know every semi-significant kernel hacker of the last 2 decades? The link is nice for everyones education, but I, for one, would appreciate a little less condescension.
  - ghshephard 15y ago
    Referring to Matt Dillon at "This guy" is akin to referring to Linus Torvalds, Theo De Raadt, Jony Ives, Zed Shaw, John Gruber, etc.. as "This Guy" - particularly in this community - everyone should know who Matt Dillon is. And, yes, I would expect everyone in this community to recognize who these people are, and roughly what their contributions have been.
    - Vitaly 15y ago
      I'm sorry but no, its not on the same "scale" of being universally known. You see, I'm in this industry from there late 80s, and I know who Lunus is but I have no idea who is Matt Dillon, I was never too much into BSD, mostly using Linux, but I think even most Windows guys know who Linus is but I doubt that many of them even know who is Theo (I do). > And, yes, I would expect everyone in this community to recognize who these people are, and roughly what their contributions have been. I'm sorry but I (and everyone else) don't owe no one any such things. This is excepting too much from people. Yes, not knowing who is Linus, or Bill Gates or Woz would be strange, but its ridiculous to say that Madd Dillon is as well known. I can't (and don't want to) know every significant linux/bsd contributor. I have enough information filling up my limited brains as it is.
bgrainger 15y ago
If you're interested in the types of bugs that are present in modern CPUs, AMD makes their errata documentation publicly available. (As far as I know, Intel's errata are not public. Edit: See tedunangst's comment below for a correction.) The errata documentation for AMD Family 10h Processors (Athlon, Opteron, Phenom, etc.) is here: http://support.amd.com/us/Processor_TechDocs/41322_10h_Rev_Gd.pdf http://support.amd.com/us/Processor_TechDocs/41322_10h_Rev_G... The errata for AMD Family 12h Processors (A-Series APU, etc.): http://support.amd.com/us/Processor_TechDocs/44739_12h_Rev_Gd.pdf http://support.amd.com/us/Processor_TechDocs/44739_12h_Rev_G... I found this out when an AMD engineer confirmed an AMD CPU bug for me: http://stackoverflow.com/questions/7004728/is-this-should-not-happen-crash-an-amd-fusion-cpu-bug http://stackoverflow.com/questions/7004728/is-this-should-no...
- tedunangst 15y ago
  Intel includes errata in updated spec sheets for each CPU.
  - yuhong 15y ago
    "Specification Updates", to be more precise.
- jebblue 15y ago
  AMD should be commended and the guy who found the bug especially. This is how I got into software from the hardware world, in a complex custom system sometimes the bug is in the hardware.
augustl 15y ago
In order to reliably reproduce the bug, he wrote his own operating system. A small one, but still, an operating system. That's pretty badass..
ot 15y ago
Original thread with all the analysis performed before the bug was attributed to the CPU: http://thread.gmane.org/gmane.os.dragonfly-bsd.kernel/14471 http://thread.gmane.org/gmane.os.dragonfly-bsd.kernel/14471 (Check out in particular the section "EFFORTS AT FINDING A KERNEL BUG THAT WASN'T A KERNEL BUG")

303 points by there 15y ago

daenz 15y ago

Amazing. I'm happy his sanity survived!

First off, I would say that is some pretty awesome work by this guy to chase this down. Including his work with the manufacturer to help them reliably recreate the issue. Second, I would say that over the course of my 10 year career in managing developers, I've heard many, many times that the bug was in the kernel, or in the hardware, or in the complier, or in the other lower level thing the developer had no control over. This has been the correct diagnosis exactly once. If I had to guess, I would say about 5%.

huhtenberg 15y ago
I had few bugs traced down to the kernel, on Windows. ALL of them were in 3rd party antivirus packages. In fact, it a machine blue-screened after installing our stuff, you can rest assured it had an antivirus and it was a Kaspersky.
- wolf550e 15y ago
  Didn't Raymond Chen write once how most Windows crashes (of some version, during some year) were due to beta nvidia drivers installed by gamers to increase FPS?
warmfuzzykitten 15y ago
I've been a developer for over 30 years, from mainframes to micros, and a hardware problem has been responsible for a bug in my code exactly zero times.
- forgotusername 15y ago
  It's interesting how with significantly worse (or at the very least, comparable) complexity to manage, and uniformly horrific costs for repairing production bugs, the ASIC design industry and Intel/AMD in particular have managed to scrape by with something like <20 bugs between them in the past decade. Perhaps we need to incentivize software developers with fear of execution, or something.
  - wmf 15y ago
    A recent x86 processor model has at least 20 "errata" that they'll tell you about; in the last decade they must have had hundreds. But most of them are just worked around so they don't affect you.
  - sdbbp 15y ago
    The amount of effort spent on what is generally called "functional verification" is much higher for hardware than for software. Also, the specifications tend to be clearer and the source code size is smaller than you might imagine.
  - Tuna-Fish 15y ago
    Modern x86 CPUs can trap arbitrary instructions to microcode. This means that most hardware bugs can be fixed with a firmware update that just slows down the cpu somewhat when it encounters the offending instruction. There certainly are a lot of hardware bugs in cpus -- it's just that most of them get fixed before anyone outside the cpu company ever sees them.
- ryanmolden 15y ago
  Yeah, when I find myself drifting towards the 'maybe it is a bug in the compiler/OS/debugger' territory I know it is time to take a break from the debugging as it is rarely true[1]. Nice to see it occasionally happens :) [1] I work on Visual Studio so I have found compiler/debugger bugs as I am generally using 'in development' bits, but far more often than not the bug turns out to be mine and mine alone :)
- tomjen3 15y ago
deleted 15y ago
[deleted]
finnw 15y ago
I have only twice thought one of my problems was due to a compiler bug, and I was right one of those times (and that was because my company was stuck with a 4-year old version of the compiler; The bug had already been fixed in the latest version.)
VBprogrammer 15y ago
I would go as far as to say 'finding' bugs in the Compiler / Optimizer / OS / Hardware is a warning signal of a poor programmer. Always expect you are doing it wrong. It will so rarely be the case that this expectation is wrong that you can discount it as insignificant.
- JOnAgain 15y ago
  Disagree. In fact, were I to come up with a rule of thumb, I'd say the opposite is true. Want to find bugs in Sun's Java 6 compiler for X64 Linux , use annotations (yeah, I found one in their V30 release last week). Want to find bugs in MS' C++ compiler, write your own templates (this was a few years ago, maybe it's better?). The best programmers push the limit of their tools because they know what's "supposed to happen". Poor programmers hit something that doesn't work, and just try something else, cause, well they're just trying shit. I would go so far to say that poor programmers, in fact, are unable to find compiler, optimizer, OS, or hardware bugs because, by definition, they probably don't have a firm handle on what's "supposed to happen".
  - sparsevector 15y ago
    I think what VBprogrammer meant is that thinking you've found a bug in a compiler / OS / CPU is often a warning sign you're a poor programmer. Often times a beginner will have a bug in their code that is too subtle for them to identify, so they end up attributing it to some external factor. Actually finding a bug in a compiler / OS / CPU is as you suggest likely a sign you're doing something advanced or unusual and therefore are perhaps more knowledgeable than most.
    - VBprogrammer 15y ago
      Yeah, that's exactly what I meant. Sorry if the sarcasm didn't quite carry. I know these things can and do happen. I've come across one or two of these strange ones before, but too often I've seen people jump to the conclusion that someone / something else was to blame. Without any other real evidence other than that they have exhausted their shallow back of talent.
mansr 15y ago
I have been the first to trigger two CPU bugs and came across a third a few days after it was discovered, before it was published. Once errata are published, software workarounds are usually put in place quickly, and tripping over them is rare. Compiler bugs are another story entirely. I have found dozens of them (confirmed), and I can find more whenever I feel like it.
- haberman 15y ago
  Out of curiosity, if someone paid you to find compiler bugs for a day, how would you go about it? (I've found several missed-optimization bugs in gcc, but I found them while working on a project where I examine assembly frequently; I have no idea how I'd go about looking for a compiler bug).
  - mansr 15y ago
    One way of actively looking for compiler bugs is using a tool like Csmith[1]. Another is to compile some known-difficult code (e.g. Libav[2]) with various combinations of (optimisation) flags until the test suite fails. Most of the bugs I've found were during routine testing of Libav. While I don't consider missed optimisations bugs as such, they are easy to find. Simply compile some non-trivial function and look at the output. There's usually something that could be done better, especially if some exotic instruction can be used. [1] http://embed.cs.utah.edu/csmith/ http://embed.cs.utah.edu/csmith/ [2] http://libav.org/ http://libav.org/
    - haberman 15y ago
      > While I don't consider missed optimisations bugs as such, they are easy to find. Simply compile some non-trivial function and look at the output. Perhaps you'll give me a little credit :) if I mention that I found missed optimization bugs in extremely trivial functions. One of them involved gcc generating several completely useless stores even at -O3: http://gcc.gnu.org/bugzilla/show_bug.cgi?id=44194 http://gcc.gnu.org/bugzilla/show_bug.cgi?id=44194
      - 15y ago
ahi 15y ago
When I'm working, it's always a bug in the compiler, kernel or hardware. The semicolon was implied. Just give me a minute to work around the compiler bug.

bebop 15y ago

Great job tracking down a hardware bug! That must be really exciting, and you get your name in the AMD errata I assume? One of my comp sci professors found a bug in an Intel chip and got his name in the errata. I think that gives you +100 to nerd credibility :)

methoddk 15y ago
+100 nerd cred indeed. That's something that trumps any normal bug.
- rhizome 15y ago
  He already had it. Matt Dillon is the business, and that's not a fake name of his. He quit FreeBSD-core because they chafed at his awesomeness, from which he went to start a FreeBSD fork, Dragonfly BSD, one of whose goals is process and state portability across CPUs and machines. He was also the technical stud behind Best Internet, one of the earliest and largest and highest-performance ISPs of the Mom&Pop era of the Internet (~93-97).

etrain 15y ago

My hat's off to this guy for the work he did, and indeed, finding a CPU is quite the accomplishment. That said - what is it about the hardware manufacturers that makes them relatively immune to this sort of thing? Is it formal verification and rigid engineering process? Is it that they spend so much money developing these things that they better do them right, god dammit? Sometimes I think that the whole industry would be much better off if everyone up the stack was held to these kinds of standards. If that were the case though, where would we be? We'd have rock solid systems, but how sophisticated would they be? Would UNIX exist? What about (a more bulletproof and less feature complete) Java?

boyter 15y ago
Considering it takes years to design a CPU I don't think it would be worth it. The point of software is that you can change it quickly and so it evolves faster then the hardware that runs it. IMO that's an advantage that easily overcomes any instability that new software brings. That said there was an interesting article/interview (cant find it sorry maybe someone else can) with one of the creators of hotmail. Off the top of my head he said that because he came from the hardware side when creating the hotmail software it never broke due to the processes and practices he followed. Perhaps there is a middle ground that we can take and get benefits all around.
- etrain 15y ago
  On the other hand, you've got FPGAs, which bring the software stuff to the hardware guys - and based on my (limited) understanding of the hardware industry, they've improved certain prototyping exercises by an order of magnitude. Agreed that the rapid development cycles and malleability of the product is part of what's made software great - but my feeling is that maybe we've let that slip a little too much - leading to the bloated, slow, buggy software that everyone runs all the time.
amackera 15y ago
Formal verification goes a long way.
sliverstorm 15y ago

gue5t 15y ago

Here are some more details about this particular bug: http://leaf.dragonflybsd.org/mailarchive/commits/2011-12/msg00259.html http://leaf.dragonflybsd.org/mailarchive/commits/2011-12/msg...

there 15y ago
Such fun work to be doing on Christmas day...

jaylevitt 15y ago

As someone who found four compiler bugs in three weeks - in a five-nines fault-tolerant OS, yet! - and who found a PostgreSQL optimizer bug within weeks of learning SQL, I think the key to being "that guy" is playing five-whys with every single bug you encounter. I work with some very talented developers who, when they try something and it doesn't work, try something else. I am fundamentally incapable of that. If it doesn't work, I MUST KNOW WHY. Even if that requires building a debug version of my entire stack, adding all sorts of traces, and wolf-fence debugging until I have a minimal fail case. It's a real limitation; if I hit an undebuggable brick wall, I have no ability to attack the problem from a different angle. Luckily, there are few things that are fundamentally undebuggable.

jacques_chester 15y ago
I had to google "wolf-fence debugging". I found I knew of it under a different name: binary search debugging. Git includes built-in support under the bisect command.
- mbateman 15y ago
  I also looked this up, and found the original paper that coined the term, which starts: > The "Wolf Fence" method of debugging time-sharing programs in higher languages evolved from the "Lions in South Africa" method that I have taught since the vacuum-tube machine language days. It is a quickly converging iteration that serves to catch run-time errors. http://dl.acm.org/citation.cfm?id=358695 http://dl.acm.org/citation.cfm?id=358695 (if you have access) Anyone know what the "Lions in South Africa" method is? I couldn't find it via Google, it just kept turning up references to the same paper.
  - prolepunk 15y ago
    After a quick googling I found this explanation of wolf-fence debugging: http://coreygoldberg.blogspot.com/2008/12/wolf-fence-debugging.html http://coreygoldberg.blogspot.com/2008/12/wolf-fence-debuggi... It stipulates that the state of Alaska has got exactly one wolf, so you build a fence across the middle of the state to find on which side wolf would howl, then subdivide the problem, etc... I'm assuming in your case the wolf got replace with a lion and Alaska with South Africa.

comice 15y ago

Next time my code isn't working as expected, I'm going to shout "cpu bug!" and cite this article.

sjwright 15y ago

When a CPU bug is discovered, what options are available for remedying the situation?

forgotusername 15y ago
Patch around it in microcode (applied by the OS or BIOS on every boot; releasable as an OS update), disable the related CPU feature if possible (twiddling bits during OS initialization), or trap any related exception the CPU throws, detect the bug's condition, and patch up the running task's state from the exception handler (again, another OS update). If all else fails, issue a product recall or downplay the bug's severity.
- troymc 15y ago
  or recall all CPUs and replace them with new repaired ones. That's what they do in Wonderland. :D On a more serious note, I wonder what auto manufacturers do. (There are many CPUs in modern automobiles, and auto manufacturers are often compelled to do recalls.)
  - marshray 15y ago
    Traditional embedded devices (i.e. not smartphones) tend to use very mature and well understood CPUs. Moreso for things holding life and propery like cars. If a bug does occur and cause a crash, normally a watchdog timer will reset the CPU quickly enough to avoid unrecoverable problems. They're certainly not pushing the bleeding edge at all like the 3 GHz desktop/laptop processors.

throwawayderp 15y ago

Nice catch. It would be interesting if he has accidentally triggered a backdoor, such as mentioned in this post. http://theinvisiblethings.blogspot.com.au/2009/03/trusting-hardware.html http://theinvisiblethings.blogspot.com.au/2009/03/trusting-h...

dhruvbird 15y ago

wow! this is quite a rare thing...

16s 15y ago

Please stop referring to him as "this guy", he's well known in the BSD and Linux worlds. He had commit access to FreeBSD before many things we take for granted today even existed. His name is Matt Dillon and he's one hell of a hardware/OS hacker. http://en.wikipedia.org/wiki/Matt_Dillon_%28computer_scientist%29 http://en.wikipedia.org/wiki/Matt_Dillon_%28computer_scienti...

etrain 15y ago
I'm an offender of the 'this guy' thing. I have heard of Matt before, but come on, there is almost no context as to who he is given in the posting, and do you really expect everyone in this community to know every semi-significant kernel hacker of the last 2 decades? The link is nice for everyones education, but I, for one, would appreciate a little less condescension.
- ghshephard 15y ago
  Referring to Matt Dillon at "This guy" is akin to referring to Linus Torvalds, Theo De Raadt, Jony Ives, Zed Shaw, John Gruber, etc.. as "This Guy" - particularly in this community - everyone should know who Matt Dillon is. And, yes, I would expect everyone in this community to recognize who these people are, and roughly what their contributions have been.
  - Vitaly 15y ago
    I'm sorry but no, its not on the same "scale" of being universally known. You see, I'm in this industry from there late 80s, and I know who Lunus is but I have no idea who is Matt Dillon, I was never too much into BSD, mostly using Linux, but I think even most Windows guys know who Linus is but I doubt that many of them even know who is Theo (I do). > And, yes, I would expect everyone in this community to recognize who these people are, and roughly what their contributions have been. I'm sorry but I (and everyone else) don't owe no one any such things. This is excepting too much from people. Yes, not knowing who is Linus, or Bill Gates or Woz would be strange, but its ridiculous to say that Madd Dillon is as well known. I can't (and don't want to) know every significant linux/bsd contributor. I have enough information filling up my limited brains as it is.

bgrainger 15y ago

If you're interested in the types of bugs that are present in modern CPUs, AMD makes their errata documentation publicly available. (As far as I know, Intel's errata are not public. Edit: See tedunangst's comment below for a correction.) The errata documentation for AMD Family 10h Processors (Athlon, Opteron, Phenom, etc.) is here: http://support.amd.com/us/Processor_TechDocs/41322_10h_Rev_Gd.pdf http://support.amd.com/us/Processor_TechDocs/41322_10h_Rev_G... The errata for AMD Family 12h Processors (A-Series APU, etc.): http://support.amd.com/us/Processor_TechDocs/44739_12h_Rev_Gd.pdf http://support.amd.com/us/Processor_TechDocs/44739_12h_Rev_G... I found this out when an AMD engineer confirmed an AMD CPU bug for me: http://stackoverflow.com/questions/7004728/is-this-should-not-happen-crash-an-amd-fusion-cpu-bug http://stackoverflow.com/questions/7004728/is-this-should-no...

tedunangst 15y ago
Intel includes errata in updated spec sheets for each CPU.
- yuhong 15y ago
  "Specification Updates", to be more precise.
jebblue 15y ago
AMD should be commended and the guy who found the bug especially. This is how I got into software from the hardware world, in a complex custom system sometimes the bug is in the hardware.

augustl 15y ago

In order to reliably reproduce the bug, he wrote his own operating system. A small one, but still, an operating system. That's pretty badass..

ot 15y ago

Original thread with all the analysis performed before the bug was attributed to the CPU: http://thread.gmane.org/gmane.os.dragonfly-bsd.kernel/14471 http://thread.gmane.org/gmane.os.dragonfly-bsd.kernel/14471 (Check out in particular the section "EFFORTS AT FINDING A KERNEL BUG THAT WASN'T A KERNEL BUG")