6 ms·

Tracking down a kernel bug with git bisect

204 points by fauria 12y ago

incision 12y ago
Another article on bisecting which I bookmarked last year, but with physical hardware [1]. 1: http://rtg.in.ua/blog/kernel-bisect-pxe/ http://rtg.in.ua/blog/kernel-bisect-pxe/
CUViper 12y ago
One tip for the module configuration: kbuild has "make localyesconfig" which looks at your currently-loaded modules and converts them to CONFIG_FOO=Y builtins. See also "make help" for more available targets.
tlrobinson 12y ago
The first time I used "git bisect run" I felt like I was cheating. Unfortunately (or fortunately) I don't have many opportunities to use it.
- chriscool 12y ago
  There is this LWN article (that I wrote about it): http://lwn.net/Articles/317154/ http://lwn.net/Articles/317154/
tieTYT 12y ago
One thing that seems difficult with bisect are these series of steps: 1. "Hey... this bug didn't use to happen. Where'd that get introduced?" 2. OK, this unit test will tell me when the bug is fixed. 3. Now lets automate bisect to tell me where this test first failed even though I just wrote it. How do you do that? I think as long as you DON'T commit your unit test, bisect will carry over to every commit check. But, you'll have to make sure you commit all your other changes or else they'll be carried around, too: Conflict-city. Also, if you do commit your unit test (a "bad" habit of mine) I have no idea how I'm supposed to work with this. I end up copying the unit test by hand to each commit it tries to test. In summary, it seems like bisect was built with manual testing in mind. I know you can automate running a shell script, but I don't write my automated tests in shell script. EDIT: Keep in mind, not all projects are interpreted. Some are compiled.
- ajross 12y ago
  The whole idea of bisection is that you have an objective behavior that has changed. If the test (which is, by definition, the tool used to measure that objective behavior) is not constant across the range of changes, then you simply don't have a problem that can be solved by bisection. You're breaking rule one ("This bug" isn't a single behavior). Obviously in practice you'd probably just use whatever the most recent version of the test is. Really I think you're problem is that you're overspecifying rule 2. It's not about identifying a "unit test" per se, it's about identifying any specific process to detect the bug. If the tests in your tree work, great. If not, you may have to do a little development to make one that is automatable.
  - taeric 12y ago
    I think you misunderstood. The test is brand new. As in, you find a problem that nobody has identified before and want to see how/when it was introduced. Easy way to do this is put your test external to the tree. Then, bisect as normal for git, and run your test. This is probably not uncommon for kernel tests, as you often have a test which could be "does this random laptop resume from sleep correctly?" Not really easy to automate that. And doesn't live in your tree.
    - Hello71 12y ago
      > "does this random laptop resume from sleep correctly?" Not really easy to automate that. that's what rtcwake is for
      - deleted 12y ago
        [deleted]
      - taeric 12y ago
        Right, my point was more that often everything will be working great for the main developer's laptop. I did not think it was uncommon to have someone come out of the weeds with a specific laptop that doesn't work. Or worse, it works if you sleep using the "sleep" button, but not if you close the lid. So, yes, you may be able to automate it later, once you fully understand the specifics that cause it to not rewake. Possibly specific to how it went to sleep with the lid close. Until then, it is nice to be able to do ad hoc tests when necessary.
Arnavion 12y ago
It's a pity that while the bisect did point in the general direction of where the bug was in 3.15, it didn't actually find the right bug (the one it found had already been fixed and a different one introduced that led to the same errant behavior). With something as complex as the kernel, I also wonder about automatic bisect being led astray by unrelated changes. The bisect script might fail for a different reason and zone in on the wrong commit.
- chriscool 12y ago
  The bisect pointed at the right patch series which is already very nice. If the bisect script fails because of another problem, it is often possible to refine the script, or run bisect again on a fixed up branch.
  - Arnavion 12y ago
    If I'm not mistaken, it didn't find the right patch series. The article says the function changed in 3.15 and the underlying reason for his issue was different and not part of that series.
    - seabee 12y ago
      When you update the kernel and your program stops working, there are two possibilities: 1) it's a kernel bug; there is a regression in the kernel's documented behaviour 2) it's a bug in your tool because it took advantage of undefined or undocumented behaviour The bisect found exactly the right patch series, but bisect can only answer the question 'when did it stop working', not 'why'. You can read about many instances of 2) on Raymond Chen's blog The Old New Thing.
djhoffma 12y ago
This is part of why the Linux kernel needs a kernel debugger. I've worked with other kernels where we can actually observe what is going on using tools (DTrace on Illumos), and problems like this become something that can be tackled directly, without having to go through all this nonsense.
- CUViper 12y ago
  kgdb, kdb, systemtap, ktap, perf, ftrace, ...
  - stusmall 12y ago
    I've even seen a Visual Studio plugin to debug the Linux kernel. No joke. Really looked to just be a layer over kgdb but still.
    - CUViper 12y ago
      I suppose that's sysprogs.com/VisualKernel/ ? I'm amazed that this exists, but more power to them! :)
      - stusmall 12y ago
        That's the one. I'd be worried if there was more than one. Like you said, more power to them. Pretty cool it exists and honestly wouldn't mind trying it sometime.
amluto 12y ago
A lot of the time spent mucking around with QEMU can be avoided by using a tool like virtme: https://git.kernel.org/cgit/utils/kernel/virtme/virtme.git https://git.kernel.org/cgit/utils/kernel/virtme/virtme.git
josephlord 12y ago
Good article. It might be worth considering doing git bisect ignore if a build fails. I suspect (not sure it might just halt) at the moment it would result in a git bisect bad so might give you a wrong change commit.
- chriscool 12y ago
  You mean "git bisect skip". There is no "ignore" sub command.
  - josephlord 12y ago
    Yes, faulty memory.
snorkel 12y ago
Interesting way to use to git to find a bug by process of elimination.
mcardillo55 12y ago
So... technically a kernel bug was fixed and revealed a bug in Docker.
torrent-of-ions 12y ago
> there were close to 15,000 commits, which seemed like a large space to search by hand. Does the author not understand binary search? It would take about 14 tries to find the failed commit.
- laichzeit0 12y ago
  I have no idea why people are downvoting you. Sigh. A comment as to why you're wrong by the downvoter(s) would be infinitely more helpful.
  - erikb 12y ago
    Building 14 kernels and checking if they contain the bug manually sounds to be a lot of work for me. In that regard I think his post might not be a reasonable criticism. I think that criticism and being wrong is everybody's right, though. Therefore I wouldn't down vote a post for something like this but others seem to disagree.
  - philh 12y ago
    I downvoted for tone, not because ve's wrong. I'm reasonably confident that someone who can write this blog post, understands binary search. I wouldn't have downvoted a post that simply said, for example: > Binary search over 15,000 commits would take about 14 tries to find the failed one. although I don't think that's an important point to be making. (As erikb said, that's still a lot of commits, and that was only the first run.)
callesgg 12y ago
This was cool i will definitely try to use the bisect command sometime.

204 points by fauria 12y ago

Another article on bisecting which I bookmarked last year, but with physical hardware [1]. 1: http://rtg.in.ua/blog/kernel-bisect-pxe/ http://rtg.in.ua/blog/kernel-bisect-pxe/

CUViper 12y ago

One tip for the module configuration: kbuild has "make localyesconfig" which looks at your currently-loaded modules and converts them to CONFIG_FOO=Y builtins. See also "make help" for more available targets.

tlrobinson 12y ago

The first time I used "git bisect run" I felt like I was cheating. Unfortunately (or fortunately) I don't have many opportunities to use it.

chriscool 12y ago
There is this LWN article (that I wrote about it): http://lwn.net/Articles/317154/ http://lwn.net/Articles/317154/

tieTYT 12y ago

One thing that seems difficult with bisect are these series of steps: 1. "Hey... this bug didn't use to happen. Where'd that get introduced?" 2. OK, this unit test will tell me when the bug is fixed. 3. Now lets automate bisect to tell me where this test first failed even though I just wrote it. How do you do that? I think as long as you DON'T commit your unit test, bisect will carry over to every commit check. But, you'll have to make sure you commit all your other changes or else they'll be carried around, too: Conflict-city. Also, if you do commit your unit test (a "bad" habit of mine) I have no idea how I'm supposed to work with this. I end up copying the unit test by hand to each commit it tries to test. In summary, it seems like bisect was built with manual testing in mind. I know you can automate running a shell script, but I don't write my automated tests in shell script. EDIT: Keep in mind, not all projects are interpreted. Some are compiled.

ajross 12y ago
The whole idea of bisection is that you have an objective behavior that has changed. If the test (which is, by definition, the tool used to measure that objective behavior) is not constant across the range of changes, then you simply don't have a problem that can be solved by bisection. You're breaking rule one ("This bug" isn't a single behavior). Obviously in practice you'd probably just use whatever the most recent version of the test is. Really I think you're problem is that you're overspecifying rule 2. It's not about identifying a "unit test" per se, it's about identifying any specific process to detect the bug. If the tests in your tree work, great. If not, you may have to do a little development to make one that is automatable.
- taeric 12y ago
  I think you misunderstood. The test is brand new. As in, you find a problem that nobody has identified before and want to see how/when it was introduced. Easy way to do this is put your test external to the tree. Then, bisect as normal for git, and run your test. This is probably not uncommon for kernel tests, as you often have a test which could be "does this random laptop resume from sleep correctly?" Not really easy to automate that. And doesn't live in your tree.
  - Hello71 12y ago
    > "does this random laptop resume from sleep correctly?" Not really easy to automate that. that's what rtcwake is for
    - deleted 12y ago
      [deleted]
    - taeric 12y ago
      Right, my point was more that often everything will be working great for the main developer's laptop. I did not think it was uncommon to have someone come out of the weeds with a specific laptop that doesn't work. Or worse, it works if you sleep using the "sleep" button, but not if you close the lid. So, yes, you may be able to automate it later, once you fully understand the specifics that cause it to not rewake. Possibly specific to how it went to sleep with the lid close. Until then, it is nice to be able to do ad hoc tests when necessary.

Arnavion 12y ago

It's a pity that while the bisect did point in the general direction of where the bug was in 3.15, it didn't actually find the right bug (the one it found had already been fixed and a different one introduced that led to the same errant behavior). With something as complex as the kernel, I also wonder about automatic bisect being led astray by unrelated changes. The bisect script might fail for a different reason and zone in on the wrong commit.

chriscool 12y ago
The bisect pointed at the right patch series which is already very nice. If the bisect script fails because of another problem, it is often possible to refine the script, or run bisect again on a fixed up branch.
- Arnavion 12y ago
  If I'm not mistaken, it didn't find the right patch series. The article says the function changed in 3.15 and the underlying reason for his issue was different and not part of that series.
  - seabee 12y ago
    When you update the kernel and your program stops working, there are two possibilities: 1) it's a kernel bug; there is a regression in the kernel's documented behaviour 2) it's a bug in your tool because it took advantage of undefined or undocumented behaviour The bisect found exactly the right patch series, but bisect can only answer the question 'when did it stop working', not 'why'. You can read about many instances of 2) on Raymond Chen's blog The Old New Thing.

djhoffma 12y ago

This is part of why the Linux kernel needs a kernel debugger. I've worked with other kernels where we can actually observe what is going on using tools (DTrace on Illumos), and problems like this become something that can be tackled directly, without having to go through all this nonsense.

CUViper 12y ago
kgdb, kdb, systemtap, ktap, perf, ftrace, ...
- stusmall 12y ago
  I've even seen a Visual Studio plugin to debug the Linux kernel. No joke. Really looked to just be a layer over kgdb but still.
  - CUViper 12y ago
    I suppose that's sysprogs.com/VisualKernel/ ? I'm amazed that this exists, but more power to them! :)
    - stusmall 12y ago
      That's the one. I'd be worried if there was more than one. Like you said, more power to them. Pretty cool it exists and honestly wouldn't mind trying it sometime.

amluto 12y ago

A lot of the time spent mucking around with QEMU can be avoided by using a tool like virtme: https://git.kernel.org/cgit/utils/kernel/virtme/virtme.git https://git.kernel.org/cgit/utils/kernel/virtme/virtme.git

josephlord 12y ago

Good article. It might be worth considering doing git bisect ignore if a build fails. I suspect (not sure it might just halt) at the moment it would result in a git bisect bad so might give you a wrong change commit.

chriscool 12y ago
You mean "git bisect skip". There is no "ignore" sub command.
- josephlord 12y ago
  Yes, faulty memory.

snorkel 12y ago

Interesting way to use to git to find a bug by process of elimination.

mcardillo55 12y ago

So... technically a kernel bug was fixed and revealed a bug in Docker.

torrent-of-ions 12y ago

> there were close to 15,000 commits, which seemed like a large space to search by hand. Does the author not understand binary search? It would take about 14 tries to find the failed commit.

laichzeit0 12y ago
I have no idea why people are downvoting you. Sigh. A comment as to why you're wrong by the downvoter(s) would be infinitely more helpful.
- erikb 12y ago
  Building 14 kernels and checking if they contain the bug manually sounds to be a lot of work for me. In that regard I think his post might not be a reasonable criticism. I think that criticism and being wrong is everybody's right, though. Therefore I wouldn't down vote a post for something like this but others seem to disagree.
- philh 12y ago
  I downvoted for tone, not because ve's wrong. I'm reasonably confident that someone who can write this blog post, understands binary search. I wouldn't have downvoted a post that simply said, for example: > Binary search over 15,000 commits would take about 14 tries to find the failed one. although I don't think that's an important point to be making. (As erikb said, that's still a lot of commits, and that was only the first run.)

callesgg 12y ago

This was cool i will definitely try to use the bisect command sometime.