4 ms·

Automatically bisecting the Linux kernel to find a bug (2020)

52 points by luu 3y ago

csense 3y ago
Protip: You can compile the Linux kernel with User-mode Linux, ARCH=um to produce a kernel that runs as a regular binary on a host Linux OS. User-mode Linux is super helpful for diagnosing any pure software kernel bug like this. You don't have to muck with emulators, startup is very fast, and if you need GDB you can use it on the guest kernel like any other C program. User-mode Linux is a significant enough kernel feature to have its own Wikipedia page: https://en.wikipedia.org/wiki/User-mode_Linux https://en.wikipedia.org/wiki/User-mode_Linux
- jchw 3y ago
  If I remember correctly, Linode used to use UML for their VPS service. (And probably a lot of other hosts before OpenVZ and KVM became available/more popular.)
- interlinked 3y ago
  Why doesn't docker use it for linux guest on linux host instead of kvm?
  - csense 3y ago
    You seem to be...confused. Docker guest processes run on the host kernel, Docker just uses cgroups to request the host kernel isolate or instance various components (filesystems, UID/GID spaces, networking...maybe other things). These instancing / isolation capabilities and interfaces are very Linux-specific. So to run Docker on a non-Linux OS, you need to run Linux in a VM. If you have a Linux host, I'm not sure why you'd run Docker inside KVM or a user-mode Linux kernel. Wouldn't you rather just run Docker directly inside the host kernel?
    - regularfry 3y ago
      It's more interesting when you ask the question "why don't we use it to run docker containers on Darwin"... I dread to imagine what mapping cgroups back and forth between the two APIs might look like, but I also can't imagine I'm the first person to have wondered if it was, at least in principle, possible.
    - als0 3y ago
      > I'm not sure why you'd run Docker inside KVM or a user-mode Linux kernel. Wouldn't you rather just run Docker directly inside the host kernel? For better isolation. Privilege escalation would only affect the user mode kernel.
- Twirrim 3y ago
  Oh that's really interesting. Thanks for sharing! I might revisit this whole post & bug and try using User-mode, see if I can reproduce it that way.
xt00 3y ago
I’ve always found the git bisect “interface” tedious to use so I do it with a gedit window just writing down how many commits back I jump to etc. Probably the most annoying bisecting is when you have to apply patches to a source tree at each point you bisect as well.. but this guy putting building the Linux kernel running it and testing a thing inside his bisect loop automatically is super slick.
- _flux 3y ago
  I think the interface is actually quite nice! When I need to analyze bisect results myself in such a situation, I usually just do it like (cd src; patch < needed_patches && make && do_the_test; git reset --hard) git bisect good (or bad or skip it patches failed to apply) and repeat those from my command line history as needed. I think it works out fine. Git bisect can also dig inside merges, which I think would be a bit more annoying with just a list of commits.
bilekas 3y ago
This is actually really nice advice on the bisecting the commit log. I've done it more times than I would like to but would take advantage of the "git blame" when I had a fair idea of the effected area to narrow down results. This is great for automating that process. Super nice article
Twirrim 3y ago
Original post author here. Not sure what sparked this to be posted now. Happy to answer questions if folks have them.
- rc00 3y ago
  I would wager this post from the other day made someone think of your post. https://news.ycombinator.com/item?id=36325986 https://news.ycombinator.com/item?id=36325986 Both are phenomenally fantastic feats.

52 points by luu 3y ago

Protip: You can compile the Linux kernel with User-mode Linux, ARCH=um to produce a kernel that runs as a regular binary on a host Linux OS. User-mode Linux is super helpful for diagnosing any pure software kernel bug like this. You don't have to muck with emulators, startup is very fast, and if you need GDB you can use it on the guest kernel like any other C program. User-mode Linux is a significant enough kernel feature to have its own Wikipedia page: https://en.wikipedia.org/wiki/User-mode_Linux https://en.wikipedia.org/wiki/User-mode_Linux

jchw 3y ago
If I remember correctly, Linode used to use UML for their VPS service. (And probably a lot of other hosts before OpenVZ and KVM became available/more popular.)
interlinked 3y ago
Why doesn't docker use it for linux guest on linux host instead of kvm?
- csense 3y ago
  You seem to be...confused. Docker guest processes run on the host kernel, Docker just uses cgroups to request the host kernel isolate or instance various components (filesystems, UID/GID spaces, networking...maybe other things). These instancing / isolation capabilities and interfaces are very Linux-specific. So to run Docker on a non-Linux OS, you need to run Linux in a VM. If you have a Linux host, I'm not sure why you'd run Docker inside KVM or a user-mode Linux kernel. Wouldn't you rather just run Docker directly inside the host kernel?
  - regularfry 3y ago
    It's more interesting when you ask the question "why don't we use it to run docker containers on Darwin"... I dread to imagine what mapping cgroups back and forth between the two APIs might look like, but I also can't imagine I'm the first person to have wondered if it was, at least in principle, possible.
  - als0 3y ago
    > I'm not sure why you'd run Docker inside KVM or a user-mode Linux kernel. Wouldn't you rather just run Docker directly inside the host kernel? For better isolation. Privilege escalation would only affect the user mode kernel.
Twirrim 3y ago
Oh that's really interesting. Thanks for sharing! I might revisit this whole post & bug and try using User-mode, see if I can reproduce it that way.

xt00 3y ago

I’ve always found the git bisect “interface” tedious to use so I do it with a gedit window just writing down how many commits back I jump to etc. Probably the most annoying bisecting is when you have to apply patches to a source tree at each point you bisect as well.. but this guy putting building the Linux kernel running it and testing a thing inside his bisect loop automatically is super slick.

_flux 3y ago
I think the interface is actually quite nice! When I need to analyze bisect results myself in such a situation, I usually just do it like (cd src; patch < needed_patches && make && do_the_test; git reset --hard) git bisect good (or bad or skip it patches failed to apply) and repeat those from my command line history as needed. I think it works out fine. Git bisect can also dig inside merges, which I think would be a bit more annoying with just a list of commits.

bilekas 3y ago

This is actually really nice advice on the bisecting the commit log. I've done it more times than I would like to but would take advantage of the "git blame" when I had a fair idea of the effected area to narrow down results. This is great for automating that process. Super nice article

Twirrim 3y ago

Original post author here. Not sure what sparked this to be posted now. Happy to answer questions if folks have them.

rc00 3y ago
I would wager this post from the other day made someone think of your post. https://news.ycombinator.com/item?id=36325986 https://news.ycombinator.com/item?id=36325986 Both are phenomenally fantastic feats.