8 ms·

How is the Linux kernel tested?

159 points by sprado 7y ago

umvi 7y ago
So what happens when Linus Torvalds dies? Is there someone else who will become BDFL?
- LukeShu 7y ago
  Greg Kroah-Hartman has filled in before when Linus has had to take time off.
thedance 7y ago
Long way of saying "it's not".
- GhettoMaestro 7y ago
  Fuck you. I hate comments like this on HN. The article clearly elaborates which methods/tools are used in their test process.
- DSingularity 7y ago
  No, it is. https://github.com/linux-test-project/ltp https://github.com/linux-test-project/ltp
  - 0xFFC 7y ago
    This has not been mainlined as far as I know.
    - recov 7y ago
      The commit history looks pretty healthy
      - yjftsjthsd-h 7y ago
        "Mainlined", not "maintained".
  - bonzini 7y ago
    LTP is only a small part of testing Linux.
- userbinator 7y ago
  It is, by everyone who uses it every day.
alkonaut 7y ago
Why is there such little emphasis on “traditional” testing, that is, regular unit tests? At least some portion of the code base is surely suitable for normal unit tests. For example data structures, scheduling algorithms, file systems, ...
- EdSchouten 7y ago
  I think the main reason is that the Linux kernel (and similarly the *BSD kernels) are written in a programming language (C) that doesn't make it easy to do that. Code is often directly built on top of other kernel subsystems without any dependency injection whatsoever. This means that it's still possible to do unit testing of parts of the kernel, but it takes a crazy amount of effort, such as overriding symbols, overriding include paths and provide stub headers, etc.. I am well aware that it's also possible to have dependency injection in C by using structs with function pointers, but I think we can all agree that it's a lot less pleasant to use than C++ abstract base classes, Go interfaces or Rust traits. This is why the Linux kernel only tends to use this sparingly (e.g., inode operations).
  - deleted 7y ago
    [deleted]
  - rustybolt 7y ago
    This is probably the reason. I work at a place where most software is written in C, and I see the same thing here: literally all tests are either manual or integration tests. Unfortunately, this also means that it takes about about half a day to 'run the tests'.
  - joveian 7y ago
    This is a major advantage of NetBSD's Rump kernels that are used for automated testing. Some people have tried doing the same for Linux but I'm not sure if any such efforts are still in progress.
  - eschaton 7y ago
    You don’t need a dependency injection framework to write unit tests, you just need cleanly separable units with well defined interfaces.
ndesaulniers 7y ago
We run our own ci for building Linux kernels with clang. (ClangBuiltLinux.github.io). We take Debian's nightly package of ToT, make a docker image with the minimum tools we need, and use buildroot ramdisks that have a custom init script that powers down the machine once it reaches init. Our CI fetches various kernel trees and branches, builds them, then boots them in Qemu. The machine has 2 minutes to power up and down (usually takes less than 10s), otherwise we consider the machine hung and fail the run. We use travisci for the reporting, but are looking to offload the building. Also, Linaro's Tool chain Working Group runs a ton of CI on the kernel as well. There's an effort from RedHat called KCI to aggregate all of these reports.
- cat199 7y ago
  so you have a very well tested boot sequence - is this not a slightly better kernel equivalent of 'it builds, ship it' ?
  - yjftsjthsd-h 7y ago
    > a slightly better kernel equivalent of 'it builds, ship it' ? Surely, "it runs, ship it"? That seems quite a bit better.
    - blattimwind 7y ago
      The grand majority of code in the Linux kernel will never be hit by booting it in a QEMU with init=/bin/shutdown. The pure line coverage of these builds is probably like 5 or 10 %.
      - ndesaulniers 7y ago
        Note that our CI is doing way way more testing than most kernel developers do. It's not meant to exercise 100% of the kernel, just be a basic smoke test to see if we really messed something up. From triaging reports daily from kbuild test robot aka "0day" bot, I'd say 1/3 to 1/2 of kernel commits pushed by various developers to their trees have never even been compiled (very obvious mistakes regardless of toolchain). We get way more coverage in actually shipping these kernels in Android and ChromeOS.
dk8086 7y ago
Oh my! I though they were doing TDD :)
jimmyswimmy 7y ago
> Another major challenge is to automate device drivers and hardware platform tests because they require the hardware to be tested. Anyone have any thoughts on how to automate testing of coupled hardware-software systems? This is really hard; we've attacked it in the past by writing hardware simulators in accordance with the ICD. This falls flat once you find that the hardware doesn't precisely match the ICD, and usually it's much cheaper to change the software than the hardware. And at that point, the simulator hasn't actually helped you at all. I recently built a system which involved several tightly coupled hardware components and we fought many bugs on a tight schedule. It would have been nice to find a good way to think about this beyond the basic hardware-in-the-loop manual testing.
- rxhernandez 7y ago
  I can't tell you precisely how it was done but I worked at a company with many different types of hardware that contained many complex configurations of FPGAs, lasers, optics, and microcontrollers, coupled to a computer and they managed to simulate it quite well for what seems like a decade. One of the scientists there was one of the few geniuses I've ever met and they managed to simulate all those devices and a sufficient amount of their variations. So I can confirm it's possible, maybe it just requires an overworked genius?
  - yjftsjthsd-h 7y ago
    So I can confirm it's possible, maybe it just requires an overworked genius? I believe that, but unless you can find a genius and/or mass-produce their work, does it help the rest of us?
    - rxhernandez 7y ago
      Maybe. Knowing something is possible is sufficient for enough people to give things a try. I know I've been successful in doing so; I've never built a simulator of this magnitude but I've successfully solved difficult problems with novel solutions simply from hearing it was possible to solve them in a given manner.
- 7y ago
aorth 7y ago
There is automated static analysis from smatch for over ten years now: https://lwn.net/Articles/691882/ https://lwn.net/Articles/691882/ This has found thousands of bugs in the kernel.

159 points by sprado 7y ago

umvi 7y ago

So what happens when Linus Torvalds dies? Is there someone else who will become BDFL?

LukeShu 7y ago
Greg Kroah-Hartman has filled in before when Linus has had to take time off.

thedance 7y ago

Long way of saying "it's not".

GhettoMaestro 7y ago
Fuck you. I hate comments like this on HN. The article clearly elaborates which methods/tools are used in their test process.
DSingularity 7y ago
No, it is. https://github.com/linux-test-project/ltp https://github.com/linux-test-project/ltp
- 0xFFC 7y ago
  This has not been mainlined as far as I know.
  - recov 7y ago
    The commit history looks pretty healthy
    - yjftsjthsd-h 7y ago
      "Mainlined", not "maintained".
- bonzini 7y ago
  LTP is only a small part of testing Linux.
userbinator 7y ago
It is, by everyone who uses it every day.

alkonaut 7y ago

Why is there such little emphasis on “traditional” testing, that is, regular unit tests? At least some portion of the code base is surely suitable for normal unit tests. For example data structures, scheduling algorithms, file systems, ...

EdSchouten 7y ago
I think the main reason is that the Linux kernel (and similarly the *BSD kernels) are written in a programming language (C) that doesn't make it easy to do that. Code is often directly built on top of other kernel subsystems without any dependency injection whatsoever. This means that it's still possible to do unit testing of parts of the kernel, but it takes a crazy amount of effort, such as overriding symbols, overriding include paths and provide stub headers, etc.. I am well aware that it's also possible to have dependency injection in C by using structs with function pointers, but I think we can all agree that it's a lot less pleasant to use than C++ abstract base classes, Go interfaces or Rust traits. This is why the Linux kernel only tends to use this sparingly (e.g., inode operations).
- deleted 7y ago
  [deleted]
- rustybolt 7y ago
  This is probably the reason. I work at a place where most software is written in C, and I see the same thing here: literally all tests are either manual or integration tests. Unfortunately, this also means that it takes about about half a day to 'run the tests'.
- joveian 7y ago
  This is a major advantage of NetBSD's Rump kernels that are used for automated testing. Some people have tried doing the same for Linux but I'm not sure if any such efforts are still in progress.
- eschaton 7y ago
  You don’t need a dependency injection framework to write unit tests, you just need cleanly separable units with well defined interfaces.

ndesaulniers 7y ago

We run our own ci for building Linux kernels with clang. (ClangBuiltLinux.github.io). We take Debian's nightly package of ToT, make a docker image with the minimum tools we need, and use buildroot ramdisks that have a custom init script that powers down the machine once it reaches init. Our CI fetches various kernel trees and branches, builds them, then boots them in Qemu. The machine has 2 minutes to power up and down (usually takes less than 10s), otherwise we consider the machine hung and fail the run. We use travisci for the reporting, but are looking to offload the building. Also, Linaro's Tool chain Working Group runs a ton of CI on the kernel as well. There's an effort from RedHat called KCI to aggregate all of these reports.

cat199 7y ago
so you have a very well tested boot sequence - is this not a slightly better kernel equivalent of 'it builds, ship it' ?
- yjftsjthsd-h 7y ago
  > a slightly better kernel equivalent of 'it builds, ship it' ? Surely, "it runs, ship it"? That seems quite a bit better.
  - blattimwind 7y ago
    The grand majority of code in the Linux kernel will never be hit by booting it in a QEMU with init=/bin/shutdown. The pure line coverage of these builds is probably like 5 or 10 %.
    - ndesaulniers 7y ago
      Note that our CI is doing way way more testing than most kernel developers do. It's not meant to exercise 100% of the kernel, just be a basic smoke test to see if we really messed something up. From triaging reports daily from kbuild test robot aka "0day" bot, I'd say 1/3 to 1/2 of kernel commits pushed by various developers to their trees have never even been compiled (very obvious mistakes regardless of toolchain). We get way more coverage in actually shipping these kernels in Android and ChromeOS.

dk8086 7y ago

Oh my! I though they were doing TDD :)

jimmyswimmy 7y ago

> Another major challenge is to automate device drivers and hardware platform tests because they require the hardware to be tested. Anyone have any thoughts on how to automate testing of coupled hardware-software systems? This is really hard; we've attacked it in the past by writing hardware simulators in accordance with the ICD. This falls flat once you find that the hardware doesn't precisely match the ICD, and usually it's much cheaper to change the software than the hardware. And at that point, the simulator hasn't actually helped you at all. I recently built a system which involved several tightly coupled hardware components and we fought many bugs on a tight schedule. It would have been nice to find a good way to think about this beyond the basic hardware-in-the-loop manual testing.

rxhernandez 7y ago
I can't tell you precisely how it was done but I worked at a company with many different types of hardware that contained many complex configurations of FPGAs, lasers, optics, and microcontrollers, coupled to a computer and they managed to simulate it quite well for what seems like a decade. One of the scientists there was one of the few geniuses I've ever met and they managed to simulate all those devices and a sufficient amount of their variations. So I can confirm it's possible, maybe it just requires an overworked genius?
- yjftsjthsd-h 7y ago
  So I can confirm it's possible, maybe it just requires an overworked genius? I believe that, but unless you can find a genius and/or mass-produce their work, does it help the rest of us?
  - rxhernandez 7y ago
    Maybe. Knowing something is possible is sufficient for enough people to give things a try. I know I've been successful in doing so; I've never built a simulator of this magnitude but I've successfully solved difficult problems with novel solutions simply from hearing it was possible to solve them in a given manner.
7y ago

aorth 7y ago

There is automated static analysis from smatch for over ten years now: https://lwn.net/Articles/691882/ https://lwn.net/Articles/691882/ This has found thousands of bugs in the kernel.