3 ms·

> There are no automatic tests #til and also #wtf In all seriousness, can you recommend any resources on why is that the case and how kernel is tested?

bruce343434 4y ago
I can't speak about the kernel but personally I prefer proofs of correctness over testing.
- zasdffaa 4y ago
  If you haven't any experience of doing proofs of correctness, it may be worth getting some before being certain about what you prefer. Proofs are hard. Proofs of something like the kernel... dream on.
- zelphirkalt 4y ago
  Those are very hard to achieve though for some non-trivial C code.
gjulianm 4y ago
There are tests: https://www.kernel.org/doc/html/latest/dev-tools/testing-overview.html https://www.kernel.org/doc/html/latest/dev-tools/testing-ove... However, the kernel is fairly hard to test because it's the kernel: - A lot of code paths are fully dependent on hardware that can't be mocked. How would you test the bootloader, for example? Or code that depends on a certain processor architecture? The only way is to actually run the code on that hardware. - Similarly, it's hard to test the drivers. A lot of them are made by vendors, and the only way to test is against the devices themselves. Even if you could emulate the device and connect that to the driver, there's a lot of things that you won't cover (the communication path, the device misbehaving...) - It's hard to test things that the runtime depends on. For example, if you try and test the memory subsystem, how do you manage the memory to actually run the tests? Kernel testing is hard for the same reasons kernel development alone is hard, you're building the thing that all other tools require to run.
- valenterry 4y ago
  I don't really agree with that. Yeah, sure it is hard or even harder. But think about browsers. We run applications and we want to make sure they look and behave the same, but there are so many OS, browsers, versions, devices, ... So do we have to buy a laptop for each? Maybe in the past, but today we "just" emulate it. The same can be done with kernel testing. Assume that the whole kernel runs emulated (which is the hard part) and then test the outout of that (in the emulator).
  - gjulianm 4y ago
    Browser and kernel development couldn't be more different. > Maybe in the past, but today we "just" emulate it. "Just emulating" is almost impossible. For testing to be actually reliable, you'd need to recreate all the hardware behavior in code. OS virtualization relies on re-exposing the hardware and kernel primitives to VMs, no VM is reimplementing the Intel architecture for memory management, for example. > then test the outout of that (in the emulator). Another hard topic. What's the output of a kernel? In an emulator the only thing you have is access to the memory areas (RAM, screen, devices). You can't really rebuild the state of the kernel, or access internal structures that tell you information about how it is. If you're lucky and advanced enough in the boot sequence that the kernel has configured the serial port for comms and debugging, you might be able to get some information, but not that much, and it'd still be impossible to test certain parts of the boot sequence.
    - valenterry 4y ago
      > OS virtualization relies on re-exposing the hardware and kernel primitives to VMs, no VM is reimplementing the Intel architecture for memory management, for example. That's fair. But in the same way that the VM relies on that the host OS handles operations correctly, the kernel can assume the same, as long as there is a clean separation (so that one kernel doesn't influence the other). Of course, if you have a bug in the (old/other) kernel that influences the tests themselves then that wouldn't work, but I would say that this probably makes up a very small minority of cases. Just like if Intel runs tests for their CPUs on other CPUs and those other CPUs are buggy... > Another hard topic. What's the output of a kernel The kernel has an interface and if I ask it to write something and then read it back, I have an expectation about what happens. I don't need to care about the state of the kernel (neither should I) as long as the result is good. Only for certain performance-relevant things I can see that this might be a problem. > and it'd still be impossible to test certain parts of the boot sequence I'm neither a kernel nor a browser developer ;) but I would be interested in an example. You are probably right, but I fail to see what that would be.
badpun 4y ago
Last I checked, the testing strategy was „all maintainers’ computers should be on the nightly build and report bugs/crashes they experience”. I wonder how Microsoft does this with Windows? A friend of mine worked on Intel GPU drivers and they had entire farms of PCs where the tested driver would be installed on top of fresh Windows installation. For kernels, you’d probably need something like that, but times a thousand…

bruce343434 4y ago

I can't speak about the kernel but personally I prefer proofs of correctness over testing.

zasdffaa 4y ago
If you haven't any experience of doing proofs of correctness, it may be worth getting some before being certain about what you prefer. Proofs are hard. Proofs of something like the kernel... dream on.
zelphirkalt 4y ago
Those are very hard to achieve though for some non-trivial C code.

gjulianm 4y ago

There are tests: https://www.kernel.org/doc/html/latest/dev-tools/testing-overview.html https://www.kernel.org/doc/html/latest/dev-tools/testing-ove... However, the kernel is fairly hard to test because it's the kernel: - A lot of code paths are fully dependent on hardware that can't be mocked. How would you test the bootloader, for example? Or code that depends on a certain processor architecture? The only way is to actually run the code on that hardware. - Similarly, it's hard to test the drivers. A lot of them are made by vendors, and the only way to test is against the devices themselves. Even if you could emulate the device and connect that to the driver, there's a lot of things that you won't cover (the communication path, the device misbehaving...) - It's hard to test things that the runtime depends on. For example, if you try and test the memory subsystem, how do you manage the memory to actually run the tests? Kernel testing is hard for the same reasons kernel development alone is hard, you're building the thing that all other tools require to run.

valenterry 4y ago
I don't really agree with that. Yeah, sure it is hard or even harder. But think about browsers. We run applications and we want to make sure they look and behave the same, but there are so many OS, browsers, versions, devices, ... So do we have to buy a laptop for each? Maybe in the past, but today we "just" emulate it. The same can be done with kernel testing. Assume that the whole kernel runs emulated (which is the hard part) and then test the outout of that (in the emulator).
- gjulianm 4y ago
  Browser and kernel development couldn't be more different. > Maybe in the past, but today we "just" emulate it. "Just emulating" is almost impossible. For testing to be actually reliable, you'd need to recreate all the hardware behavior in code. OS virtualization relies on re-exposing the hardware and kernel primitives to VMs, no VM is reimplementing the Intel architecture for memory management, for example. > then test the outout of that (in the emulator). Another hard topic. What's the output of a kernel? In an emulator the only thing you have is access to the memory areas (RAM, screen, devices). You can't really rebuild the state of the kernel, or access internal structures that tell you information about how it is. If you're lucky and advanced enough in the boot sequence that the kernel has configured the serial port for comms and debugging, you might be able to get some information, but not that much, and it'd still be impossible to test certain parts of the boot sequence.
  - valenterry 4y ago
    > OS virtualization relies on re-exposing the hardware and kernel primitives to VMs, no VM is reimplementing the Intel architecture for memory management, for example. That's fair. But in the same way that the VM relies on that the host OS handles operations correctly, the kernel can assume the same, as long as there is a clean separation (so that one kernel doesn't influence the other). Of course, if you have a bug in the (old/other) kernel that influences the tests themselves then that wouldn't work, but I would say that this probably makes up a very small minority of cases. Just like if Intel runs tests for their CPUs on other CPUs and those other CPUs are buggy... > Another hard topic. What's the output of a kernel The kernel has an interface and if I ask it to write something and then read it back, I have an expectation about what happens. I don't need to care about the state of the kernel (neither should I) as long as the result is good. Only for certain performance-relevant things I can see that this might be a problem. > and it'd still be impossible to test certain parts of the boot sequence I'm neither a kernel nor a browser developer ;) but I would be interested in an example. You are probably right, but I fail to see what that would be.

badpun 4y ago

Last I checked, the testing strategy was „all maintainers’ computers should be on the nightly build and report bugs/crashes they experience”. I wonder how Microsoft does this with Windows? A friend of mine worked on Intel GPU drivers and they had entire farms of PCs where the tested driver would be installed on top of fresh Windows installation. For kernels, you’d probably need something like that, but times a thousand…