3 ms·

There are tests: https://www.kernel.org/doc/html/latest/dev-tools/testing-overview.html https://www.kernel.org/doc/html/latest/dev-tools/testing-ove... However

There are tests: https://www.kernel.org/doc/html/latest/dev-tools/testing-overview.html https://www.kernel.org/doc/html/latest/dev-tools/testing-ove... However, the kernel is fairly hard to test because it's the kernel: - A lot of code paths are fully dependent on hardware that can't be mocked. How would you test the bootloader, for example? Or code that depends on a certain processor architecture? The only way is to actually run the code on that hardware. - Similarly, it's hard to test the drivers. A lot of them are made by vendors, and the only way to test is against the devices themselves. Even if you could emulate the device and connect that to the driver, there's a lot of things that you won't cover (the communication path, the device misbehaving...) - It's hard to test things that the runtime depends on. For example, if you try and test the memory subsystem, how do you manage the memory to actually run the tests? Kernel testing is hard for the same reasons kernel development alone is hard, you're building the thing that all other tools require to run.

valenterry 4y ago
I don't really agree with that. Yeah, sure it is hard or even harder. But think about browsers. We run applications and we want to make sure they look and behave the same, but there are so many OS, browsers, versions, devices, ... So do we have to buy a laptop for each? Maybe in the past, but today we "just" emulate it. The same can be done with kernel testing. Assume that the whole kernel runs emulated (which is the hard part) and then test the outout of that (in the emulator).
- gjulianm 4y ago
  Browser and kernel development couldn't be more different. > Maybe in the past, but today we "just" emulate it. "Just emulating" is almost impossible. For testing to be actually reliable, you'd need to recreate all the hardware behavior in code. OS virtualization relies on re-exposing the hardware and kernel primitives to VMs, no VM is reimplementing the Intel architecture for memory management, for example. > then test the outout of that (in the emulator). Another hard topic. What's the output of a kernel? In an emulator the only thing you have is access to the memory areas (RAM, screen, devices). You can't really rebuild the state of the kernel, or access internal structures that tell you information about how it is. If you're lucky and advanced enough in the boot sequence that the kernel has configured the serial port for comms and debugging, you might be able to get some information, but not that much, and it'd still be impossible to test certain parts of the boot sequence.
  - valenterry 4y ago
    > OS virtualization relies on re-exposing the hardware and kernel primitives to VMs, no VM is reimplementing the Intel architecture for memory management, for example. That's fair. But in the same way that the VM relies on that the host OS handles operations correctly, the kernel can assume the same, as long as there is a clean separation (so that one kernel doesn't influence the other). Of course, if you have a bug in the (old/other) kernel that influences the tests themselves then that wouldn't work, but I would say that this probably makes up a very small minority of cases. Just like if Intel runs tests for their CPUs on other CPUs and those other CPUs are buggy... > Another hard topic. What's the output of a kernel The kernel has an interface and if I ask it to write something and then read it back, I have an expectation about what happens. I don't need to care about the state of the kernel (neither should I) as long as the result is good. Only for certain performance-relevant things I can see that this might be a problem. > and it'd still be impossible to test certain parts of the boot sequence I'm neither a kernel nor a browser developer ;) but I would be interested in an example. You are probably right, but I fail to see what that would be.
- xorcist 4y ago
  It's a lot more work to write a perfect bit-for-bit emulator for some hardware than to write a kernel driver for said hardware. You could absolutely run tests on emulated hardware, but you probably learn more about the emulator than about the kernel.
  - valenterry 4y ago
    Yeah I think that's true. From my experience it's the same for testing software in general - you can mock things but having actual tests that test the behaviour of dependencies is tricky since you have to emulate those for your tests and that can be a lot of work. But since it covers your for all future changes, it might still be worth it, because otherwise you either have to manually retest with actual hardware or accept a higher risk of bugs.
bena 4y ago
Not to mention, if the emulation isn't 100% accurate, you could still run into something odd. Especially if you only emulate what's in the spec and not the actual hardware.
rpastuszak 4y ago
Interesting, I quickly checked the source before posting that comment and didn't find any, but I was expecting them to be co-located for some reason. Thanks for sharing. > - A lot of code paths are fully dependent on hardware that can't be mocked. How would you test the bootloader, for example? Or code that depends on a certain processor architecture? The only way is to actually run the code on that hardware. My incorrect assumption was that one could run those tests in some sort of a virtualised environment where the host VM provides the tested component with the correct virtual hardware. The "hardware" itself contains mocks the test can access to run assertions. In other words, testing would involve checking the side effects. Come to think of it, that doesn't sound very practical. In fact, it sounds almost impossible to scale.
- gjulianm 4y ago
  I know some developers test with VMs with emulated architectures, but it almost requires manual testing and seeing how things work. As you say, it's fairly impractical and impossible to scale, you can't really get too much useful information out of VM memory and registers.
- blueflow 4y ago
  Virtual Machines are a really poor rip-off of the original thing. Users rarely notice tho because the kernel hides that away. Like GP said, real hardware is difficult to mock good enough to be useful for testing. If you try it in practice, your emulator will only be one bullet point in the list of platforms to support... I had to attempt to write an interface to the floppy and serial controller for IBM PC compatibles to learn this.
  - AlotOfReading 4y ago
    The direction people are moving towards is re-hosting and directed emulation, where you run the code in an environment where the analysis tools are better and emulate just enough of the platform that the code thinks it's executing natively. Some of the published work even looks at the expectations encoded in the source to emulate what it's expecting from the "hardware". Since the HW is fake, testing new configurations is just a config change away. Take a look at [1], [2], [3] for published work. I also have a tool that's used in a couple companies able to rehost simpler linux device drivers and embedded firmware without source code modifications. Everything's encoded as DT files, so you can just use the configs already in the kernel tree and even do fun things like simulating physical hardware in a game engine. [1] https://www.usenix.org/conference/raid2019/presentation/gustafson https://www.usenix.org/conference/raid2019/presentation/gust... [2] https://doi.org/10.48550/arXiv.2107.12867 https://doi.org/10.48550/arXiv.2107.12867 [3] https://www.usenix.org/conference/usenixsecurity20/presentation/clements https://www.usenix.org/conference/usenixsecurity20/presentat...
q7xvh97o2pDhNrh 4y ago
Naive question: Couldn't you manually test against the hardware/devices, confirm the code is passing, identify the salient behavior that is hardware-specific and needs to remain invariant under refactoring, and write unit tests for just that? I realize that's not perfect -- not even close -- and I'm sure smarter people than me have considered that idea and rejected it. But, out of curiosity, I wonder if you know the reason why it wouldn't help at least add some marginal value?
- gjulianm 4y ago
  > identify the salient behavior that is hardware-specific and needs to remain invariant under refactoring, and write unit tests for just that? I don't think you really can. There will be things that can be different between runs (e.g., a DMA transfer) and not necessarily mean bugs. I don't think there are many fully stateless drivers where that kind of effort would both cover a significant part of hardware behavior and yield good enough code coverage.
  - nyanpasu64 4y ago
    Can you record the register writes, DMA transfers of a successful run, then run the kernel in an emulated environment and ensure it makes the same writes, and presents the same results to userspace given the same data returned back? This is probably a lot more practical for a mouse than a high-bandwidth webcam or display where the "expected" logs can run into the megabytes or gigabytes for a few seconds of recording.
lupire 4y ago
Well funded orgs can automatically deploy the test code to real hardware, and run tests on it. They use test server farms.

valenterry 4y ago

I don't really agree with that. Yeah, sure it is hard or even harder. But think about browsers. We run applications and we want to make sure they look and behave the same, but there are so many OS, browsers, versions, devices, ... So do we have to buy a laptop for each? Maybe in the past, but today we "just" emulate it. The same can be done with kernel testing. Assume that the whole kernel runs emulated (which is the hard part) and then test the outout of that (in the emulator).

gjulianm 4y ago
Browser and kernel development couldn't be more different. > Maybe in the past, but today we "just" emulate it. "Just emulating" is almost impossible. For testing to be actually reliable, you'd need to recreate all the hardware behavior in code. OS virtualization relies on re-exposing the hardware and kernel primitives to VMs, no VM is reimplementing the Intel architecture for memory management, for example. > then test the outout of that (in the emulator). Another hard topic. What's the output of a kernel? In an emulator the only thing you have is access to the memory areas (RAM, screen, devices). You can't really rebuild the state of the kernel, or access internal structures that tell you information about how it is. If you're lucky and advanced enough in the boot sequence that the kernel has configured the serial port for comms and debugging, you might be able to get some information, but not that much, and it'd still be impossible to test certain parts of the boot sequence.
- valenterry 4y ago
  > OS virtualization relies on re-exposing the hardware and kernel primitives to VMs, no VM is reimplementing the Intel architecture for memory management, for example. That's fair. But in the same way that the VM relies on that the host OS handles operations correctly, the kernel can assume the same, as long as there is a clean separation (so that one kernel doesn't influence the other). Of course, if you have a bug in the (old/other) kernel that influences the tests themselves then that wouldn't work, but I would say that this probably makes up a very small minority of cases. Just like if Intel runs tests for their CPUs on other CPUs and those other CPUs are buggy... > Another hard topic. What's the output of a kernel The kernel has an interface and if I ask it to write something and then read it back, I have an expectation about what happens. I don't need to care about the state of the kernel (neither should I) as long as the result is good. Only for certain performance-relevant things I can see that this might be a problem. > and it'd still be impossible to test certain parts of the boot sequence I'm neither a kernel nor a browser developer ;) but I would be interested in an example. You are probably right, but I fail to see what that would be.
xorcist 4y ago
It's a lot more work to write a perfect bit-for-bit emulator for some hardware than to write a kernel driver for said hardware. You could absolutely run tests on emulated hardware, but you probably learn more about the emulator than about the kernel.
- valenterry 4y ago
  Yeah I think that's true. From my experience it's the same for testing software in general - you can mock things but having actual tests that test the behaviour of dependencies is tricky since you have to emulate those for your tests and that can be a lot of work. But since it covers your for all future changes, it might still be worth it, because otherwise you either have to manually retest with actual hardware or accept a higher risk of bugs.

bena 4y ago

Not to mention, if the emulation isn't 100% accurate, you could still run into something odd. Especially if you only emulate what's in the spec and not the actual hardware.

rpastuszak 4y ago

Interesting, I quickly checked the source before posting that comment and didn't find any, but I was expecting them to be co-located for some reason. Thanks for sharing. > - A lot of code paths are fully dependent on hardware that can't be mocked. How would you test the bootloader, for example? Or code that depends on a certain processor architecture? The only way is to actually run the code on that hardware. My incorrect assumption was that one could run those tests in some sort of a virtualised environment where the host VM provides the tested component with the correct virtual hardware. The "hardware" itself contains mocks the test can access to run assertions. In other words, testing would involve checking the side effects. Come to think of it, that doesn't sound very practical. In fact, it sounds almost impossible to scale.

gjulianm 4y ago
I know some developers test with VMs with emulated architectures, but it almost requires manual testing and seeing how things work. As you say, it's fairly impractical and impossible to scale, you can't really get too much useful information out of VM memory and registers.
blueflow 4y ago
Virtual Machines are a really poor rip-off of the original thing. Users rarely notice tho because the kernel hides that away. Like GP said, real hardware is difficult to mock good enough to be useful for testing. If you try it in practice, your emulator will only be one bullet point in the list of platforms to support... I had to attempt to write an interface to the floppy and serial controller for IBM PC compatibles to learn this.
- AlotOfReading 4y ago
  The direction people are moving towards is re-hosting and directed emulation, where you run the code in an environment where the analysis tools are better and emulate just enough of the platform that the code thinks it's executing natively. Some of the published work even looks at the expectations encoded in the source to emulate what it's expecting from the "hardware". Since the HW is fake, testing new configurations is just a config change away. Take a look at [1], [2], [3] for published work. I also have a tool that's used in a couple companies able to rehost simpler linux device drivers and embedded firmware without source code modifications. Everything's encoded as DT files, so you can just use the configs already in the kernel tree and even do fun things like simulating physical hardware in a game engine. [1] https://www.usenix.org/conference/raid2019/presentation/gustafson https://www.usenix.org/conference/raid2019/presentation/gust... [2] https://doi.org/10.48550/arXiv.2107.12867 https://doi.org/10.48550/arXiv.2107.12867 [3] https://www.usenix.org/conference/usenixsecurity20/presentation/clements https://www.usenix.org/conference/usenixsecurity20/presentat...

q7xvh97o2pDhNrh 4y ago

Naive question: Couldn't you manually test against the hardware/devices, confirm the code is passing, identify the salient behavior that is hardware-specific and needs to remain invariant under refactoring, and write unit tests for just that? I realize that's not perfect -- not even close -- and I'm sure smarter people than me have considered that idea and rejected it. But, out of curiosity, I wonder if you know the reason why it wouldn't help at least add some marginal value?

gjulianm 4y ago
> identify the salient behavior that is hardware-specific and needs to remain invariant under refactoring, and write unit tests for just that? I don't think you really can. There will be things that can be different between runs (e.g., a DMA transfer) and not necessarily mean bugs. I don't think there are many fully stateless drivers where that kind of effort would both cover a significant part of hardware behavior and yield good enough code coverage.
- nyanpasu64 4y ago
  Can you record the register writes, DMA transfers of a successful run, then run the kernel in an emulated environment and ensure it makes the same writes, and presents the same results to userspace given the same data returned back? This is probably a lot more practical for a mouse than a high-bandwidth webcam or display where the "expected" logs can run into the megabytes or gigabytes for a few seconds of recording.

lupire 4y ago

Well funded orgs can automatically deploy the test code to real hardware, and run tests on it. They use test server farms.