4 ms·

As huge parts of the kernel are effectively device driver code which interacts with hardware on a low level, classic unit testing techniques like mocking can't

As huge parts of the kernel are effectively device driver code which interacts with hardware on a low level, classic unit testing techniques like mocking can't be applied - or they could be, by writing a mock for the device in question, but that's usually prohibitively expensive if you aim for a fidelity level at which such a mock would actually be of any worth and wouldn't just provide you the "good feeling" of having some unit test coverage while not really increasing the confidence in the tested code. This is especially true when we're talking about reverse-engineered device communication protocols, where you don't have a clue how the device actually works internally, and thus lack the basic means of constructing a mock that does more than just implement the exact same assumptions you've already based your driver code on (resulting in always-green tests that never find actual bugs). But also in cases where you have a protocol spec, the vast majority of ugly bugs in drivers usually originate from differences between that spec and the behavior of the device in the real world.

rbanffy 4y ago
> or they could be, by writing a mock for the device in question That would create a good description of the hardware that could be used for emulators and to test whether actual hardware behaves the way it was specified to behave. > the vast majority of ugly bugs in drivers usually originate from differences between that spec and the behavior of the device in the real world. Having the model hardware encoded in software would make it possible to test its behavior against real hardware for comparison and allow making adjustments.
- ilyt 4y ago
  That takes massive amount of effort (most likely writing the emulator is way more than the actual driver code) and still only allows you to say "given those assumptions, that might be not correct, the code works". > Having the model hardware encoded in software would make it possible to test its behavior against real hardware for comparison and allow making adjustments. So you waste weeks to set up a hardware test jig, trying to match it with software and next firmware update changes some stuff that makes model incorrect again. I mean I would expect that if you're a company that wants to put a driver for your hardware into kernel, but many drivers are just RE stuff and setting up a test jig for hardware that allows for fully automated testing can be quite an effort
closeparen 4y ago
> if you aim for a fidelity level at which such a mock would actually be of any worth and wouldn't just provide you the "good feeling" of having some unit test coverage while not really increasing the confidence in the tested code When did you acquire such taste for luxuries? I’m convinced that fully 75% of engineering effort in Silicon Valley is exactly this kind of testing. At least it is in my company, and we get people from all the big names, and none of them think it’s weird. Sometimes I wonder if I’m insane.

rbanffy 4y ago

> or they could be, by writing a mock for the device in question That would create a good description of the hardware that could be used for emulators and to test whether actual hardware behaves the way it was specified to behave. > the vast majority of ugly bugs in drivers usually originate from differences between that spec and the behavior of the device in the real world. Having the model hardware encoded in software would make it possible to test its behavior against real hardware for comparison and allow making adjustments.

ilyt 4y ago
That takes massive amount of effort (most likely writing the emulator is way more than the actual driver code) and still only allows you to say "given those assumptions, that might be not correct, the code works". > Having the model hardware encoded in software would make it possible to test its behavior against real hardware for comparison and allow making adjustments. So you waste weeks to set up a hardware test jig, trying to match it with software and next firmware update changes some stuff that makes model incorrect again. I mean I would expect that if you're a company that wants to put a driver for your hardware into kernel, but many drivers are just RE stuff and setting up a test jig for hardware that allows for fully automated testing can be quite an effort

closeparen 4y ago

> if you aim for a fidelity level at which such a mock would actually be of any worth and wouldn't just provide you the "good feeling" of having some unit test coverage while not really increasing the confidence in the tested code When did you acquire such taste for luxuries? I’m convinced that fully 75% of engineering effort in Silicon Valley is exactly this kind of testing. At least it is in my company, and we get people from all the big names, and none of them think it’s weird. Sometimes I wonder if I’m insane.