3 ms·

As a HW guy this seems insane to me. Just because it's hard to do doesnt mean you can't randomise and try to create and catch failures. Might not catch 100%, bu

As a HW guy this seems insane to me. Just because it's hard to do doesnt mean you can't randomise and try to create and catch failures. Might not catch 100%, but better than not trying. HW testing pre silicon is impressively comprehensive as the cost of respinning a chip is in the order of 10s of millions, so the industry found a way. While software is less costly to edit, the cost of updating and rolling out is essentially huge, but uncounted. Finding bugs by extensive testing, even for difficult to catch items, is inherently the right thing to do

lgeorget 4y ago
They do apply a lot of fuzzing tests actually and the kernel maintainers also contribute state-of-the-art dynamic analysis tools like UBSAN to check for undefined behaviors or leaks.
ilyt 4y ago
As HW guy you are making the hardware so you can make a mock of it relatively easily, that is not the case if driver is essentially reverse-engineered out of how windows driver worked. And many other drivers are "some vendor made it, and they won't tell us how the chip actually work, just give us driver code". Then as other people mentioned hardware isn't exactly 100% predictable, HW bugs happen, firmware upgrades change how stuff works etc. so your test suite might be entirely unable to catch that.
- tails4e 4y ago
  I know hardware isn't 100% precitable, which is why we expliclty try to test that behaviour through randomized testing, formal proofs, etc. I'm not mocking anything, I'm just saying it's possible to attempt to cover difficult to create scenarios.
  - asalahli 4y ago
    I believe the parent was referring to Test mocks[0], not implying you mocking anything. 0. https://en.wikipedia.org/wiki/Mock_object#Use_in_test-driven_development https://en.wikipedia.org/wiki/Mock_object#Use_in_test-driven...
    - tails4e 4y ago
      I see, thanks. Well we do have events in HW that are non determistic, i.e. metastability due to async signal synchrinzation, but that is something we can and do model as best we can
pclmulqdq 4y ago
As a combined hardware/software person, the testing for both types of systems is very different. In hardware, you often have rigid boundaries with well-defined behavior, and almost no state in each unit. A big part of hardware verification is just testing that you did the basics of the interface right and that you won't deadlock anything. The rest of it is testing the behavior of your nearly-stateless thing. Software testing tends to involve exponentially larger state machines, and fuzzier abstraction boundaries. The basics are mostly done for you, and you tend to do complicated logic that holds and manipulates a lot of state. "Fuzzing" in software is very similar to constrained random testing in the hardware world, but it still doesn't catch a lot of basic issues due to the size of the state machines involved. The testing philosophy you need to apply is not the same at all.
- tails4e 4y ago
  It's interesting you have the perspective that hardware has realtiveley few states. It can be the case, but there are very complex interactions between functions also. A million gate design is realatively modest by modern standards, and there can but tens to hundreds of interacting state machines in that. Of course to mange this we do try define clean boundaries and interfaces between blocks, but this clean approach can be taken in SW also. Fuzzing is a bit different in that actually tries to execute different code paths by changing the input, so it's more of an exploration of the code state space. Constrained random is more trying to create inputs that randomly cover the valid input space. It does not introspect the internal state to determine the next input. Perhaps thats an enhancement hardware testing could take from the SW world.
  - pclmulqdq 4y ago
    I have seen functional-coverage-driven constrained random verification of hardware, which does what you describe. Someone beat you to it!
TheDesolate0 4y ago
anaisbetts 4y ago
I think you have conflated "tests aren't useful" with "*unit* tests aren't useful". I am asserting the latter, not the former! Testing in general is absolutely something that you can and should do when working on a production kernel

lgeorget 4y ago

They do apply a lot of fuzzing tests actually and the kernel maintainers also contribute state-of-the-art dynamic analysis tools like UBSAN to check for undefined behaviors or leaks.

ilyt 4y ago

As HW guy you are making the hardware so you can make a mock of it relatively easily, that is not the case if driver is essentially reverse-engineered out of how windows driver worked. And many other drivers are "some vendor made it, and they won't tell us how the chip actually work, just give us driver code". Then as other people mentioned hardware isn't exactly 100% predictable, HW bugs happen, firmware upgrades change how stuff works etc. so your test suite might be entirely unable to catch that.

tails4e 4y ago
I know hardware isn't 100% precitable, which is why we expliclty try to test that behaviour through randomized testing, formal proofs, etc. I'm not mocking anything, I'm just saying it's possible to attempt to cover difficult to create scenarios.
- asalahli 4y ago
  I believe the parent was referring to Test mocks[0], not implying you mocking anything. 0. https://en.wikipedia.org/wiki/Mock_object#Use_in_test-driven_development https://en.wikipedia.org/wiki/Mock_object#Use_in_test-driven...
  - tails4e 4y ago
    I see, thanks. Well we do have events in HW that are non determistic, i.e. metastability due to async signal synchrinzation, but that is something we can and do model as best we can

pclmulqdq 4y ago

As a combined hardware/software person, the testing for both types of systems is very different. In hardware, you often have rigid boundaries with well-defined behavior, and almost no state in each unit. A big part of hardware verification is just testing that you did the basics of the interface right and that you won't deadlock anything. The rest of it is testing the behavior of your nearly-stateless thing. Software testing tends to involve exponentially larger state machines, and fuzzier abstraction boundaries. The basics are mostly done for you, and you tend to do complicated logic that holds and manipulates a lot of state. "Fuzzing" in software is very similar to constrained random testing in the hardware world, but it still doesn't catch a lot of basic issues due to the size of the state machines involved. The testing philosophy you need to apply is not the same at all.

tails4e 4y ago
It's interesting you have the perspective that hardware has realtiveley few states. It can be the case, but there are very complex interactions between functions also. A million gate design is realatively modest by modern standards, and there can but tens to hundreds of interacting state machines in that. Of course to mange this we do try define clean boundaries and interfaces between blocks, but this clean approach can be taken in SW also. Fuzzing is a bit different in that actually tries to execute different code paths by changing the input, so it's more of an exploration of the code state space. Constrained random is more trying to create inputs that randomly cover the valid input space. It does not introspect the internal state to determine the next input. Perhaps thats an enhancement hardware testing could take from the SW world.
- pclmulqdq 4y ago
  I have seen functional-coverage-driven constrained random verification of hardware, which does what you describe. Someone beat you to it!

TheDesolate0 4y ago

anaisbetts 4y ago

I think you have conflated "tests aren't useful" with "*unit* tests aren't useful". I am asserting the latter, not the former! Testing in general is absolutely something that you can and should do when working on a production kernel