5 ms·

Unit tests are poorly suited for kernel code. The vast majority of production failures in kernel code are due to race conditions and other timing-specific iss

Unit tests are poorly suited for kernel code. The *vast* majority of production failures in kernel code are due to race conditions and other timing-specific issues, which unit tests are extremely poor at reproducing in a way that mimics how the code is used in real life Another huge source of issues is due to hardware doing something unexpected/not-to-spec - another thing that unit tests would very poorly verify given that, any unit test will simply reproduce how the developer thinks some piece of hardware works, rather than what it does in real life Production kernels are usually better served by long-running stress tests, that try to reproduce real-world use-cases but inject randomness into what they are doing. And indeed, both NT and Linux kernels are extensively tested in this fashion

darau1 4y ago
I don't suppose those long-running stress tests can be automated in any way, can they?
- hannob 4y ago
  They are. There's a project by google called syzcaller, which pretty much does that (it's a bit more intelligent than that, and also includes running sanitizers, which find otherwise hard to detect memory safety and concurrency issues). It's actually causing an unexpected problem: It finds so many issues that kernel devs have troule keeping up fixing them.
  - galangalalgol 4y ago
    Excellent! I keep complaining after each vuln pops up here that there isn't anyone with a kernel pipeline. Hopefully they have some static analysis too? Does the data out of this effort support the ggp that most of these defects are concurrency related vs the norm of memory safety?
  - vincnetas 4y ago
    So it turns out kernel does have unit tests, just that they are running after release.
    - haspok 4y ago
      No, this is way above the unit test level. Fuzzers usually are.
- FivenSeven 4y ago
  They are. Many many companies run tests against the various kernel trees/branches, and will flood you with emails if you break the build. I can't remember who (I think IBM maybe?), but Greg KH said one company made a fantastic test suite out of nowhere. If you submit six patches, it will try each one, tell you what you broke in patch 3, and recommend a way you can fix it. And this just gets emailed to the person who submitted the patches, and the maintainers if they actually merged it. He said it was insanely advanced, and was made without the knowledge of the kernel maintainers. A company just made it and started sending emails one day.
  - darau1 4y ago
    Is there are literature on that IBM system? It sounds fascinating
    - saagarjha 4y ago
      Perhaps they’re talking about the (Intel) kbuild bot?
javier2 4y ago
I think that is fair to say for many hardware device drivers, but things like filesystems definitely should have a place for automated testing, eg. just having a bunch of golden values that verify correctness in backward and forwardcompatibility of the fs drivers should be great.
- unmole 4y ago
  > but things like filesystems definitely should have a place for automated testing And that is what xfstets is used for.
tails4e 4y ago
As a HW guy this seems insane to me. Just because it's hard to do doesnt mean you can't randomise and try to create and catch failures. Might not catch 100%, but better than not trying. HW testing pre silicon is impressively comprehensive as the cost of respinning a chip is in the order of 10s of millions, so the industry found a way. While software is less costly to edit, the cost of updating and rolling out is essentially huge, but uncounted. Finding bugs by extensive testing, even for difficult to catch items, is inherently the right thing to do
- lgeorget 4y ago
  They do apply a lot of fuzzing tests actually and the kernel maintainers also contribute state-of-the-art dynamic analysis tools like UBSAN to check for undefined behaviors or leaks.
- ilyt 4y ago
  As HW guy you are making the hardware so you can make a mock of it relatively easily, that is not the case if driver is essentially reverse-engineered out of how windows driver worked. And many other drivers are "some vendor made it, and they won't tell us how the chip actually work, just give us driver code". Then as other people mentioned hardware isn't exactly 100% predictable, HW bugs happen, firmware upgrades change how stuff works etc. so your test suite might be entirely unable to catch that.
  - tails4e 4y ago
    I know hardware isn't 100% precitable, which is why we expliclty try to test that behaviour through randomized testing, formal proofs, etc. I'm not mocking anything, I'm just saying it's possible to attempt to cover difficult to create scenarios.
    - asalahli 4y ago
      I believe the parent was referring to Test mocks[0], not implying you mocking anything. 0. https://en.wikipedia.org/wiki/Mock_object#Use_in_test-driven_development https://en.wikipedia.org/wiki/Mock_object#Use_in_test-driven...
bamboozled 4y ago
Honestly, it's pretty rare I find a use case for testing period. If I'm writing a library which contains algorithms or something like that it's pretty easy, seems useful or necessary. But a lot of the time I'm gluing things together through API calls and it just seems pointless ?
- otikik 4y ago
  It’s exactly the other way around for me. Because a test takes one input and checks the output. An algorithm for, say, quick sort, will “test positive” on the same unit test that uses a brute force approach. A performance test might pick some differences up, but at that point that’s not a unit test any more
  - bamboozled 4y ago
    So basically, you find it important to test already tested code, with...your own tests?
    - otikik 4y ago
      Yes, that is precisely what I said, you got it right. Have a good day!
Gwypaas 4y ago
> The vast majority of production failures in kernel code are due to race conditions and other timing-specific issues, which unit tests are extremely poor at reproducing in a way that mimics how the code is used in real life. Is this not due to the legacy of a language that allows race conditions? Take Loom [1] in Rust which is a tool to test your concurrency primitives, ensuring that deterministically all interleavings are tested and verified to be sound. Then, when you expose a safe interface, you know that the only potential issues you have are user logic errors or not covering the entire problem space with your tests. https://docs.rs/loom/latest/loom/ https://docs.rs/loom/latest/loom/
- bakugo 4y ago
  I'm surprised that I had to scroll this far down to find a Rust fanatic complaining about C.
  - Gwypaas 4y ago
    I complain similarly about writing deeply multithreaded code with synchronized data structures in Java, Go or whatever. It's a nightmare the day you want to refactor anything and have to keep the entire implicit state in your mind to make sure you don't trip over yourself.
- Cacti 4y ago
  Race conditions are not limited to those “caused by language”, whatever that means.
- IgorPartola 4y ago
  You also interact with hardware. An RC-safe language does nothing when you suddenly realize there is an RC baked into the disk driver you are using and if you send it specific commands in a specific order too fast it does something unexpected.
  - Gwypaas 4y ago
    Yes, that widens the problem space considerably, especially if it depends on the rate of access or something similarly non-deterministic. But you're still fundamentally testing interleavings through a synchronization library. Still, reducing the problem space to such problems instead of any time you reach for atomics or a mutex, I imagine, removes the majority of bugs, so lets not throw the baby out with the bathwater because it's not perfect.
fulafel 4y ago
> The vast majority of production failures in kernel code are due to race conditions and other timing-specific issues This assertion calls for links to data that backs this up I think. Published counterarguments seemed to be easy to find, eg this survey suggests that most bugs are reproducible ("bohrbugs"): https://xiaotingdu.github.io/publications/TR2018.pdf https://xiaotingdu.github.io/publications/TR2018.pdf (journal link: https://ieeexplore.ieee.org/xpl/RecentIssue.jsp?punumber=24 https://ieeexplore.ieee.org/xpl/RecentIssue.jsp?punumber=24)

Unit tests are poorly suited for kernel code. The *vast* majority of production failures in kernel code are due to race conditions and other timing-specific iss

darau1 4y ago

I don't suppose those long-running stress tests can be automated in any way, can they?

hannob 4y ago
They are. There's a project by google called syzcaller, which pretty much does that (it's a bit more intelligent than that, and also includes running sanitizers, which find otherwise hard to detect memory safety and concurrency issues). It's actually causing an unexpected problem: It finds so many issues that kernel devs have troule keeping up fixing them.
- galangalalgol 4y ago
  Excellent! I keep complaining after each vuln pops up here that there isn't anyone with a kernel pipeline. Hopefully they have some static analysis too? Does the data out of this effort support the ggp that most of these defects are concurrency related vs the norm of memory safety?
- vincnetas 4y ago
  So it turns out kernel does have unit tests, just that they are running after release.
  - haspok 4y ago
    No, this is way above the unit test level. Fuzzers usually are.
FivenSeven 4y ago
They are. Many many companies run tests against the various kernel trees/branches, and will flood you with emails if you break the build. I can't remember who (I think IBM maybe?), but Greg KH said one company made a fantastic test suite out of nowhere. If you submit six patches, it will try each one, tell you what you broke in patch 3, and recommend a way you can fix it. And this just gets emailed to the person who submitted the patches, and the maintainers if they actually merged it. He said it was insanely advanced, and was made without the knowledge of the kernel maintainers. A company just made it and started sending emails one day.
- darau1 4y ago
  Is there are literature on that IBM system? It sounds fascinating
  - saagarjha 4y ago
    Perhaps they’re talking about the (Intel) kbuild bot?

javier2 4y ago

I think that is fair to say for many hardware device drivers, but things like filesystems definitely should have a place for automated testing, eg. just having a bunch of golden values that verify correctness in backward and forwardcompatibility of the fs drivers should be great.

unmole 4y ago
> but things like filesystems definitely should have a place for automated testing And that is what xfstets is used for.

tails4e 4y ago

As a HW guy this seems insane to me. Just because it's hard to do doesnt mean you can't randomise and try to create and catch failures. Might not catch 100%, but better than not trying. HW testing pre silicon is impressively comprehensive as the cost of respinning a chip is in the order of 10s of millions, so the industry found a way. While software is less costly to edit, the cost of updating and rolling out is essentially huge, but uncounted. Finding bugs by extensive testing, even for difficult to catch items, is inherently the right thing to do

lgeorget 4y ago
They do apply a lot of fuzzing tests actually and the kernel maintainers also contribute state-of-the-art dynamic analysis tools like UBSAN to check for undefined behaviors or leaks.
ilyt 4y ago
As HW guy you are making the hardware so you can make a mock of it relatively easily, that is not the case if driver is essentially reverse-engineered out of how windows driver worked. And many other drivers are "some vendor made it, and they won't tell us how the chip actually work, just give us driver code". Then as other people mentioned hardware isn't exactly 100% predictable, HW bugs happen, firmware upgrades change how stuff works etc. so your test suite might be entirely unable to catch that.
- tails4e 4y ago
  I know hardware isn't 100% precitable, which is why we expliclty try to test that behaviour through randomized testing, formal proofs, etc. I'm not mocking anything, I'm just saying it's possible to attempt to cover difficult to create scenarios.
  - asalahli 4y ago
    I believe the parent was referring to Test mocks[0], not implying you mocking anything. 0. https://en.wikipedia.org/wiki/Mock_object#Use_in_test-driven_development https://en.wikipedia.org/wiki/Mock_object#Use_in_test-driven...

bamboozled 4y ago

Honestly, it's pretty rare I find a use case for testing period. If I'm writing a library which contains algorithms or something like that it's pretty easy, seems useful or necessary. But a lot of the time I'm gluing things together through API calls and it just seems pointless ?

otikik 4y ago
It’s exactly the other way around for me. Because a test takes one input and checks the output. An algorithm for, say, quick sort, will “test positive” on the same unit test that uses a brute force approach. A performance test might pick some differences up, but at that point that’s not a unit test any more
- bamboozled 4y ago
  So basically, you find it important to test already tested code, with...your own tests?
  - otikik 4y ago
    Yes, that is precisely what I said, you got it right. Have a good day!

Gwypaas 4y ago

> The vast majority of production failures in kernel code are due to race conditions and other timing-specific issues, which unit tests are extremely poor at reproducing in a way that mimics how the code is used in real life. Is this not due to the legacy of a language that allows race conditions? Take Loom [1] in Rust which is a tool to test your concurrency primitives, ensuring that deterministically all interleavings are tested and verified to be sound. Then, when you expose a safe interface, you know that the only potential issues you have are user logic errors or not covering the entire problem space with your tests. https://docs.rs/loom/latest/loom/ https://docs.rs/loom/latest/loom/

bakugo 4y ago
I'm surprised that I had to scroll this far down to find a Rust fanatic complaining about C.
- Gwypaas 4y ago
  I complain similarly about writing deeply multithreaded code with synchronized data structures in Java, Go or whatever. It's a nightmare the day you want to refactor anything and have to keep the entire implicit state in your mind to make sure you don't trip over yourself.
Cacti 4y ago
Race conditions are not limited to those “caused by language”, whatever that means.
IgorPartola 4y ago
You also interact with hardware. An RC-safe language does nothing when you suddenly realize there is an RC baked into the disk driver you are using and if you send it specific commands in a specific order too fast it does something unexpected.
- Gwypaas 4y ago
  Yes, that widens the problem space considerably, especially if it depends on the rate of access or something similarly non-deterministic. But you're still fundamentally testing interleavings through a synchronization library. Still, reducing the problem space to such problems instead of any time you reach for atomics or a mutex, I imagine, removes the majority of bugs, so lets not throw the baby out with the bathwater because it's not perfect.

fulafel 4y ago

> The vast majority of production failures in kernel code are due to race conditions and other timing-specific issues This assertion calls for links to data that backs this up I think. Published counterarguments seemed to be easy to find, eg this survey suggests that most bugs are reproducible ("bohrbugs"): https://xiaotingdu.github.io/publications/TR2018.pdf https://xiaotingdu.github.io/publications/TR2018.pdf (journal link: https://ieeexplore.ieee.org/xpl/RecentIssue.jsp?punumber=24 https://ieeexplore.ieee.org/xpl/RecentIssue.jsp?punumber=24)