3 ms·

This patch got me curious how the kernel is even tested. I have yet to see a single kernel commit that included any sort of tests.

by enw 5y ago

YZF 5y ago
The answer to how is the kernel tested is (historically) mostly manually by a lot of people: https://stackoverflow.com/questions/3177338/how-is-the-linux-kernel-tested https://stackoverflow.com/questions/3177338/how-is-the-linux... Basically what used to be known as alpha/beta testing. Seems like there's more automated testing these days. I work on a smaller piece of software with many automated tests that is significantly less reliable. I wonder if there's some lessons there. Another interesting tidbit is that our tests break a lot as well. In "The Olde Days" many a pieces of complex/reliable software was written without a single unit test or any automated test for that matter. Sure, some of it had bugs, maybe even nasty bugs, but so does most software today. Have we made progress? maybe.
- josephg 5y ago
  Tests let you modify code that’s unfamiliar to you without fear you’ll accidentally break something. If you live in a codebase and work with it every day, automated testing is arguably less important because you build an intuition around what kind of changes might break things. I wouldn’t be surprised if the lack of testing has actively fostered the ownership model Linux has adopted. You would need something like that to keep the bugs at bay. But there’s lots of code out there which doesn’t (or can’t) have a dedicated owner in the same kind of way. I had an issue opened on one of my GitHub projects the other day. I’m the sole maintainer of this project, the code is fiendishly complex and it’s working without any changes for about 18 months. Somebody found a bug! But after 18 months I’ve forgotten the mental context I had while writing that code - so I was terrified I’d break something else while fixing that problem. But I had an extensive test suite to fall back on. So when I fixed the bug and the existing tests all passed, I had the confidence to publish a new version. (And of course, I added the bug’s repro to the test suite for next time). Lots of projects need guard rails like this. Maybe they’re solo projects. Or maybe there’s a lot of churn in the team. Sometimes there’s just more code than there are engineers to keep track of everything, so people are bouncing around a lot and don’t build much expertise in a single area. For the long tail of software, I’m pretty confident in saying unit tests have dramatically improved reliability beyond the standard we had a couple of decades ago. Have they made us lazier too? Maybe! But in my book they’re definitely a win on the whole.
caskstrength 5y ago
You probably couldn't find any because tests are usually submitted as standalone commits. You can find tests for several kernel subsystems here: https://github.com/torvalds/linux/tree/master/tools/testing https://github.com/torvalds/linux/tree/master/tools/testing

This patch got me curious how the kernel is even tested. I have yet to see a single kernel commit that included any sort of tests.

by enw 5y ago

This patch got me curious how the kernel is even tested. I have yet to see a single kernel commit that included any sort of tests.

YZF 5y ago

The answer to how is the kernel tested is (historically) mostly manually by a lot of people: https://stackoverflow.com/questions/3177338/how-is-the-linux-kernel-tested https://stackoverflow.com/questions/3177338/how-is-the-linux... Basically what used to be known as alpha/beta testing. Seems like there's more automated testing these days. I work on a smaller piece of software with many automated tests that is significantly less reliable. I wonder if there's some lessons there. Another interesting tidbit is that our tests break a lot as well. In "The Olde Days" many a pieces of complex/reliable software was written without a single unit test or any automated test for that matter. Sure, some of it had bugs, maybe even nasty bugs, but so does most software today. Have we made progress? maybe.

josephg 5y ago
Tests let you modify code that’s unfamiliar to you without fear you’ll accidentally break something. If you live in a codebase and work with it every day, automated testing is arguably less important because you build an intuition around what kind of changes might break things. I wouldn’t be surprised if the lack of testing has actively fostered the ownership model Linux has adopted. You would need something like that to keep the bugs at bay. But there’s lots of code out there which doesn’t (or can’t) have a dedicated owner in the same kind of way. I had an issue opened on one of my GitHub projects the other day. I’m the sole maintainer of this project, the code is fiendishly complex and it’s working without any changes for about 18 months. Somebody found a bug! But after 18 months I’ve forgotten the mental context I had while writing that code - so I was terrified I’d break something else while fixing that problem. But I had an extensive test suite to fall back on. So when I fixed the bug and the existing tests all passed, I had the confidence to publish a new version. (And of course, I added the bug’s repro to the test suite for next time). Lots of projects need guard rails like this. Maybe they’re solo projects. Or maybe there’s a lot of churn in the team. Sometimes there’s just more code than there are engineers to keep track of everything, so people are bouncing around a lot and don’t build much expertise in a single area. For the long tail of software, I’m pretty confident in saying unit tests have dramatically improved reliability beyond the standard we had a couple of decades ago. Have they made us lazier too? Maybe! But in my book they’re definitely a win on the whole.

caskstrength 5y ago

You probably couldn't find any because tests are usually submitted as standalone commits. You can find tests for several kernel subsystems here: https://github.com/torvalds/linux/tree/master/tools/testing https://github.com/torvalds/linux/tree/master/tools/testing