8 ms·

I know that Linus hates unit tests, but these kinds of scenarios are perfect for regression tests. The investigation of the issue took so long, that you'd reall

I know that Linus hates unit tests, but these kinds of scenarios are perfect for regression tests. The investigation of the issue took so long, that you'd really want to spend those extra few hours writing the test to save yourself investigation in the future. The patch doesn't include any comment about a race condition in the actual code, so let's assume that all the knowledge is more-or-less lost and forgotten in 12 months or so.

enw 5y ago
This patch got me curious how the kernel is even tested. I have yet to see a single kernel commit that included any sort of tests.
- YZF 5y ago
  The answer to how is the kernel tested is (historically) mostly manually by a lot of people: https://stackoverflow.com/questions/3177338/how-is-the-linux-kernel-tested https://stackoverflow.com/questions/3177338/how-is-the-linux... Basically what used to be known as alpha/beta testing. Seems like there's more automated testing these days. I work on a smaller piece of software with many automated tests that is significantly less reliable. I wonder if there's some lessons there. Another interesting tidbit is that our tests break a lot as well. In "The Olde Days" many a pieces of complex/reliable software was written without a single unit test or any automated test for that matter. Sure, some of it had bugs, maybe even nasty bugs, but so does most software today. Have we made progress? maybe.
  - josephg 5y ago
    Tests let you modify code that’s unfamiliar to you without fear you’ll accidentally break something. If you live in a codebase and work with it every day, automated testing is arguably less important because you build an intuition around what kind of changes might break things. I wouldn’t be surprised if the lack of testing has actively fostered the ownership model Linux has adopted. You would need something like that to keep the bugs at bay. But there’s lots of code out there which doesn’t (or can’t) have a dedicated owner in the same kind of way. I had an issue opened on one of my GitHub projects the other day. I’m the sole maintainer of this project, the code is fiendishly complex and it’s working without any changes for about 18 months. Somebody found a bug! But after 18 months I’ve forgotten the mental context I had while writing that code - so I was terrified I’d break something else while fixing that problem. But I had an extensive test suite to fall back on. So when I fixed the bug and the existing tests all passed, I had the confidence to publish a new version. (And of course, I added the bug’s repro to the test suite for next time). Lots of projects need guard rails like this. Maybe they’re solo projects. Or maybe there’s a lot of churn in the team. Sometimes there’s just more code than there are engineers to keep track of everything, so people are bouncing around a lot and don’t build much expertise in a single area. For the long tail of software, I’m pretty confident in saying unit tests have dramatically improved reliability beyond the standard we had a couple of decades ago. Have they made us lazier too? Maybe! But in my book they’re definitely a win on the whole.
- caskstrength 5y ago
  You probably couldn't find any because tests are usually submitted as standalone commits. You can find tests for several kernel subsystems here: https://github.com/torvalds/linux/tree/master/tools/testing https://github.com/torvalds/linux/tree/master/tools/testing
TeeMassive 5y ago
I used to be a huge fan boy of the Linux kernel development. But when I got to actually try to see how it's done I've just given up. I mean, setting up your email so it's like the 80s? Coding standards based on old tty terminals? No unit testing? Toxic leaders? No thank you. While the GNU/Linux was 20 years ahead, it's now more than 10 years late. Also relevant: https://www.usenix.org/system/files/1311_05-08_mickens.pdf https://www.usenix.org/system/files/1311_05-08_mickens.pdf > This is not the world of the systems hacker. When you debug a distributed system or an OS kernel, you do it Texas-style. You gather some mean, stoic people, people who have seen things die, and you get some primitive tools, like a compass and a rucksack and a stick that’s pointed on one end, and you walk into the wilderness and you look for trouble, possibly while using chewing tobacco. As a systems hacker, you must be pre-pared to do savage things, unspeakable things, to kill runaway threads with your bare hands, to write directly to network ports using telnet and an old copy of an RFC that you found in the Vatican. When you debug systems code, there are no high-level debates about font choices and the best kind of turquoise, because this is the Old Testament, an angry and monochro-matic world, and it doesn’t matter whether your Arial is Bold or Condensed when people are covered in boils and pestilence and Egyptian pharaoh oppression. HCI people discover bugs by receiving a concerned email from their therapist. Systems people discover bugs by waking up and discovering that their first-born children are missing and “ETIMEDOUT ” has been written in blood on the wall.
- encryptluks2 5y ago
  > setting up your email so it's like the 80s Using email for development isn't the 80s. It is very streamlined and built into the Git code itself and mail clients that are actively maintained like Mutt. Just because you can't imagine what life would be like without Gmail doesn't mean that these maintainers are less efficient and antiquated than you are. > Coding standards based on old tty terminals Don't even know what this means. What coding standard is based on old TTY terminals? TTY is part of Linux. > No unit testing Linux is the most widely used OS in enterprise. The internet runs on Linux. If you want unit tests for everyone then start an open source project to do this, or complain about the enterprises that aren't testing. > Toxic leaders Who is toxic? I love Linus's approach. It has allowed him to get things done without toxic people in the community trying to social engineer their way into destroying something good.
biryani_chicken 5y ago
If Linus doesn't like unit tests, can't a separate entity keep their own independent unit test repository, test the kernel against it and report regressions? I think it would be a worthy project.
- mh- 5y ago
  There's this: https://github.com/linux-test-project/ltp https://github.com/linux-test-project/ltp
- nijave 5y ago
  Depending on the stability of the interfaces, you introduce a game of cat-and-mouse of sorts where people make test breaking changes but some other entity is constantly trying to keep the tests update. Maybe, less of an issue if you're not making changes that require test refactoring (which probably also comes down to the quality of the test)
- nitrogen 5y ago
  I haven't read it in years, but last time I did Phoronix was doing this implicitly by running every new kernel against the Phoronix Test Suite.
wolfi1 5y ago
I don't see how a classical unit test would have catched this, as all other components are mocked away. Regression tests on the other hand can only catch errors that already occurred. I#m not sure that was the case.
- Izkata 5y ago
  The idea is to write a regression test now, to catch it if it's accidentally reintroduced by a future developer who knows nothing about this bug.
  - pydry 5y ago
    Still not something a unit test can really catch. An end to end test maybe (run multiple times coz it's a race condition). I've seen unit tests that try to reproduce race conditions. They're naive mirrors of the code that break as soon as you even think about changing the implementation. They're kind of pointless.
    - mnahkies 5y ago
      Personally I've found that the value in the tests you are describing is in proving that you've fixed the issue. When you have a race condition that is difficult to reason about or reproduce, if you can distil it down to a test case that consistently fails until you fix it then you can be comfortable that your solution holds - yes it may be of little value going forward but it can be a very valuable part of the process. It can also serve as documentation going forward, as whilst it may be brittle, when does fail hopefully the person will check the description/blame and instantly find the context as to why it exists, before using their judgement as to whether it is still valid or not. (Rather than changing something that looks odd and not realising that they've re-introduced the edge case)
      - aetherspawn 5y ago
        Yes, I think the most value comes from reminding someone that the problem exists in say 2 years time. At work the other day I refactored a major system and discovered several completely left of field legacy behaviours because of regression tests that didn’t seem to make sense (“Why would a sane person want THIS to happen? Eh? Ooooh”)
iamgopal 5y ago
Any rationale why he hates unit test ?
- encryptluks2 5y ago
  He doesn't, he just wants the billion dollar enterprises to be testing it not the kernel team which are more of an academic community. Makes sense to me.
  - bitcharmer 5y ago
    It doesn't make much sense to me. There is nothing positive in forfeiting principles of good software engineering in the name of falsely perceived imbalance of stakes in the project. Instead of falling back on the biggest beneficiaries of the project for testing, we should have a participation model that welcomes contributions and improvements at all stages of development. I've contributed to the kernel multiple times over the last few years and it was always a pain to some degree. Sometimes due to outdated contribution model, sometimes because of maintainer's personal preferences and allegiances. Sadly current kernel dev culture still promotes exclusion and building walls instead of reducing tension and removing barriers for entry.
    - unglaublich 5y ago
      They do welcome contributions at all stages. Just because it doesn't happen at your terms doesn't make it exclusive or about building walls. Do you think there is a contribution model that will make everyone happy AND the kernel good?
      - bitcharmer 5y ago
        > Just because it doesn't happen at your terms That's a good example of the toxic attitude I'm referring to. I never said anything about my terms. Just a healthy OSS ecosystem with more balanced control over what and how gets done. > Do you think there is a contribution model that will make everyone happy I don't know, do you? It'd be safe to assume however that the Linux project could benefit from some improvements. Or are you trying to suggest it's perfect in its current form?
encryptluks2 5y ago
Its not that he hates unit tests, it is that the billion dollar enterprises and countless other people using Linux are the ones that should be testing it.
- alpaca128 5y ago
  Doesn't make sense here, doesn't make sense with Windows 10 being mainly tested by actual users, doesn't make sense with AAA games being alpha/beta versions at release. The result is never as good as it could be.
quickthrower2 5y ago
This seems so strange to me in 2021 that perhaps the most widely used OS kernel in the world wouldn’t be aiming to have test coverage, given the billions in value obtained from it. I guess this is for legacy reasons, and I know as a dev it’s very hard to add tests to an untested system both practically and psychologically and that’s in a team environment not open source
- londons_explore 5y ago
  The Linux kernel is pretty well laid out. There are lots of unit and integration tests that could be usefully run against it with no code changes.
- slver 5y ago
  I'm fine with that premise: it's strange. But if you think about the implications of this strangeness, it can be equally interpreted as "Linus is playing a dangerous game" or "maybe we've kind of exaggerated the importance of unit tests, seeing Linux is doing fine". Linux doesn't seem to stand out as exceptionally buggy. It has bugs, but then so does every project that's unit tested.
  - Retric 5y ago
    Alternatively, unit tests are simply a poor fit when context is so critical for Linux kernel bugs. In much the way unit tests are really hard to write for concurrency issues. In theory they should still be useful, but in practice perhaps not.
    - slver 5y ago
      Unit tests are honestly a poor fit for many problems. Unfortunately it takes Linus levels of determination to resist and carve out an exception status for your project.
    - sangnoir 5y ago
      Unit tests also have to be maintained - which compounds the cost
arve0 5y ago
> The patch doesn't include any comment about a race condition in the actual code …but it’s in the commit message, which is basically the same as a comment. https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=8199be001a470209f5c938570cc199abb012fe53 https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/lin...
- aetherspawn 5y ago
  Yes, as long as you know which line to run git blame
  - TeMPOraL 5y ago
    It'll work until the next big rewrite of that component :). I love it when people put this level of detail into their commit messages, but this technique has one weak spot I don't know what to do about: if someone makes a change that confuses git's semantic heuristics - rename a function, split a file into two, etc. (or maybe all of them in a single commit), it creates a boundary for blame/log-trace that's hard to bridge. But then, maybe I'm just not clever enough with git log -L ...
    - nijave 5y ago
      Hopefully whoever is git blaming understands enough about git to hop past any intermediate changes back to the source I've never done any kernel development, but with commercial code it's pretty common to have to dig backwards including going back to the issue tracker or even searching through company chat (Slack, email, etc) if the shop doesn't do [a good job with] documentation
    - nitrogen 5y ago
      The --follow, -M, and -C options have usually helped me out of most blame dead ends, but sometimes I do still have to look up each blamed commit manually to keep following history.
Denvercoder9 5y ago
It's no longer true that the kernel doesn't have unit/regression tests. The kernel ships with a thousand or so selftests nowadays, Linus actually requires selftests for some patches, and there's also a unit testing framework (kunit). Regardless, from the commit message: > I didn't manage to come up with a reproducer in test environment, and the problem can't be reproduced after rebooting.

enw 5y ago

This patch got me curious how the kernel is even tested. I have yet to see a single kernel commit that included any sort of tests.

YZF 5y ago
The answer to how is the kernel tested is (historically) mostly manually by a lot of people: https://stackoverflow.com/questions/3177338/how-is-the-linux-kernel-tested https://stackoverflow.com/questions/3177338/how-is-the-linux... Basically what used to be known as alpha/beta testing. Seems like there's more automated testing these days. I work on a smaller piece of software with many automated tests that is significantly less reliable. I wonder if there's some lessons there. Another interesting tidbit is that our tests break a lot as well. In "The Olde Days" many a pieces of complex/reliable software was written without a single unit test or any automated test for that matter. Sure, some of it had bugs, maybe even nasty bugs, but so does most software today. Have we made progress? maybe.
- josephg 5y ago
  Tests let you modify code that’s unfamiliar to you without fear you’ll accidentally break something. If you live in a codebase and work with it every day, automated testing is arguably less important because you build an intuition around what kind of changes might break things. I wouldn’t be surprised if the lack of testing has actively fostered the ownership model Linux has adopted. You would need something like that to keep the bugs at bay. But there’s lots of code out there which doesn’t (or can’t) have a dedicated owner in the same kind of way. I had an issue opened on one of my GitHub projects the other day. I’m the sole maintainer of this project, the code is fiendishly complex and it’s working without any changes for about 18 months. Somebody found a bug! But after 18 months I’ve forgotten the mental context I had while writing that code - so I was terrified I’d break something else while fixing that problem. But I had an extensive test suite to fall back on. So when I fixed the bug and the existing tests all passed, I had the confidence to publish a new version. (And of course, I added the bug’s repro to the test suite for next time). Lots of projects need guard rails like this. Maybe they’re solo projects. Or maybe there’s a lot of churn in the team. Sometimes there’s just more code than there are engineers to keep track of everything, so people are bouncing around a lot and don’t build much expertise in a single area. For the long tail of software, I’m pretty confident in saying unit tests have dramatically improved reliability beyond the standard we had a couple of decades ago. Have they made us lazier too? Maybe! But in my book they’re definitely a win on the whole.
caskstrength 5y ago
You probably couldn't find any because tests are usually submitted as standalone commits. You can find tests for several kernel subsystems here: https://github.com/torvalds/linux/tree/master/tools/testing https://github.com/torvalds/linux/tree/master/tools/testing

TeeMassive 5y ago

I used to be a huge fan boy of the Linux kernel development. But when I got to actually try to see how it's done I've just given up. I mean, setting up your email so it's like the 80s? Coding standards based on old tty terminals? No unit testing? Toxic leaders? No thank you. While the GNU/Linux was 20 years ahead, it's now more than 10 years late. Also relevant: https://www.usenix.org/system/files/1311_05-08_mickens.pdf https://www.usenix.org/system/files/1311_05-08_mickens.pdf > This is not the world of the systems hacker. When you debug a distributed system or an OS kernel, you do it Texas-style. You gather some mean, stoic people, people who have seen things die, and you get some primitive tools, like a compass and a rucksack and a stick that’s pointed on one end, and you walk into the wilderness and you look for trouble, possibly while using chewing tobacco. As a systems hacker, you must be pre-pared to do savage things, unspeakable things, to kill runaway threads with your bare hands, to write directly to network ports using telnet and an old copy of an RFC that you found in the Vatican. When you debug systems code, there are no high-level debates about font choices and the best kind of turquoise, because this is the Old Testament, an angry and monochro-matic world, and it doesn’t matter whether your Arial is Bold or Condensed when people are covered in boils and pestilence and Egyptian pharaoh oppression. HCI people discover bugs by receiving a concerned email from their therapist. Systems people discover bugs by waking up and discovering that their first-born children are missing and “ETIMEDOUT ” has been written in blood on the wall.

encryptluks2 5y ago
> setting up your email so it's like the 80s Using email for development isn't the 80s. It is very streamlined and built into the Git code itself and mail clients that are actively maintained like Mutt. Just because you can't imagine what life would be like without Gmail doesn't mean that these maintainers are less efficient and antiquated than you are. > Coding standards based on old tty terminals Don't even know what this means. What coding standard is based on old TTY terminals? TTY is part of Linux. > No unit testing Linux is the most widely used OS in enterprise. The internet runs on Linux. If you want unit tests for everyone then start an open source project to do this, or complain about the enterprises that aren't testing. > Toxic leaders Who is toxic? I love Linus's approach. It has allowed him to get things done without toxic people in the community trying to social engineer their way into destroying something good.

biryani_chicken 5y ago

If Linus doesn't like unit tests, can't a separate entity keep their own independent unit test repository, test the kernel against it and report regressions? I think it would be a worthy project.

mh- 5y ago
There's this: https://github.com/linux-test-project/ltp https://github.com/linux-test-project/ltp
nijave 5y ago
Depending on the stability of the interfaces, you introduce a game of cat-and-mouse of sorts where people make test breaking changes but some other entity is constantly trying to keep the tests update. Maybe, less of an issue if you're not making changes that require test refactoring (which probably also comes down to the quality of the test)
nitrogen 5y ago
I haven't read it in years, but last time I did Phoronix was doing this implicitly by running every new kernel against the Phoronix Test Suite.

wolfi1 5y ago

I don't see how a classical unit test would have catched this, as all other components are mocked away. Regression tests on the other hand can only catch errors that already occurred. I#m not sure that was the case.

Izkata 5y ago
The idea is to write a regression test now, to catch it if it's accidentally reintroduced by a future developer who knows nothing about this bug.
- pydry 5y ago
  Still not something a unit test can really catch. An end to end test maybe (run multiple times coz it's a race condition). I've seen unit tests that try to reproduce race conditions. They're naive mirrors of the code that break as soon as you even think about changing the implementation. They're kind of pointless.
  - mnahkies 5y ago
    Personally I've found that the value in the tests you are describing is in proving that you've fixed the issue. When you have a race condition that is difficult to reason about or reproduce, if you can distil it down to a test case that consistently fails until you fix it then you can be comfortable that your solution holds - yes it may be of little value going forward but it can be a very valuable part of the process. It can also serve as documentation going forward, as whilst it may be brittle, when does fail hopefully the person will check the description/blame and instantly find the context as to why it exists, before using their judgement as to whether it is still valid or not. (Rather than changing something that looks odd and not realising that they've re-introduced the edge case)
    - aetherspawn 5y ago
      Yes, I think the most value comes from reminding someone that the problem exists in say 2 years time. At work the other day I refactored a major system and discovered several completely left of field legacy behaviours because of regression tests that didn’t seem to make sense (“Why would a sane person want THIS to happen? Eh? Ooooh”)

iamgopal 5y ago

Any rationale why he hates unit test ?

encryptluks2 5y ago
He doesn't, he just wants the billion dollar enterprises to be testing it not the kernel team which are more of an academic community. Makes sense to me.
- bitcharmer 5y ago
  It doesn't make much sense to me. There is nothing positive in forfeiting principles of good software engineering in the name of falsely perceived imbalance of stakes in the project. Instead of falling back on the biggest beneficiaries of the project for testing, we should have a participation model that welcomes contributions and improvements at all stages of development. I've contributed to the kernel multiple times over the last few years and it was always a pain to some degree. Sometimes due to outdated contribution model, sometimes because of maintainer's personal preferences and allegiances. Sadly current kernel dev culture still promotes exclusion and building walls instead of reducing tension and removing barriers for entry.
  - unglaublich 5y ago
    They do welcome contributions at all stages. Just because it doesn't happen at your terms doesn't make it exclusive or about building walls. Do you think there is a contribution model that will make everyone happy AND the kernel good?
    - bitcharmer 5y ago
      > Just because it doesn't happen at your terms That's a good example of the toxic attitude I'm referring to. I never said anything about my terms. Just a healthy OSS ecosystem with more balanced control over what and how gets done. > Do you think there is a contribution model that will make everyone happy I don't know, do you? It'd be safe to assume however that the Linux project could benefit from some improvements. Or are you trying to suggest it's perfect in its current form?

encryptluks2 5y ago

Its not that he hates unit tests, it is that the billion dollar enterprises and countless other people using Linux are the ones that should be testing it.

alpaca128 5y ago
Doesn't make sense here, doesn't make sense with Windows 10 being mainly tested by actual users, doesn't make sense with AAA games being alpha/beta versions at release. The result is never as good as it could be.

quickthrower2 5y ago

This seems so strange to me in 2021 that perhaps the most widely used OS kernel in the world wouldn’t be aiming to have test coverage, given the billions in value obtained from it. I guess this is for legacy reasons, and I know as a dev it’s very hard to add tests to an untested system both practically and psychologically and that’s in a team environment not open source

londons_explore 5y ago
The Linux kernel is pretty well laid out. There are lots of unit and integration tests that could be usefully run against it with no code changes.
slver 5y ago
I'm fine with that premise: it's strange. But if you think about the implications of this strangeness, it can be equally interpreted as "Linus is playing a dangerous game" or "maybe we've kind of exaggerated the importance of unit tests, seeing Linux is doing fine". Linux doesn't seem to stand out as exceptionally buggy. It has bugs, but then so does every project that's unit tested.
- Retric 5y ago
  Alternatively, unit tests are simply a poor fit when context is so critical for Linux kernel bugs. In much the way unit tests are really hard to write for concurrency issues. In theory they should still be useful, but in practice perhaps not.
  - slver 5y ago
    Unit tests are honestly a poor fit for many problems. Unfortunately it takes Linus levels of determination to resist and carve out an exception status for your project.
  - sangnoir 5y ago
    Unit tests also have to be maintained - which compounds the cost

arve0 5y ago

> The patch doesn't include any comment about a race condition in the actual code …but it’s in the commit message, which is basically the same as a comment. https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=8199be001a470209f5c938570cc199abb012fe53 https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/lin...

aetherspawn 5y ago
Yes, as long as you know which line to run git blame
- TeMPOraL 5y ago
  It'll work until the next big rewrite of that component :). I love it when people put this level of detail into their commit messages, but this technique has one weak spot I don't know what to do about: if someone makes a change that confuses git's semantic heuristics - rename a function, split a file into two, etc. (or maybe all of them in a single commit), it creates a boundary for blame/log-trace that's hard to bridge. But then, maybe I'm just not clever enough with git log -L ...
  - nijave 5y ago
    Hopefully whoever is git blaming understands enough about git to hop past any intermediate changes back to the source I've never done any kernel development, but with commercial code it's pretty common to have to dig backwards including going back to the issue tracker or even searching through company chat (Slack, email, etc) if the shop doesn't do [a good job with] documentation
  - nitrogen 5y ago
    The --follow, -M, and -C options have usually helped me out of most blame dead ends, but sometimes I do still have to look up each blamed commit manually to keep following history.

Denvercoder9 5y ago

It's no longer true that the kernel doesn't have unit/regression tests. The kernel ships with a thousand or so selftests nowadays, Linus actually requires selftests for some patches, and there's also a unit testing framework (kunit). Regardless, from the commit message: > I didn't manage to come up with a reproducer in test environment, and the problem can't be reproduced after rebooting.