4 ms·

Unittests is not the same thing as TDD religion. At its base, it just means dividing things up into small units and testing their functionality as exhaustively

by titzer 4y ago

Unittests is not the same thing as TDD religion. At its base, it just means dividing things up into small units and testing their functionality as exhaustively as possible. It's a divide-and-conquer. The hard part in adding unittests is deciding what a unit is, and when a unit is important enough that it should have its own battery of tests. Choosing the wrong boundaries means a lot of wasted time and effort testing things that likely won't break or change so fast that put a drag on refactoring. I disagree that a kernel can't or shouldn't be unit tested. At the very least, it has a strong interface in the userspace system calls. Of course you should unit test the system calls. Especially because Linux's motto is not to break userspace. The other benefit from unit testing that is overlooked is that it accelerates optimization. If you have a good set of unit tests that test all observable behaviors of a system means you can optimize the logic inside and constantly rerun those tests to make sure it's not broken. This speeds up the experimentation process and clearly delineates the interface so that you can see when and how you can "cheat" on things that aren't observable. So, hard disagree. Test the bejesus out of the kernel. It will harden it and open up the ability hyperoptimize the units inside. edit to add: well-written (i.e. terse and readable) tests are excellent documentation on how a unit should behave.

nu11ptr 4y ago
I both agree and disagree. There are PARTS of the kernel that probably should be unit tested. As I specified in my answer, these tend to be algorithmic. I would also agree that an interface that cannot change ("don't break user space!") makes a good candidate possibly. However, I think there are parts that probably should NOT be unit tested, because unit testing isn't free and slows down change by getting very close to internals. As an example, device drivers that are tied tightly to hardware. While it would sound nice to have an nvidia GPU fake/mock, in reality it is probably not possible to create one, keep it up to date, etc. The complexity would be enormous and might not show anything as you wouldn't know if it was a bug in the fake or driver. As such, I retain my answer of: it depends what you are trying to accomplish. Different testing strategies for different types of code and domains.
- titzer 4y ago
  There are some gray areas, sure. I usually find a way around making mocks, but maybe it's just the domains I am in. My biggest mistakes are mostly just writing tests that are too intertwined with internal interfaces. So they break all the time and need to be adjusted when refactoring. That's a drag.
- thayne 4y ago
  I think the same is true of any sufficiently complex project. There are areas where unit tests are valuable, and areas where the costs outweigh the benefits. There isn't anything really special about a kernel in that regard (except for the specific things that aren't worth unit testing).
  - closeparen 4y ago
    In a commercial setting, unit test coverage inevitably becomes a KPI that needs to be driven upwards regardless of whether rank and file engineers consider it worthwhile in particular cases.
- xg15 4y ago
  > As an example, device drivers that are tied tightly to hardware. While it would sound nice to have an nvidia GPU fake/mock, in reality it is probably not possible to create one, keep it up to date, etc. The complexity would be enormous and might not show anything as you wouldn't know if it was a bug in the fake or driver. Ok, but then again: How would you make sure your Nvidia driver is working correctly?
hnfong 4y ago
Your comment sounds like TDD religion to me. "Deciding what a unit is" isn't the hard part. The hard part is finding the units that benefit from unit tests (and convincing other religious people about this, as I am trying to now, which in this case - not many). The Linux system calls are "deceptively simple". Simple to test, right? How complicated can say `write(2)` be? But if you actually tried doing it, I'd be surprised if you can write reliable "unit" tests beyond writing to /dev/null. The practical way to test system calls are to test it with real world usage. Unit tests here might catch some problems, but the vast majority of kernel bugs aren't those that can be caught with unit tests. (If they're bad enough, the bug will cause the OS to crash before it completes booting, no unit tests needed.) The more subtle bugs are often hard to reproduce, only happens in certain loads and hardware. As to your final edit-to-add: you're joking right? Like Linux syscalls need more documentation. POSIX (I'm aware this is not exactly Linux documentation, but still) was a spec literally before Linux was started. edit PS: https://tenor.com/view/unittest-unit-test-gif-10813141 https://tenor.com/view/unittest-unit-test-gif-10813141
- makapuf 4y ago
  I don't know how many LOC you could trigger with a write. Including all drivers, filesystems , ..., perhaps millions. Hardly a unit test. And then, a driver is difficult to unit test. BUT some parts of the kernel could use unit tests. The thing is, they don't and the usage of the kernel shows that they would be triggered by usage, but it wouldn't harm.
- Izkata 4y ago
  > Your comment sounds like TDD religion to me. GP is using the original definition of unit test, GGP is using the religious implementation-level definition TDD advocates have come up with. From GGP: > but integration and behaviour tests for the external interface/API are more important than exercising an internal implementation detail. They call this an integration test, but by the original definition, testing the interface is a unit test. An integration test is how multiple units interact with each other. Testing the implementation details is what TDD advocates have turned unit testing into, by changing the definition of "unit" from semantics (a self-contained module that does one thing from the business perspective) to syntax (a function). > edit PS: https://tenor.com/view/unittest-unit-test-gif-10813141 https://tenor.com/view/unittest-unit-test-gif-10813141 Seems like you agree with the original definition. The TDD version of this GIF would be a test for each part that makes up the handle (the implementation details instead of the interface).
deleted 4y ago
[deleted]
hbrn 4y ago
> If you have a good set of unit tests that test all observable behaviors of a system means you can optimize the logic inside and constantly rerun those tests to make sure it's not broken This is a naïve utopian world view. The word "good" is doing all the heavy lifting for you. In the last 15 years I haven't seen a single company that had "good" unit tests. I'm inclined to believe they don't exist. Some of the tests I saw turned out to be good. Most of them weren't. The reality is that if you're committing to unit tests, a big chunk of your tests will be shit. And when that's the case, accelerated optimization is far from guaranteed.

by titzer 4y ago

nu11ptr 4y ago

I both agree and disagree. There are PARTS of the kernel that probably should be unit tested. As I specified in my answer, these tend to be algorithmic. I would also agree that an interface that cannot change ("don't break user space!") makes a good candidate possibly. However, I think there are parts that probably should NOT be unit tested, because unit testing isn't free and slows down change by getting very close to internals. As an example, device drivers that are tied tightly to hardware. While it would sound nice to have an nvidia GPU fake/mock, in reality it is probably not possible to create one, keep it up to date, etc. The complexity would be enormous and might not show anything as you wouldn't know if it was a bug in the fake or driver. As such, I retain my answer of: it depends what you are trying to accomplish. Different testing strategies for different types of code and domains.

titzer 4y ago
There are some gray areas, sure. I usually find a way around making mocks, but maybe it's just the domains I am in. My biggest mistakes are mostly just writing tests that are too intertwined with internal interfaces. So they break all the time and need to be adjusted when refactoring. That's a drag.
thayne 4y ago
I think the same is true of any sufficiently complex project. There are areas where unit tests are valuable, and areas where the costs outweigh the benefits. There isn't anything really special about a kernel in that regard (except for the specific things that aren't worth unit testing).
- closeparen 4y ago
  In a commercial setting, unit test coverage inevitably becomes a KPI that needs to be driven upwards regardless of whether rank and file engineers consider it worthwhile in particular cases.
xg15 4y ago
> As an example, device drivers that are tied tightly to hardware. While it would sound nice to have an nvidia GPU fake/mock, in reality it is probably not possible to create one, keep it up to date, etc. The complexity would be enormous and might not show anything as you wouldn't know if it was a bug in the fake or driver. Ok, but then again: How would you make sure your Nvidia driver is working correctly?

hnfong 4y ago

Your comment sounds like TDD religion to me. "Deciding what a unit is" isn't the hard part. The hard part is finding the units that benefit from unit tests (and convincing other religious people about this, as I am trying to now, which in this case - not many). The Linux system calls are "deceptively simple". Simple to test, right? How complicated can say `write(2)` be? But if you actually tried doing it, I'd be surprised if you can write reliable "unit" tests beyond writing to /dev/null. The practical way to test system calls are to test it with real world usage. Unit tests here might catch some problems, but the vast majority of kernel bugs aren't those that can be caught with unit tests. (If they're bad enough, the bug will cause the OS to crash before it completes booting, no unit tests needed.) The more subtle bugs are often hard to reproduce, only happens in certain loads and hardware. As to your final edit-to-add: you're joking right? Like Linux syscalls need more documentation. POSIX (I'm aware this is not exactly Linux documentation, but still) was a spec literally before Linux was started. edit PS: https://tenor.com/view/unittest-unit-test-gif-10813141 https://tenor.com/view/unittest-unit-test-gif-10813141

makapuf 4y ago
I don't know how many LOC you could trigger with a write. Including all drivers, filesystems , ..., perhaps millions. Hardly a unit test. And then, a driver is difficult to unit test. BUT some parts of the kernel could use unit tests. The thing is, they don't and the usage of the kernel shows that they would be triggered by usage, but it wouldn't harm.
Izkata 4y ago
> Your comment sounds like TDD religion to me. GP is using the original definition of unit test, GGP is using the religious implementation-level definition TDD advocates have come up with. From GGP: > but integration and behaviour tests for the external interface/API are more important than exercising an internal implementation detail. They call this an integration test, but by the original definition, testing the interface is a unit test. An integration test is how multiple units interact with each other. Testing the implementation details is what TDD advocates have turned unit testing into, by changing the definition of "unit" from semantics (a self-contained module that does one thing from the business perspective) to syntax (a function). > edit PS: https://tenor.com/view/unittest-unit-test-gif-10813141 https://tenor.com/view/unittest-unit-test-gif-10813141 Seems like you agree with the original definition. The TDD version of this GIF would be a test for each part that makes up the handle (the implementation details instead of the interface).

deleted 4y ago

[deleted]

hbrn 4y ago

> If you have a good set of unit tests that test all observable behaviors of a system means you can optimize the logic inside and constantly rerun those tests to make sure it's not broken This is a naïve utopian world view. The word "good" is doing all the heavy lifting for you. In the last 15 years I haven't seen a single company that had "good" unit tests. I'm inclined to believe they don't exist. Some of the tests I saw turned out to be good. Most of them weren't. The reality is that if you're committing to unit tests, a big chunk of your tests will be shit. And when that's the case, accelerated optimization is far from guaranteed.