8 ms·

Isn't SQLite a counterexample? They have more code that has tests than their actually running code. https://www.sqlite.org/testing.html https://www.sqlite.org/t

Isn't SQLite a counterexample? They have more code that has tests than their actually running code. https://www.sqlite.org/testing.html https://www.sqlite.org/testing.html

omginternets 4y ago
Isn't the public API quite narrow, though?
- dahfizz 4y ago
  The API is massive. Just the number build parameters and boot parameters is very high. There is also a ton of things you can tune at runtime in /sys/proc . Not to mention all the syscalls and the subtle ways they all interact with each other.
  - omginternets 4y ago
    Surely it’s miles smaller than the public Linux API.
  - interactivecode 4y ago
    sounds like someone should write tests for that
iLoveOncall 4y ago
> They have more code that has tests than their actually running code. I'd say that's very common and not exclusive to some specific project. I made a change of about 30 lines of code recently, and it had 140 lines of tests (and I, by far, do not cover every situation). I think I've rarely encountered well tested pieces of code that were not way smaller than the tests that came with them.
ilyt 4y ago
SQLite doesn't have hardware drivers (that IIRC are still majority of kernel bugs) that need actual hardware to test (as hardware mocks are at best mildly useful, coz real hardware can be... weird) And unit testing is relatively useless, what would be useful are end to end tests that start with userspace API but that's much harder task, altho hopefully those tests wouldn't need to be changed that often.
- silvestrov 4y ago
  In other words: unit testing requires that your code is a unit. i.e. does not depend on external timing or fickle hardware. Very, very difficult to unit test someting that runs using hardware interupts.
- mastax 4y ago
  A project as large and well funded as the Linux kernel could have a hardware test farm at least with reasonable coverage of popular hardware. "But Linux isn't that well funded!" Sure, but it's orders of magnitude better funded than Embassy[0], which runs tests on real hardware automatically before every merge. There's also the Linux testing project, which is technically third party. It's not clear to me how extensive it is but for a project as important as Linux I think it has to be graded as "needs improvement." [0] https://github.com/embassy-rs/embassy https://github.com/embassy-rs/embassy [1] https://linux-test-project.github.io/ https://linux-test-project.github.io/
- exDM69 4y ago
  I work as a software developer in a hardware company. We have A LOT of tests. But testing costs a lot more when there is hardware present. It's a whole new dimension in your testing matrix. You can't run on cloud provider CI. Hardware has failures, someone needs to be there to reboot the systems and swap out broken ones (and thing break when they run hot 24/7). And you need some way to test whether the hardware did what it is supposed to do. Although some kernel driver developers have testing clusters with real hardware (like GPU drivers devs), there is no effin way Linux could be effectively tested without someone paying a lot of money to set up real hardware and people to keep it running. Of course the hardware can be simulated but simulation and emulation are slow (and potentially expensive). At work we run tests on simulation for next gen hardware which doesn't exist yet. It is about 100,000x slower than the real deal, so it's obviously not a solution that scales up. Testing purely software products without a hardware dimension is so much simpler and cheaper.
  - q-big 4y ago
    > Although some kernel driver developers have testing clusters with real hardware (like GPU drivers devs), there is no effin way Linux could be effectively tested without someone paying a lot of money to set up real hardware and people to keep it running. Just crowdsource this testing: I am sure that there exist some people who own the piece of hardware and are willing to run a test script, say, every * night * week * new testing version of the kernel (depending on your level of passion for the hardware and/or Linux). I do believe that there do exist a lot of people who would join such a crowdsourcing effort if the necessary infrastructure existed (i.e. it is very easy to run and submit the results).
KptMarchewa 4y ago
I don't think you can argue they are classic, mock-the-world test one function tests. Giant majority of those are just huge amounts of SQL statements running all over - which are integration tests.
closeparen 4y ago
Those are more integration tests (load stuff into the DB and query it out) vs. unit tests (set up mocks around functions and call them with different arguments).
- bvrmn 4y ago
  It's a cursed notion that unit-tests are "set up mocks around functions and call them with different arguments". It's more useful to categorize tests by how hard to setup an environment. In reality cost/usefulness line lies on this boundary. * in single process * multiple process using IPC * multiple processes using network * tests validate functions calling foreign services I see many devs (including myself) call "integration" tests as "unit" tests because in your particular app/system they are easy to spawn locally even without any container.
  - closeparen 4y ago
    While I tend to agree, the Go toolchain authors are very specific: it doesn’t count as unit test coverage when the code and the test are in different packages. And our management has no interest in breaking with them on this. You are free to write tests that compose packages, and I’ll do it for my own sanity and confidence. But that’s in addition to, not instead of, the minimum “not horrifically irresponsible engineering” standard of universal, exhaustive, very detail oriented mock driven unit tests within all packages.

Isn't SQLite a counterexample? They have more code that has tests than their actually running code. https://www.sqlite.org/testing.html https://www.sqlite.org/testing.html

omginternets 4y ago

Isn't the public API quite narrow, though?

dahfizz 4y ago
The API is massive. Just the number build parameters and boot parameters is very high. There is also a ton of things you can tune at runtime in /sys/proc . Not to mention all the syscalls and the subtle ways they all interact with each other.
- omginternets 4y ago
  Surely it’s miles smaller than the public Linux API.
- interactivecode 4y ago
  sounds like someone should write tests for that

iLoveOncall 4y ago

> They have more code that has tests than their actually running code. I'd say that's very common and not exclusive to some specific project. I made a change of about 30 lines of code recently, and it had 140 lines of tests (and I, by far, do not cover every situation). I think I've rarely encountered well tested pieces of code that were not way smaller than the tests that came with them.

ilyt 4y ago

SQLite doesn't have hardware drivers (that IIRC are still majority of kernel bugs) that need actual hardware to test (as hardware mocks are at best mildly useful, coz real hardware can be... weird) And unit testing is relatively useless, what would be useful are end to end tests that start with userspace API but that's much harder task, altho hopefully those tests wouldn't need to be changed that often.

silvestrov 4y ago
In other words: unit testing requires that your code is a unit. i.e. does not depend on external timing or fickle hardware. Very, very difficult to unit test someting that runs using hardware interupts.
mastax 4y ago
A project as large and well funded as the Linux kernel could have a hardware test farm at least with reasonable coverage of popular hardware. "But Linux isn't that well funded!" Sure, but it's orders of magnitude better funded than Embassy[0], which runs tests on real hardware automatically before every merge. There's also the Linux testing project, which is technically third party. It's not clear to me how extensive it is but for a project as important as Linux I think it has to be graded as "needs improvement." [0] https://github.com/embassy-rs/embassy https://github.com/embassy-rs/embassy [1] https://linux-test-project.github.io/ https://linux-test-project.github.io/
exDM69 4y ago
I work as a software developer in a hardware company. We have A LOT of tests. But testing costs a lot more when there is hardware present. It's a whole new dimension in your testing matrix. You can't run on cloud provider CI. Hardware has failures, someone needs to be there to reboot the systems and swap out broken ones (and thing break when they run hot 24/7). And you need some way to test whether the hardware did what it is supposed to do. Although some kernel driver developers have testing clusters with real hardware (like GPU drivers devs), there is no effin way Linux could be effectively tested without someone paying a lot of money to set up real hardware and people to keep it running. Of course the hardware can be simulated but simulation and emulation are slow (and potentially expensive). At work we run tests on simulation for next gen hardware which doesn't exist yet. It is about 100,000x slower than the real deal, so it's obviously not a solution that scales up. Testing purely software products without a hardware dimension is so much simpler and cheaper.
- q-big 4y ago
  > Although some kernel driver developers have testing clusters with real hardware (like GPU drivers devs), there is no effin way Linux could be effectively tested without someone paying a lot of money to set up real hardware and people to keep it running. Just crowdsource this testing: I am sure that there exist some people who own the piece of hardware and are willing to run a test script, say, every * night * week * new testing version of the kernel (depending on your level of passion for the hardware and/or Linux). I do believe that there do exist a lot of people who would join such a crowdsourcing effort if the necessary infrastructure existed (i.e. it is very easy to run and submit the results).

KptMarchewa 4y ago

I don't think you can argue they are classic, mock-the-world test one function tests. Giant majority of those are just huge amounts of SQL statements running all over - which are integration tests.

closeparen 4y ago

Those are more integration tests (load stuff into the DB and query it out) vs. unit tests (set up mocks around functions and call them with different arguments).

bvrmn 4y ago
It's a cursed notion that unit-tests are "set up mocks around functions and call them with different arguments". It's more useful to categorize tests by how hard to setup an environment. In reality cost/usefulness line lies on this boundary. * in single process * multiple process using IPC * multiple processes using network * tests validate functions calling foreign services I see many devs (including myself) call "integration" tests as "unit" tests because in your particular app/system they are easy to spawn locally even without any container.
- closeparen 4y ago
  While I tend to agree, the Go toolchain authors are very specific: it doesn’t count as unit test coverage when the code and the test are in different packages. And our management has no interest in breaking with them on this. You are free to write tests that compose packages, and I’ll do it for my own sanity and confidence. But that’s in addition to, not instead of, the minimum “not horrifically irresponsible engineering” standard of universal, exhaustive, very detail oriented mock driven unit tests within all packages.