5 ms·

Actually, I am not entirely sure how much of the testing there is automated. Presumably there are at least some automated test tools floating around here and th

by cjfd 5y ago

Actually, I am not entirely sure how much of the testing there is automated. Presumably there are at least some automated test tools floating around here and there for the kernel. Major parts of the kernel could fall under the 'systems' exceptions where they are very close to the hardware. It is possible to do TDD to stuff that is close to hardware but it certainly is harder and it is more likely that some failures fall through the cracks when doing TDD. One thing that is different about the linux kernel, as far as I know, is how thorough the reviewing is that is going on for every patch. Also, there is the merge window followed by a long stabilization period. I would say that if you are not doing TDD you are going to need processes like that if you want to maintain stability. Also note that processes like thorough review by multiple people and a stabilization period four or five times longer than the merge window sound really expensive. Much more expensive than TDD. But also if one does TDD, still system/end-to-end tests are desirable. Not as much as without TDD but without it things would still break somewhat regularly.

Chris_Newton 5y ago
With systems programming, performance matters, and all that extra indirection can be expensive. Indirection can also make it harder to rigorously analyse all possible paths through a piece of code and make sure you’ve handled all required cases.
- dnautics 5y ago
  It's very possible to do the indirection at compile-time, set a flag only active during tests.
  - Chris_Newton 5y ago
    It’s possible, but typically you then pay a higher price on the other point I mentioned. I used to work on a library that solved a certain type of mathematical problem, written mainly in C++. It was built separately for many target platforms in production: different operating systems, processor architectures, compilers. Each supported combination of these might require special provisions in the code due to compiler bugs, hardware limitations, etc. Then you’d have diagnostic code that was only included in the debug builds used during development. There are more than enough preprocessor shenanigans to go around in that kind of environment already, and another layer for test-related flags and checks isn’t going to break any records for most readable code.
    - dnautics 5y ago
      in the languages I work with this is not a problem. Mox, for example, is what I use in Elixir, and even though "swapping out a module" is kind of a global change, it has facilities to track these mappings in runtime, even in concurrent testing, but the module is selected at compile-time, and only in test, so it is a "zero-cost abstraction" for running in prod. I think this is just internalized pain of working with C++ (I suffered a lot with that 20 years ago). For systems-level stuff, I'm getting very excited about Zig, I imagine it will be basically "equally easy" to instrument these sorts of things at compile-time in Zig, as types are first-class values at compile-time. Zig is also super-composable; I have an example prime sieve that can run 60+ settings (combinations of single threaded, multithreaded, bool array, bitmap, hyperthread-awareness) by using a compile-time for loop across a series of instrumented settings; so long as I stay within the standard library the language itself already takes care of the polyfill for platform stuff, and I have access to those flags, if, say I wanted hyper-optimize for things like sse. I don't see any mocking libraries yet, but I doubt that instrumenting a mocking namespace in test will be any more difficult than it would be, in, say, elixir.
chitowneats 5y ago
Tests != TDD. TDD encourages an architecture with lots of indirection so that it can achieve the necessary polymorphism for injecting dependencies in test. I can assure you the linux kernel is not TDD'ed.
- wpietri 5y ago
  I think that's typically true, but that's because most TDD is done in languages where there's not a lot of cost for indirection. I expect we'd see a different outcome if something like a kernel were started with a TDD approach. For example, we might see ways of expressing indirection for testing that then get removed when building the final product. Or we can look at the great strides made in virtualization since the Linux kernel was started. Something previously hard to test becomes much easier. And I'd expect more so if the testing had co-evolved. Of course, there still might be hard-to-test areas. But given the massive ingenuity applied to the kernel development process over the years, I expect that they would have advanced what's possible in TDD if they had pursued it.
  - chitowneats 5y ago
    Why do you think that linux kernel developers have not changed their approach? TDD is not a new idea. If indeed they have continued for decades to do it "the wrong way", imposing costs onto the project and society proportional to the number of preventable bugs creeping in, is this not a major scandal for the industry? I suspect that the core team have reasons that they have not developed tooling to enable TDD of the Linux kernel. And that those reasons are more substantive than "meh".
    - wpietri 5y ago
      TDD is not a new idea, but it's about 10 years newer than the Linux kernel. My experience is that projects very rarely make a switch like the one described by the Factorio team, so I assumed the Linux folks haven't. If you're saying they do work in a TDD fashion, I'm happy to take your word for it.
- dnautics 5y ago
berkes 5y ago
> One thing that is different about the linux kernel, as far as I know, is how thorough the reviewing is that is going on for every patch. This is another underexposed feature of TDD: reviews. When I see a PR with solid test coverage, and those are green, it greatly speeds up the process. I just need to read through tests, see if they make sense, match the requirements and test those. And then glance over the added code quickly. It (unfortunately, I might add) is not always possible to flat out reject a PR without solid test coverage, so it often is less work for me to add some test myself to a PR, than to manually poke around the changed code or to read through it in detail. This is not TDD, but testing-after-the-fact, and those test often are of much lower quality, but at least this is less work, for me. Testing greatly speeds up the work for a reviewer.

by cjfd 5y ago

Chris_Newton 5y ago

With systems programming, performance matters, and all that extra indirection can be expensive. Indirection can also make it harder to rigorously analyse all possible paths through a piece of code and make sure you’ve handled all required cases.

dnautics 5y ago
It's very possible to do the indirection at compile-time, set a flag only active during tests.
- Chris_Newton 5y ago
  It’s possible, but typically you then pay a higher price on the other point I mentioned. I used to work on a library that solved a certain type of mathematical problem, written mainly in C++. It was built separately for many target platforms in production: different operating systems, processor architectures, compilers. Each supported combination of these might require special provisions in the code due to compiler bugs, hardware limitations, etc. Then you’d have diagnostic code that was only included in the debug builds used during development. There are more than enough preprocessor shenanigans to go around in that kind of environment already, and another layer for test-related flags and checks isn’t going to break any records for most readable code.
  - dnautics 5y ago
    in the languages I work with this is not a problem. Mox, for example, is what I use in Elixir, and even though "swapping out a module" is kind of a global change, it has facilities to track these mappings in runtime, even in concurrent testing, but the module is selected at compile-time, and only in test, so it is a "zero-cost abstraction" for running in prod. I think this is just internalized pain of working with C++ (I suffered a lot with that 20 years ago). For systems-level stuff, I'm getting very excited about Zig, I imagine it will be basically "equally easy" to instrument these sorts of things at compile-time in Zig, as types are first-class values at compile-time. Zig is also super-composable; I have an example prime sieve that can run 60+ settings (combinations of single threaded, multithreaded, bool array, bitmap, hyperthread-awareness) by using a compile-time for loop across a series of instrumented settings; so long as I stay within the standard library the language itself already takes care of the polyfill for platform stuff, and I have access to those flags, if, say I wanted hyper-optimize for things like sse. I don't see any mocking libraries yet, but I doubt that instrumenting a mocking namespace in test will be any more difficult than it would be, in, say, elixir.

chitowneats 5y ago

Tests != TDD. TDD encourages an architecture with lots of indirection so that it can achieve the necessary polymorphism for injecting dependencies in test. I can assure you the linux kernel is not TDD'ed.

wpietri 5y ago
I think that's typically true, but that's because most TDD is done in languages where there's not a lot of cost for indirection. I expect we'd see a different outcome if something like a kernel were started with a TDD approach. For example, we might see ways of expressing indirection for testing that then get removed when building the final product. Or we can look at the great strides made in virtualization since the Linux kernel was started. Something previously hard to test becomes much easier. And I'd expect more so if the testing had co-evolved. Of course, there still might be hard-to-test areas. But given the massive ingenuity applied to the kernel development process over the years, I expect that they would have advanced what's possible in TDD if they had pursued it.
- chitowneats 5y ago
  Why do you think that linux kernel developers have not changed their approach? TDD is not a new idea. If indeed they have continued for decades to do it "the wrong way", imposing costs onto the project and society proportional to the number of preventable bugs creeping in, is this not a major scandal for the industry? I suspect that the core team have reasons that they have not developed tooling to enable TDD of the Linux kernel. And that those reasons are more substantive than "meh".
  - wpietri 5y ago
    TDD is not a new idea, but it's about 10 years newer than the Linux kernel. My experience is that projects very rarely make a switch like the one described by the Factorio team, so I assumed the Linux folks haven't. If you're saying they do work in a TDD fashion, I'm happy to take your word for it.
dnautics 5y ago

berkes 5y ago

> One thing that is different about the linux kernel, as far as I know, is how thorough the reviewing is that is going on for every patch. This is another underexposed feature of TDD: reviews. When I see a PR with solid test coverage, and those are green, it greatly speeds up the process. I just need to read through tests, see if they make sense, match the requirements and test those. And then glance over the added code quickly. It (unfortunately, I might add) is not always possible to flat out reject a PR without solid test coverage, so it often is less work for me to add some test myself to a PR, than to manually poke around the changed code or to read through it in detail. This is not TDD, but testing-after-the-fact, and those test often are of much lower quality, but at least this is less work, for me. Testing greatly speeds up the work for a reviewer.