4 ms·

The argument against Tests sounds like a tooling problem to me. Why shouldn't a compiler be able to optimize the modular approach to the same code that a monoli

The argument against Tests sounds like a tooling problem to me. Why shouldn't a compiler be able to optimize the modular approach to the same code that a monolithic approach yields?

nitwit005 6y ago
As long as dynamically loading code at runtime is allowed, proving that there will only ever be one implementation of an interface isn't going to be possible. That makes fully optimizing away the cost of the indirection difficult.
tsimionescu 6y ago
Why is that relevant? The fact is that it doesn't (there are good reasons for it, but that is another matter). You can't work with the ideal tools, you can only work with the tools that actually exist. So, with the actual tools, this sacrifice must be made. And now do answer your actual question, the reason is that the languages we use are just not expressive enough, and the compiler doesn't have enough time. If you are calling an interface which could be either the actual code or the mock code, the actual code can't be inlined, and that alone usually discards dozens of other optimizations. You can't specify in a language something like 'assume this implementation in optimized builds', so we're stuck. There are ways around this, to be fair. In C#, you could ale all of your code generic, such that there could be two versions of your class - one with a real dependency, MyClass<RealDependency1, RealDependency2,...>, and one with mocked dependencies, MyClass<MockedDependency1, MockedDependency2,...>. Not sure if the C# compiler or JIT actually knows how to use this during optimization, but in principle it could. With C++ templates this would definitely allow optimization. With Java's extremely advanced JIT, you wouldn't actually need this code change, the compiler will be able to do it for you after a few thousand runs through the code, when it notices a single version is being called.
- choeger 6y ago
  It is relevant because they chose to abandon best practices because of a lack of tooling. A badly chosen language constraining your development methods is like a tail wagging the dog. There are languages and, more importantly, implementations out there that allow this kind of design. Just think of SML/NJ as an example for the principle. If performance is important and C# doesn't allow unit testing performant code then switch to C++ or Rust. It is staggering how many people seem to see the choice of language as an absolutely unchangeable decision. Rewriting even millions of lines of js or python code is definitely doable, even incrementally, and if your organization does not deem it worth the effort then maybe performance is not your most important concern.
- fpoling 6y ago
  I have seen a lot of projects that use interfaces in C++ or Java just to be able to mock the tests. This puzzled me. If it is known that the there will be exactly 2 implementations of the interface, one real one and one for mocking, then do not use interfaces. Instead use a global flag and at relevant places instead of calling interface’ methods just test the flag and branch for test/non-test code. If the flag is a compile-time constant, then compilers will be able to eliminate the checks completely. If it is not a constant, then due to branch prediction over a simple global the runtime impact will be minuscule, much smaller than with interfaces even without advanced link-time optimizations. The big plus of this approach is that test-only coupling is explicit and easily grepable. Another one is that if test mocking requires complex changes to the control flaw, providing that level of control via interfaces often leads to complex and awkward interfaces, but with the flag it is very straightforward to get the necessary behavior.
  - Leherenn 6y ago
    It massively hurts readability and can quickly become messy in my experience. If you're running on multiple targets and testing on host, then you have an #ifdef spaghetti in no time. In all the codebases I have worked on, they started like this because it is easy, but then moved on to interfaces/templates because maintainability is an issue with this method.
    - fpoling 6y ago
      I worked on QT application where we observed the opposite. As typical with QT the code was a collection of components that were glued together in the main application. For testing it turned out adding few flags to the application was enough to implement unit and integration test driver. Surely there were few ifs, but the code was very readable and such design allowed very straightforward implementation of integration tests.
- hvidgaard 6y ago
  Using generics in this way in C# is asking for a headache. And C# being JIT compiled, the compiler can, theoretically, optimize by inlining. I do not know if it does. In any case, for the vast majority if code bases, maintainability is more important than 10% better performance.
- DarkWiiPlayer 6y ago
  This kind of thing is one of the reasons I'm really interested in languages that support multi-stage programming. I've played around with Terra and lately I've read many good things about Zig, because they both let you run code at compile-time. Things like dependency-injection could be zero-cost if done at an early compilation stage, so the compiler has a chance to optimise the code with the correct dependency already in place.
- skohan 6y ago
  One of the other drawbacks besides performance that I have seen in terms of "making code testable" is that it often hurts traceability. If I am jumping into an unfamiliar codebase, the best case is that I can right click on any method invocation, "go to declaration" and see what code is being executed. Often times in codebases which make heavy use of dependency injection, "go to declaration" takes me instead to some kind of interface declaration, and then I have to figure out which code is actually being executed at runtime. I feel like this is a very real cost, and is often not accounted for in terms of the tradeoffs discussed in TDD.
  - tsimionescu 6y ago
    I don't think this is a major problem with interfaces that have a real implementation and a mock implementation. You can just as well "Go to implementation", and will see two options - XImpl and MockXImpl.
- scott00 6y ago
  With the current C# compiler and JIT, using generics to avoid a virtual function call works if the dependencies are structs, but not if they are classes. That said, in my experience it's very hard to get performance gains through inlining in C# because the compiler is very conservative about what it will inline, even if you instruct it to aggressively inline a method (there is no way to force inlining in C#). IIRC this was a conscious decision by the C# team after they did some experiments and found that the benefit of fewer function calls from inlining was outweighed by the negative impact of increased code size on the instruction cache hit rate in all but the most trivial functions. That decision makes it very hard to get some of the huge inlining wins that you see sometimes in C++ where essentially the compiler can recognize that once you recursively inline a few layers of abstraction you're in a special case that is known at compile time and a ton of code can be eliminated. With that kind of situation off the table, a virtual method call that always goes to the same implementation will not cost you very much compared to directly calling the implementation method. It amounts to the difference between a branch and an indirect branch in machine code, and modern hardware has indirect branch predictors that will do a very good job when the indirect branch always goes to the same implementation.

The argument against Tests sounds like a tooling problem to me. Why shouldn't a compiler be able to optimize the modular approach to the same code that a monolithic approach yields?

nitwit005 6y ago

As long as dynamically loading code at runtime is allowed, proving that there will only ever be one implementation of an interface isn't going to be possible. That makes fully optimizing away the cost of the indirection difficult.

tsimionescu 6y ago

Why is that relevant? The fact is that it doesn't (there are good reasons for it, but that is another matter). You can't work with the ideal tools, you can only work with the tools that actually exist. So, with the actual tools, this sacrifice must be made. And now do answer your actual question, the reason is that the languages we use are just not expressive enough, and the compiler doesn't have enough time. If you are calling an interface which could be either the actual code or the mock code, the actual code can't be inlined, and that alone usually discards dozens of other optimizations. You can't specify in a language something like 'assume this implementation in optimized builds', so we're stuck. There are ways around this, to be fair. In C#, you could ale all of your code generic, such that there could be two versions of your class - one with a real dependency, MyClass<RealDependency1, RealDependency2,...>, and one with mocked dependencies, MyClass<MockedDependency1, MockedDependency2,...>. Not sure if the C# compiler or JIT actually knows how to use this during optimization, but in principle it could. With C++ templates this would definitely allow optimization. With Java's extremely advanced JIT, you wouldn't actually need this code change, the compiler will be able to do it for you after a few thousand runs through the code, when it notices a single version is being called.

choeger 6y ago
It is relevant because they chose to abandon best practices because of a lack of tooling. A badly chosen language constraining your development methods is like a tail wagging the dog. There are languages and, more importantly, implementations out there that allow this kind of design. Just think of SML/NJ as an example for the principle. If performance is important and C# doesn't allow unit testing performant code then switch to C++ or Rust. It is staggering how many people seem to see the choice of language as an absolutely unchangeable decision. Rewriting even millions of lines of js or python code is definitely doable, even incrementally, and if your organization does not deem it worth the effort then maybe performance is not your most important concern.
fpoling 6y ago
I have seen a lot of projects that use interfaces in C++ or Java just to be able to mock the tests. This puzzled me. If it is known that the there will be exactly 2 implementations of the interface, one real one and one for mocking, then do not use interfaces. Instead use a global flag and at relevant places instead of calling interface’ methods just test the flag and branch for test/non-test code. If the flag is a compile-time constant, then compilers will be able to eliminate the checks completely. If it is not a constant, then due to branch prediction over a simple global the runtime impact will be minuscule, much smaller than with interfaces even without advanced link-time optimizations. The big plus of this approach is that test-only coupling is explicit and easily grepable. Another one is that if test mocking requires complex changes to the control flaw, providing that level of control via interfaces often leads to complex and awkward interfaces, but with the flag it is very straightforward to get the necessary behavior.
- Leherenn 6y ago
  It massively hurts readability and can quickly become messy in my experience. If you're running on multiple targets and testing on host, then you have an #ifdef spaghetti in no time. In all the codebases I have worked on, they started like this because it is easy, but then moved on to interfaces/templates because maintainability is an issue with this method.
  - fpoling 6y ago
    I worked on QT application where we observed the opposite. As typical with QT the code was a collection of components that were glued together in the main application. For testing it turned out adding few flags to the application was enough to implement unit and integration test driver. Surely there were few ifs, but the code was very readable and such design allowed very straightforward implementation of integration tests.
hvidgaard 6y ago
Using generics in this way in C# is asking for a headache. And C# being JIT compiled, the compiler can, theoretically, optimize by inlining. I do not know if it does. In any case, for the vast majority if code bases, maintainability is more important than 10% better performance.
DarkWiiPlayer 6y ago
This kind of thing is one of the reasons I'm really interested in languages that support multi-stage programming. I've played around with Terra and lately I've read many good things about Zig, because they both let you run code at compile-time. Things like dependency-injection could be zero-cost if done at an early compilation stage, so the compiler has a chance to optimise the code with the correct dependency already in place.
skohan 6y ago
One of the other drawbacks besides performance that I have seen in terms of "making code testable" is that it often hurts traceability. If I am jumping into an unfamiliar codebase, the best case is that I can right click on any method invocation, "go to declaration" and see what code is being executed. Often times in codebases which make heavy use of dependency injection, "go to declaration" takes me instead to some kind of interface declaration, and then I have to figure out which code is actually being executed at runtime. I feel like this is a very real cost, and is often not accounted for in terms of the tradeoffs discussed in TDD.
- tsimionescu 6y ago
  I don't think this is a major problem with interfaces that have a real implementation and a mock implementation. You can just as well "Go to implementation", and will see two options - XImpl and MockXImpl.
scott00 6y ago
With the current C# compiler and JIT, using generics to avoid a virtual function call works if the dependencies are structs, but not if they are classes. That said, in my experience it's very hard to get performance gains through inlining in C# because the compiler is very conservative about what it will inline, even if you instruct it to aggressively inline a method (there is no way to force inlining in C#). IIRC this was a conscious decision by the C# team after they did some experiments and found that the benefit of fewer function calls from inlining was outweighed by the negative impact of increased code size on the instruction cache hit rate in all but the most trivial functions. That decision makes it very hard to get some of the huge inlining wins that you see sometimes in C++ where essentially the compiler can recognize that once you recursively inline a few layers of abstraction you're in a special case that is known at compile time and a ton of code can be eliminated. With that kind of situation off the table, a virtual method call that always goes to the same implementation will not cost you very much compared to directly calling the implementation method. It amounts to the difference between a branch and an indirect branch in machine code, and modern hardware has indirect branch predictors that will do a very good job when the indirect branch always goes to the same implementation.