3 ms·

> The only way to test all the major code paths through the simple API is to hardcode implementation detail into your unit tests to induce all the various switc

> The only way to test all the major code paths through the simple API is to hardcode implementation detail into your unit tests to induce all the various switching contexts indirectly. Is the "only way" not property-based testing (and maybe fuzz testing)? If that doesn't get you there, it is likely that the API is poorly designed.

jandrewrogers 3y ago
You are assuming the function being tested is more trivial than it actually is. Often these are too small (code-wise) to even have private APIs, the internal behavior of the function is simply and necessarily too complex to unit test in the traditional way. Some function behavior is intentionally and intrinsically tied to temporal access patterns in the API usage. Unless you can simulate a broad cross-section of real-world runtime API access timing patterns with your testing framework, you won't test all of the code paths. You often see this test problem with scheduler-like functions, where the correct function behavior varies based on internal resource pressures that are an interaction between temporal access patterns of the API and the runtime environment. It is a single function, self-contained, and quite simple, maybe not more than 100 LoC, but test environments are so sterile that usually only a single code path is actually used no matter what you throw at the API. Some functions have critical code paths that dynamically switch strategies to mitigate when certain types of contention or resource starvation conditions are detected internally. These can be extremely rare cases across the set of possible inputs, such that fuzzing is unlikely to trigger them, or the set of inputs that can trigger them is dependent on exogenous environmental details e.g. the machine where the test is run. As a fun side-effect, sometimes these functions do not have deterministic result. Getting the same result out of the function all the time does not imply correctness, you also have to know why you got the result you did. All of the above does not apply to writing business logic in Java or similar. But for high-performance and/or high-reliability systems software, these cases come up often enough that it is a well-understood testing problem. Even if you expose all of the internal implementation detail to the unit tests it is not always possible to reliably trigger all the code paths from an arbitrary test absent purpose-built test tooling.

jandrewrogers 3y ago

You are assuming the function being tested is more trivial than it actually is. Often these are too small (code-wise) to even have private APIs, the internal behavior of the function is simply and necessarily too complex to unit test in the traditional way. Some function behavior is intentionally and intrinsically tied to temporal access patterns in the API usage. Unless you can simulate a broad cross-section of real-world runtime API access timing patterns with your testing framework, you won't test all of the code paths. You often see this test problem with scheduler-like functions, where the correct function behavior varies based on internal resource pressures that are an interaction between temporal access patterns of the API and the runtime environment. It is a single function, self-contained, and quite simple, maybe not more than 100 LoC, but test environments are so sterile that usually only a single code path is actually used no matter what you throw at the API. Some functions have critical code paths that dynamically switch strategies to mitigate when certain types of contention or resource starvation conditions are detected internally. These can be extremely rare cases across the set of possible inputs, such that fuzzing is unlikely to trigger them, or the set of inputs that can trigger them is dependent on exogenous environmental details e.g. the machine where the test is run. As a fun side-effect, sometimes these functions do not have deterministic result. Getting the same result out of the function all the time does not imply correctness, you also have to know why you got the result you did. All of the above does not apply to writing business logic in Java or similar. But for high-performance and/or high-reliability systems software, these cases come up often enough that it is a well-understood testing problem. Even if you expose all of the internal implementation detail to the unit tests it is not always possible to reliably trigger all the code paths from an arbitrary test absent purpose-built test tooling.