3 ms·

> Code that has been thoroughly tested is akin to a running system for a year Except that isn't even remotely true. Perhaps if you system has a purely machine

> Code that has been thoroughly tested is akin to a running system for a year Except that isn't even remotely true. Perhaps if you system has a purely machine interface, with a limited 'dictionary' of interactions, then a test suite could replicate deployment to a very large degree, but otherwise at best it can give only an indication of how use-worthy the system is. Take a (reasonably complex, with human interface) production system, implement as much testing as you like, and I can guarantee any significant refactoring will introduce a regression, and quite possibly a serious one. The problem with testing is that you need to know what to test. So unless you have some method of capturing every possible way a system is actually used, you will always run into the 'unknown unknowns' that you didn't realize even needed testing.

seadan83 3y ago
> and I can guarantee any significant refactoring will introduce a regression Refactoring suffers from having almost no meaning. I would interpret what you are saying as re-architecture and/or update of system design. That is what most people mean by refactoring today. The article laments that people skip the part of "make it clean" in the "make it work, make it clean, make it fast, in that order" In theory refactoring is a safe activity to do because you are only updating something that has a defined API which is tested. > Except that isn't even remotely true. This is hyperbole. How long before a system is actually fully and truly battle tested? 1 month, 1 year, 5 years? I'm making up numbers; but you could think of it in terms of iterations and code paths. How many executions of a block of logic before you start to trust it more? The passage of time adds more certainty because conditions vary over time. Without any testing, that production release is going to be iteration number one in the journey of battle testing that updated (which can be called re-written) code. All I'm saying is running that code through a battery of tests fast-forwards (some) of that battle testing time. As an analogy, car tires - testing is like running a car tire for a few thousand miles before giving it to the customer. When getting a new one (ie: any update to the code at all), the customer is not getting a new tire with zero miles on it, where perhaps the rubber is bad and that would have been discovered easily after about 10 miles. That car tire might still have issues after 20k miles that are abnormal, but nonetheless the testing released a car tire that had more confidence compared to one with no testing at all. I think my favorite analogy would be more RPG & DnD like games. Testing is like having your resurrected characters start at level 3 instead of level 1. Are they level 9? No, but at the same time it's far better than starting at scratch every time. Every time any modification is done to the code, you don't have to start back at level 1 on your journey to get back to battle tested.
seadan83 3y ago
Alternative response: >> Code that has been thoroughly tested is akin to a running system for a year > Except that isn't even remotely true. Perhaps if you system has a purely machine interface, with a limited 'dictionary' of interactions, then a test suite could replicate deployment to a very large degree, but otherwise at best it can give only an indication of how use-worthy the system is The last sentence there, I would state the exact same about battle testing. Both testing and 'battle testing' is just executing code, no real difference from that perspective. The difference comes in the form and variety of scenarios. I'm making up the duration amount for the sake of examples, and clearly that can vary based on the system and the efficacy of the testing. My assertion is that testing is akin to running code in production for 'n' units of time. Perhaps it's exactly the same as running that code for one second in production. Does one second in production give much more than an indication? Not really, but testing & battle testing are in the same boat. > The problem with testing is that you need to know what to test. So unless you have some method of capturing every possible way a system is actually used, you will always run into the 'unknown unknowns' that you didn't realize even needed testing. This is just a problem with building systems. This line of reasoning can become a bit nihilistic (why do any testing at all, manual or automated?) Let's say a system has 10 major permutations, 2 of them are unknown unknowns. Let's say further that it takes 10 seconds running in production to be aware of half of those cases. Applying tests that cover those 5 scenarios is then akin to running the system for 10 seconds. If i have 100 modules, I change one, and there is a test suite that gives me the confidence of having run all 100 for at least 10 seconds (in varying combinations of together, in isolation, in shadow environments, etc).. then I will have effectively gained 10 seconds of battle testing before having released anything.
- somewhereoutth 3y ago
  > The difference comes in the form and variety of scenarios. The difference is that one is real, whereas the other is simulated. It is very hard indeed often impossible to perfectly simulate the actual use of a system. Thus the simulation will be incomplete, and quite possibly in fundamental ways that aren't appreciated until problems occur. It doesn't matter how many tests there are or long they are run for - they can only be at best an indication of how the system will actually behave when confronted by its 'battle' environment.

seadan83 3y ago

> and I can guarantee any significant refactoring will introduce a regression Refactoring suffers from having almost no meaning. I would interpret what you are saying as re-architecture and/or update of system design. That is what most people mean by refactoring today. The article laments that people skip the part of "make it clean" in the "make it work, make it clean, make it fast, in that order" In theory refactoring is a safe activity to do because you are only updating something that has a defined API which is tested. > Except that isn't even remotely true. This is hyperbole. How long before a system is actually fully and truly battle tested? 1 month, 1 year, 5 years? I'm making up numbers; but you could think of it in terms of iterations and code paths. How many executions of a block of logic before you start to trust it more? The passage of time adds more certainty because conditions vary over time. Without any testing, that production release is going to be iteration number one in the journey of battle testing that updated (which can be called re-written) code. All I'm saying is running that code through a battery of tests fast-forwards (some) of that battle testing time. As an analogy, car tires - testing is like running a car tire for a few thousand miles before giving it to the customer. When getting a new one (ie: any update to the code at all), the customer is not getting a new tire with zero miles on it, where perhaps the rubber is bad and that would have been discovered easily after about 10 miles. That car tire might still have issues after 20k miles that are abnormal, but nonetheless the testing released a car tire that had more confidence compared to one with no testing at all. I think my favorite analogy would be more RPG & DnD like games. Testing is like having your resurrected characters start at level 3 instead of level 1. Are they level 9? No, but at the same time it's far better than starting at scratch every time. Every time any modification is done to the code, you don't have to start back at level 1 on your journey to get back to battle tested.

seadan83 3y ago

Alternative response: >> Code that has been thoroughly tested is akin to a running system for a year > Except that isn't even remotely true. Perhaps if you system has a purely machine interface, with a limited 'dictionary' of interactions, then a test suite could replicate deployment to a very large degree, but otherwise at best it can give only an indication of how use-worthy the system is The last sentence there, I would state the exact same about battle testing. Both testing and 'battle testing' is just executing code, no real difference from that perspective. The difference comes in the form and variety of scenarios. I'm making up the duration amount for the sake of examples, and clearly that can vary based on the system and the efficacy of the testing. My assertion is that testing is akin to running code in production for 'n' units of time. Perhaps it's exactly the same as running that code for one second in production. Does one second in production give much more than an indication? Not really, but testing & battle testing are in the same boat. > The problem with testing is that you need to know what to test. So unless you have some method of capturing every possible way a system is actually used, you will always run into the 'unknown unknowns' that you didn't realize even needed testing. This is just a problem with building systems. This line of reasoning can become a bit nihilistic (why do any testing at all, manual or automated?) Let's say a system has 10 major permutations, 2 of them are unknown unknowns. Let's say further that it takes 10 seconds running in production to be aware of half of those cases. Applying tests that cover those 5 scenarios is then akin to running the system for 10 seconds. If i have 100 modules, I change one, and there is a test suite that gives me the confidence of having run all 100 for at least 10 seconds (in varying combinations of together, in isolation, in shadow environments, etc).. then I will have effectively gained 10 seconds of battle testing before having released anything.

somewhereoutth 3y ago
> The difference comes in the form and variety of scenarios. The difference is that one is real, whereas the other is simulated. It is very hard indeed often impossible to perfectly simulate the actual use of a system. Thus the simulation will be incomplete, and quite possibly in fundamental ways that aren't appreciated until problems occur. It doesn't matter how many tests there are or long they are run for - they can only be at best an indication of how the system will actually behave when confronted by its 'battle' environment.