3 ms·

Alternative response: >> Code that has been thoroughly tested is akin to a running system for a year > Except that isn't even remotely true. Perhaps if you sy

Alternative response: >> Code that has been thoroughly tested is akin to a running system for a year > Except that isn't even remotely true. Perhaps if you system has a purely machine interface, with a limited 'dictionary' of interactions, then a test suite could replicate deployment to a very large degree, but otherwise at best it can give only an indication of how use-worthy the system is The last sentence there, I would state the exact same about battle testing. Both testing and 'battle testing' is just executing code, no real difference from that perspective. The difference comes in the form and variety of scenarios. I'm making up the duration amount for the sake of examples, and clearly that can vary based on the system and the efficacy of the testing. My assertion is that testing is akin to running code in production for 'n' units of time. Perhaps it's exactly the same as running that code for one second in production. Does one second in production give much more than an indication? Not really, but testing & battle testing are in the same boat. > The problem with testing is that you need to know what to test. So unless you have some method of capturing every possible way a system is actually used, you will always run into the 'unknown unknowns' that you didn't realize even needed testing. This is just a problem with building systems. This line of reasoning can become a bit nihilistic (why do any testing at all, manual or automated?) Let's say a system has 10 major permutations, 2 of them are unknown unknowns. Let's say further that it takes 10 seconds running in production to be aware of half of those cases. Applying tests that cover those 5 scenarios is then akin to running the system for 10 seconds. If i have 100 modules, I change one, and there is a test suite that gives me the confidence of having run all 100 for at least 10 seconds (in varying combinations of together, in isolation, in shadow environments, etc).. then I will have effectively gained 10 seconds of battle testing before having released anything.

somewhereoutth 3y ago
> The difference comes in the form and variety of scenarios. The difference is that one is real, whereas the other is simulated. It is very hard indeed often impossible to perfectly simulate the actual use of a system. Thus the simulation will be incomplete, and quite possibly in fundamental ways that aren't appreciated until problems occur. It doesn't matter how many tests there are or long they are run for - they can only be at best an indication of how the system will actually behave when confronted by its 'battle' environment.
- seadan83 3y ago
  I agree. Rocket science is a good analogy here, there is no way to simulate space, on the other hand there are a number of smart checks you can do when a rocket is updated before putting it back on the launchpad.

somewhereoutth 3y ago

> The difference comes in the form and variety of scenarios. The difference is that one is real, whereas the other is simulated. It is very hard indeed often impossible to perfectly simulate the actual use of a system. Thus the simulation will be incomplete, and quite possibly in fundamental ways that aren't appreciated until problems occur. It doesn't matter how many tests there are or long they are run for - they can only be at best an indication of how the system will actually behave when confronted by its 'battle' environment.

seadan83 3y ago
I agree. Rocket science is a good analogy here, there is no way to simulate space, on the other hand there are a number of smart checks you can do when a rocket is updated before putting it back on the launchpad.