2 ms·
I would like to see load testing done in some kind of 100% deterministic environment. Make a system emulator where everything is fully deterministic, including
by londons_explore 6y ago
I would like to see load testing done in some kind of 100% deterministic environment. Make a system emulator where everything is fully deterministic, including perfectly timed IO and network characteristics.
Then your load test won't have any random noise from things like an SSD with inconsistent performance, or a CPU that throttles down depending on room temperature.
A zero-noise load test can be much shorter to get good results, and means you won't be investigating spurious performance regressions. You can compare traces between tested software versions down to clock-cycle accuracy to really locate issues, rather than just guestimating.
- awinter-py 6y agoyes this love the gatling focus on CI/CD (though would imagine it's not that simple) -- repeatability and automatic regression testing are as important as testing new architecture there's value in testing with a live network because your prod system has a live network, but a network conditioner can simulate some crappiness, and it forces you to understand your system; when you can repeat a failure with a conditioner, it's much more likely you've understood the failure re noise, 100%. Many LTs have unclear acceptance criteria that generally record a 'fail', can only be interpreted by the original author, and may not always predict prod
- gkop 6y agoWhat is the state of the art in deterministic computation environments? Naively: I would start with the basics like dedicated heavy duty hardware with all the ECC and radiation hardening available. Then I would make sure to compute literally everything multiple times and check that the results match. At this point, the execution characteristics of the computation are dramatically altered from the real world, the test results are unlikely to be useful. You suggest tactics that stop short of redundant computation. Would that really work? Like, how do you gain perfect confidence in your system emulator? I don’t see how you would fully account for bitflips...