Even if user would not notice 5% difference in CPU usage can mean reduction in AWS bill significant enough to care about (but not significant enough to spend da

Even if user would not notice 5% difference in CPU usage can mean reduction in AWS bill significant enough to care about (but not significant enough to spend days to avoid adding a few lines of comments by embedding all knowledge into tests). > 5% difference in a tightly controlled environment, A test suite usually doesn't run in tightly controlled environment. To get meaningful performance testing results you'll need to run it on dedicated hardware (no noisy neighbors) and don't run different tests in parallel (so one test will not be a noisy neighbor for another test). It is something not hard to do once in a while manually but would be quite expensive to do as part of a test suite which typically is running on each change.