3 ms·

If you can't consistently measure a 5% difference in a tightly controlled environment, how are your real world users going to notice? If your users don't notice

If you can't consistently measure a 5% difference in a tightly controlled environment, how are your real world users going to notice? If your users don't notice, who cares? Furthermore, if that 5% optimization is truly critical to the application (despite not being measurable?), how are you going to ensure that some future change doesn't break that? Something as simple as a compiler update could change some optimization that breaks your assumptions. What do you do in the absence of tests to ensure that doesn't slip through?

citrin_ru 4y ago
Even if user would not notice 5% difference in CPU usage can mean reduction in AWS bill significant enough to care about (but not significant enough to spend days to avoid adding a few lines of comments by embedding all knowledge into tests). > 5% difference in a tightly controlled environment, A test suite usually doesn't run in tightly controlled environment. To get meaningful performance testing results you'll need to run it on dedicated hardware (no noisy neighbors) and don't run different tests in parallel (so one test will not be a noisy neighbor for another test). It is something not hard to do once in a while manually but would be quite expensive to do as part of a test suite which typically is running on each change.
- randomdata 4y ago
  > Even if user would not notice 5% difference in CPU usage can mean reduction in AWS bill significant enough to care about Users won't notice, but they will notice...? > A test suite usually doesn't run in tightly controlled environment. What don't you have control over? > but not significant enough to spend days to avoid adding a few lines of comments by embedding all knowledge into tests Why would you avoid comments? Why would your tests have to take longer to write than your comments? Days to write a test or two to clarify your intent?

citrin_ru 4y ago

Even if user would not notice 5% difference in CPU usage can mean reduction in AWS bill significant enough to care about (but not significant enough to spend days to avoid adding a few lines of comments by embedding all knowledge into tests). > 5% difference in a tightly controlled environment, A test suite usually doesn't run in tightly controlled environment. To get meaningful performance testing results you'll need to run it on dedicated hardware (no noisy neighbors) and don't run different tests in parallel (so one test will not be a noisy neighbor for another test). It is something not hard to do once in a while manually but would be quite expensive to do as part of a test suite which typically is running on each change.

randomdata 4y ago
> Even if user would not notice 5% difference in CPU usage can mean reduction in AWS bill significant enough to care about Users won't notice, but they will notice...? > A test suite usually doesn't run in tightly controlled environment. What don't you have control over? > but not significant enough to spend days to avoid adding a few lines of comments by embedding all knowledge into tests Why would you avoid comments? Why would your tests have to take longer to write than your comments? Days to write a test or two to clarify your intent?