I'd like to point out that if you are doing performance testing, you should read a little stats or have someone good at stats lend a hand. Because it is really

I'd like to point out that if you are doing performance testing, you should read a little stats or have someone good at stats lend a hand. Because it is really easy to fool yourself. I've fooled myself and I worry about doing it again. Take effect size, for example. Suppose I run my test before and after the latest commit. The software now has 2% fewer TPS. Is that meaningful? It depends. Should I rely on a single run? Almost certainly not. Suppose performance improves by 203%. Is that meaningful? Probably. Should I rely on a single run? Almost certainly not. Then there's the obnoxious problems of vast amounts of uncontrollable variables. Folks run buckets of testing on cloud platforms without knowing if they're landing on a physical machine they used last time and so their bits are disk-warm, or that there's a network glitch in central1-a but not central1-b but they don't know which one they're using, or they run tests at different times of day and don't realise they will get different competition for cloud resources due to diurnal demand ... tl;dr run more tests and get someone to help you. If you run ab2 three times and publish a breathless blog post, be that on your soul.