It's not trivial to go back in versions to check for improvements or regressions, because some optimizations can introduce bugs that is later discovered, or you

It's not trivial to go back in versions to check for improvements or regressions, because some optimizations can introduce bugs that is later discovered, or you introduce a vital feature that degrades performance... So you can make your life easier by having automatic performance tests that are run before each release, and if you discovered a performance issue you write a regression test as usual... What I'm trying to say is: Do performance testing!