Performance testing is hugely valuable but nothing beats having performance metrics captured and made available from production. You never really know how thing

Performance testing is hugely valuable but nothing beats having performance metrics captured and made available from production. You never really know how things are going to perform until you see them under real load against real data. There’s been many times I’ve been bitten by this, including the one time we found that a small portion of users would set the number of results returned to the max and scroll through hundreds of pages of results, blowing the cache for everyone. I tend to hear “we run performance tests so we don’t get regressions” a lot more than “we monitor the performance of releases to canaries and rollback”. It shouldn’t be one or the other, both are invaluable.