Replying to myself here but a pet peeve of mine (and all CDNs do this to some extent) is showing "improved" performance on something like... the Pingdom website

Replying to myself here but a pet peeve of mine (and all CDNs do this to some extent) is showing "improved" performance on something like... the Pingdom website tester (?) which is a one shot page and asset fetch. There's really no valid metric that can be derived from a one shot test. Just because the 2nd run with the magic on is "faster" doesn't mean anything. Hundreds of variables could have changed between runs. Most people use monitoring systems that run synthetics every minute and those are reasonable for an "overview" of trends, but they also don't show you the worst case times that are happening between runs. If you start getting more granular and monitoring at 5 or 10 second intervals you get to see more real world response times. On something like CloudFlare the averages (without Argo) are really good, but the max and standard deviation is very high. Hard to beat for $200/month but when you start paying per Gb you should start testing other providers.