Surely that depends on exactly how much it has changed and how much variation is usually present. If a metric that has been 5 +/- 1 for years suddenly reads 500

Surely that depends on exactly how much it has changed and how much variation is usually present. If a metric that has been 5 +/- 1 for years suddenly reads 500 after a new release, it might be worth looking into.