One mechanism we developed was to build a variant of our storage node that could run in isolation. This meant that synthetic testing would give us some optimal

One mechanism we developed was to build a variant of our storage node that could run in isolation. This meant that synthetic testing would give us some optimal numbers for hardware vetting and performance changes. I proved quite quickly our application was quite thread poor and the costs of fixing it was quite worth it. Using other synthetic benchmarks to compare what the systems were capable of. I was gone before that was finished, but it was quite an improvement. It also allowed cold volumes to exist in an over subscription model. None of this excuses good real world telemetry and evaluation of your outliers.