I work in SRE where a pretty common expression is “what gets measured gets fixed”. I used to take that as at least mildly inspirational, and to mean that more a

I work in SRE where a pretty common expression is “what gets measured gets fixed”. I used to take that as at least mildly inspirational, and to mean that more and better monitoring leads to more things being fixed. And to some extent that’s true. In recent years though I’ve come to see the downsides of that mantra as outweighing the good of it. Because some things are either extraordinarily difficult or expensive to measure, or because understanding what the measurement is demonstrating is beyond the intellectual reach or experience of many people. By the latter sentiment I mean, it’s not enough to just show a number or a graph, it has to be interpreted, and for some things that interpretation is very challenging if you’re not a (or the) expert in that system. As a result, it’s more like “easy to measure and understand things get fixed, everything else gets ignored”. It disdains or glosses over the idea that maybe a person or team’s subjective opinion about what’s important to fix carries any weight at all, because if it was really so important, surely they’d be able to demonstrate that in a form that someone (possibly willfully) ignorant of the system can understand. I see the same forces at work here, in marketing and a/b testing. The simple to understand metrics are what are optimized, while the more complicated ones get ignored or drowned out. The longer term benefits are hard to measure, and more importantly, hard to understand and interpret.