There’s a sweet spot that’s hard to hit. On one end you have people who ignore metrics. They get themselves into trouble breaking things without making the easy

There’s a sweet spot that’s hard to hit. On one end you have people who ignore metrics. They get themselves into trouble breaking things without making the easy measurements to know they’re breaking them. They make obvious mistakes. The next group makes subtle mistakes. They’re the ones who have learned how to use metrics and now they have a hammer, everything looks like a nail. I see this a lot in less senior data scientists. They know how to drive with data, and so everything must be done that way. But they make the mistakes this article gets at. The group I like takes a middle ground approach. We recognize that certain network effects are so coupled that you can power maybe one proper experiment on it at a time, and even those experiments might take too long. Other experiments take too long simply by virtue that the thing you want to measure might be long term behavior. So you experiment on what you can, get everything you can by proxy, and make sure to at least measure health via the metrics that matter, but you don’t try to base every decision in something measurable simply because it’s measurable.