4 ms·
How To Measure Anything[1] by Douglas Hubbard. The basic gist of the book goes something like this: in the real world (especially in a business setting) there
by mindcrime 7y ago
How To Measure Anything[1] by Douglas Hubbard.
The basic gist of the book goes something like this: in the real world (especially in a business setting) there are many things which are hard to measure directly, but which we may care about. Take, for example, "employee morale" which matters because it may affect, say, retention, or product quality. Hubbard suggests that we can measure (many|most|all|??) of these things by using a combination of "calibrated probability assessments"[2], awareness of nth order effects, and Monte Carlo simulation.
Basically, "if something matters, it's because it affects something that can be measured". So you identify the causal chain from "thing" to "measurable thing", have people who are trained in "calibrated probability assessment" estimate the weights of the effects in the causal chain, then build a mathematical model, and use a Monte Carlo simulation to work out how inputs to the system affect the outputs.
Of course it's not perfect, since estimation is always touchy, even using the calibration stuff. And you could still commit an error like leaving an important variable out of the model completely, or sampling from the wrong distribution when doing your simulation. But generally speaking, done with care, this is a way to measure the "unmeasurable" with a level of rigor that's better than just flat out guessing, or ignoring the issue altogether.
[1]: https://www.amazon.com/How-Measure-Anything-Intangibles-Business-ebook/dp/B00INUYS2U https://www.amazon.com/How-Measure-Anything-Intangibles-Busi...
[2]: https://en.wikipedia.org/wiki/Calibrated_probability_assessment https://en.wikipedia.org/wiki/Calibrated_probability_assessm...
- seshagiric 7y agoAs a product manager I can't recommend this book strong enough. It has been a saver for me in a number real life situations - solving "Fermi" problems in Interviews to handling day2day PM stuff (market sizing, analytics etc.). The writing style makes it hard to read (almost like an academic research paper) it's tough to keep urself interested. However the learnings are substantial too.
- iQuercus 7y agoIt’s a good recipe for making a lot of estimation errors. Worse, it will leave you feeling more confident than you should be. Then, when you least expect it, it’ll blow up in your face.
- mindcrime 7y agoThere is merit to what you say, but that's the reason you have to have actual domain experts working on building the model, and why the calibration training matters. That said: "All models are wrong, but some are useful"
- yen223 7y agoOne of the important points made in the book was that one should always be thinking of a range of values, not a single value. This ensures whatever model you use will capture the uncertainty that is inherent in any measurement.
- hutzlibu 7y agoIt also sounds a bit like real life parody to me. How to find out employee morale? Well, you need to hire some statistical experts, who model everything and than give you an estimate and if they got the callibration right, the result might be even close to reality. Definitely easier than the other solution. Like talk to your employees once in a while and have a culture where they are not afraid to speak up if something is bothering them. Ah no, too complicated. Back to math. That works better when dealing with those irrational emotional humans.
- eyegor 7y agoUnfortunately this only works with a small set of workers, in a physical location. If you have 500+ employees, some remote, spanning different countries/cultures, it's not easy to tell how "company morale" is. You can try to send out anonymous surveys and such, but then you run into reporting bias, etc. It's a complicated problem at scale.
- mindcrime 7y agoThe morale thing was just an example to help illustrate the principle...
- hutzlibu 7y agoI know. And my point was, that you can take the statistic too far, especially when dealing with humans. And this is happening in my opinion since a while. Even Stalin said: "the death of one man is a tragedy, but the death of millions just statistic." And by his actions, they were just statistic to him. Humans are more than a number ... but when you are at a desk remote and only have numbers, than you are disconnected and it is much easier to make inhuman decisions as you are dealing with numbers and not humans anymore. So that being said, yes, having the numbers additional that signal a decline in morale for example, is probably not a bad thing to have. If this does not replace actual human contact, because human moral also drops, when they not feel treated as humans anymore and just as numbers.
- deleted 7y ago[deleted]
- smogcutter 7y agoSounds interesting. I’m sure the book covers this, but isn’t this how you wind up with perverse incentives? Once managers figure out that you’re measuring X as a proxy for Y “thing we actually care about” they’re going to start optimizing for X, with potentially unpleasant results.
- crimsonalucard 7y agoThis goes back to the whole correlation does not equal causation thing in statistics. You're likely thinking of a correlated number, the measurement number must be a causative. Direct improvements on any causative number will by definition affect the outcome. Say we're looking at developer retention. A correlated number is like scrum points. A causative number is like salary. Of course, while the data on the causative and correlative effectiveness of both metrics is dispute-able... you still get my point.
- deleted 7y ago[deleted]
- bookofjoe 7y agoIf you like this book you'll love "The Perfectionists" by Simon Winchester; https://www.amazon.com/Perfectionists-Precision-Engineers-Created-Modern/dp/0062652567/ref=sr_1_1?keywords=perfectionists+simon&qid=1579489354&sr=8-1 https://www.amazon.com/Perfectionists-Precision-Engineers-Cr...