5 ms·

If you like the idea of this library you'll probably like the book "How to Measure Anything" by Douglas Hubbard (https://www.goodreads.com/book/show/20933591

If you like the idea of this library you'll probably like the book "How to Measure Anything" by Douglas Hubbard (https://www.goodreads.com/book/show/20933591-how-to-measure-anything https://www.goodreads.com/book/show/20933591-how-to-measure-...). It's all about how to get sensible confidence intervals for things that are often considered unmeasurable such as the value of IT security. The book mostly uses Excel to do this modelling, but it looks like riskquant would be an excellent alternative on that approach, that for the more technically minded practitioner.

jacques_chester 7y ago
The relevant book for this is Measuring and Managing Information Risk: A FAIR Approach by Freund and Jones[0]. Both books are worth reading; Hubbard's influence on FAIR is noticeable and positive. FAIR has the advantage that it comes with a fairly built-out ontology for assembling data or estimates. The OP touches on the top level (Loss Event Magnitude and Loss Event Frequency), but the ontology goes quite deep and can be used at multiple levels of detail. The calculations are not difficult, I've implemented them twice in proofs-of-concept, including one that produces pretty charts. The difficult part, to be honest, is that developing good estimates is difficult and frequently uncomfortable and the gains are not easily internalised. Additionally, serious tool support is lacking in the places where it would make a difference -- issue trackers, for example. [0] https://www.amazon.com/Measuring-Managing-Information-Risk-Approach/dp/0124202314 https://www.amazon.com/Measuring-Managing-Information-Risk-A... edit -- Another good book in this area is Waltzing with Bears by DeMarco & Lister. A short, funny, insightful read, as you'd expect from the authors of PeopleWare: https://www.amazon.com/Waltzing-Bears-Managing-Software-Projects/dp/0932633609 https://www.amazon.com/Waltzing-Bears-Managing-Software-Proj...
- lifeisstillgood 7y ago
  I regularly put a risk / loss / impact assessment into my issue tracker tickets - it's not the tool support is not there, it's that a) everyone else needs to do this across the board b) it's still just a guess - normalising my guess and you're guess is hard
  - jacques_chester 7y ago
    Tool support would ideally both of those problems easier. But good estimating is in itself fundamentally a "system 2" activity.
    - malenos 7y ago
      system 2: is that a reference to 'no silver bullet'?
      - elbear 7y ago
        I assume it's a reference to Kahneman's "Thinking, Fast and Slow".
        malenos 7y ago
        Thanks
  - mindcrime 7y ago
    b) it's still just a guess - normalising my guess and you're guess is hard True, but that's where the calibrated probability assessment stuff comes in. If you can at least establish obviously correct (if very course grained) upper and lower bounds for estimates and then gradually shrink them in until you aren't confident doing so anymore, you have something at least slightly better than a complete guess. And if you can choose the correct distribution that the actual values would vary over, then you can use Hubbard's approach using a Monte Carlo simulation, and get some insight into likely outcomes. No, it's not a perfect approach, but it gives you a little something to hang your hat on.
thanatropism 7y ago
I have that book. Basically I’m the last in a giveaway chain and can’t honestly recommend it enough that someone should lug it home. Next time I move it’s going on the trash. It’s really not very good, even for executives who shouldn’t care for technicalities. The best thing are the calibration exercises. But my advice is, skip this one.
- qznc 7y ago
  Can you elaborate? I'm half-way through it. I know most of the general stuff already but my knowledge is from lots of sources I mostly forgot. This books seems to be a good collection for this topic. At least, I don't know any substitute.
  - thanatropism 7y ago
    It overstates its claims. An admittedly unfair "take" would be that it encourages people to pin made-up numbers and take solace in having quantified the qualitative. For example: it tells you to do Monte Carlo simulations with made-up probability distributions, but silently lets the issue of the joint distribution -- how the made-up random variables correlate. But so much risk is driven not by marginals (say, I know the physical parameters for my chair and have a certain confidence that it won't crumble like carboard) but by correlations, even apparently distant ones (there's fracking or whatever going on and destabilizing the upper layers of the Earth, increasing the chances of earthquakes). An even broader critique is the McNamara fallacy: > "The first step is to measure whatever can be easily measured. This is OK as far as it goes. The second step is to disregard that which can't be easily measured or to give it an arbitrary quantitative value. This is artificial and misleading. The third step is to presume that what can't be measured easily really isn't important. This is blindness. The fourth step is to say that what can't be easily measured really doesn't exist. This is suicide." ---- There's this book I like called "Guesstimations in the back of a napkin" or something -- it's a book of exercises in Fermi-type estimation. It encourages you to consider the whole, to think bottom-up, top-down and core-out. It preserves a keen sense of the qualitative complexity of problems even as it encourages you to, well, make wild guesses.

jacques_chester 7y ago

The relevant book for this is Measuring and Managing Information Risk: A FAIR Approach by Freund and Jones[0]. Both books are worth reading; Hubbard's influence on FAIR is noticeable and positive. FAIR has the advantage that it comes with a fairly built-out ontology for assembling data or estimates. The OP touches on the top level (Loss Event Magnitude and Loss Event Frequency), but the ontology goes quite deep and can be used at multiple levels of detail. The calculations are not difficult, I've implemented them twice in proofs-of-concept, including one that produces pretty charts. The difficult part, to be honest, is that developing good estimates is difficult and frequently uncomfortable and the gains are not easily internalised. Additionally, serious tool support is lacking in the places where it would make a difference -- issue trackers, for example. [0] https://www.amazon.com/Measuring-Managing-Information-Risk-Approach/dp/0124202314 https://www.amazon.com/Measuring-Managing-Information-Risk-A... edit -- Another good book in this area is Waltzing with Bears by DeMarco & Lister. A short, funny, insightful read, as you'd expect from the authors of PeopleWare: https://www.amazon.com/Waltzing-Bears-Managing-Software-Projects/dp/0932633609 https://www.amazon.com/Waltzing-Bears-Managing-Software-Proj...

lifeisstillgood 7y ago
I regularly put a risk / loss / impact assessment into my issue tracker tickets - it's not the tool support is not there, it's that a) everyone else needs to do this across the board b) it's still just a guess - normalising my guess and you're guess is hard
- jacques_chester 7y ago
  Tool support would ideally both of those problems easier. But good estimating is in itself fundamentally a "system 2" activity.
  - malenos 7y ago
    system 2: is that a reference to 'no silver bullet'?
    - elbear 7y ago
      I assume it's a reference to Kahneman's "Thinking, Fast and Slow".
      - malenos 7y ago
        Thanks
- mindcrime 7y ago
  b) it's still just a guess - normalising my guess and you're guess is hard True, but that's where the calibrated probability assessment stuff comes in. If you can at least establish obviously correct (if very course grained) upper and lower bounds for estimates and then gradually shrink them in until you aren't confident doing so anymore, you have something at least slightly better than a complete guess. And if you can choose the correct distribution that the actual values would vary over, then you can use Hubbard's approach using a Monte Carlo simulation, and get some insight into likely outcomes. No, it's not a perfect approach, but it gives you a little something to hang your hat on.

thanatropism 7y ago

I have that book. Basically I’m the last in a giveaway chain and can’t honestly recommend it enough that someone should lug it home. Next time I move it’s going on the trash. It’s really not very good, even for executives who shouldn’t care for technicalities. The best thing are the calibration exercises. But my advice is, skip this one.

qznc 7y ago
Can you elaborate? I'm half-way through it. I know most of the general stuff already but my knowledge is from lots of sources I mostly forgot. This books seems to be a good collection for this topic. At least, I don't know any substitute.
- thanatropism 7y ago
  It overstates its claims. An admittedly unfair "take" would be that it encourages people to pin made-up numbers and take solace in having quantified the qualitative. For example: it tells you to do Monte Carlo simulations with made-up probability distributions, but silently lets the issue of the joint distribution -- how the made-up random variables correlate. But so much risk is driven not by marginals (say, I know the physical parameters for my chair and have a certain confidence that it won't crumble like carboard) but by correlations, even apparently distant ones (there's fracking or whatever going on and destabilizing the upper layers of the Earth, increasing the chances of earthquakes). An even broader critique is the McNamara fallacy: > "The first step is to measure whatever can be easily measured. This is OK as far as it goes. The second step is to disregard that which can't be easily measured or to give it an arbitrary quantitative value. This is artificial and misleading. The third step is to presume that what can't be measured easily really isn't important. This is blindness. The fourth step is to say that what can't be easily measured really doesn't exist. This is suicide." ---- There's this book I like called "Guesstimations in the back of a napkin" or something -- it's a book of exercises in Fermi-type estimation. It encourages you to consider the whole, to think bottom-up, top-down and core-out. It preserves a keen sense of the qualitative complexity of problems even as it encourages you to, well, make wild guesses.