8 ms·
Open-sourcing Riskquant, a library for quantifying risk
- fancyfredbot 7y agoIt doesnt seem to have an input for correlation between the loss scenarios? That would affect expected loss a lot.
- rubyfan 7y agoThis seems like applying insurance modeling to security. Is this a new way of looking at risk or a reinvention?
- im_down_w_otp 7y agoLooks like parts of FMEA w/ some specific measurements to inform occurance & severity.
- Eridrus 7y agoThis is part of an attempt to make information security risk modelling more quantitative that has been going on for a few years. There's very little in the way of data to really back most of this up, but actually putting numbers to things is significant progress IMO.
- rubyfan 7y agoAgree it seems like a better way to inform decisions and manage risk. Is it backed by any sort of real understanding of litigation, settlement, statutory experience? I often hear reputational risk cited by security teams at public companies... it usually follows some sort of indecision or appeal to a higher level of management or beuqacratic tool.
- Eridrus 7y agoNot really. But it's significant progress compared to sticking charts like this into reports https://www.shipownersclub.com/media/2017/02/Risk-matrix-1-768x386.png https://www.shipownersclub.com/media/2017/02/Risk-matrix-1-7... I think it's also worth saying that when people say reputational risk they're generally not thinking about litigation, they're thinking about consumer perception, which it's not crazy to believe would have large impacts for large companies.
- TeMPOraL 7y agoNeither; the idea has been around for many years now (source: worked professionally on exactly these calculations this library does, in the same space). It's just basic probability theory and Monte Carlo simulations; nothing anywhere near as complex as the quants/insurance folks do. IME, the biggest problem in this space is: on the one side, you have all kinds of metrics related to hardware, software and operations in your company; on the other side, you have FAIR and related ideas that let you model scenarios in a sane way, and let you plug in those statistical methods. But, there's no good idea how to connect these two sides. There is a vast gulf between what you can reliably and accurately measure, and what you actually want to know. It's hard to cross it while maintaining any kind of accuracy or statistical validity. I've been doing work on this "middle layer", but never got the chance to test it out in the open, as the company I worked for suddenly imploded. (If anyone is working in this space and is accepting remote workers, I'm available for a chat.)
- leecb 7y agoRequires registration to read. Tab closed.
- muglug 7y agoIf the authors are reading, I think you mistakenly marked this as a paywalled article. I can't believe this was done deliberately from someone on the Netflix side.
- thenightcrawler 7y agoNassim Nicholas Taleb is enraged by this!
- rq1 7y agoWhy would he be? I suppose it is tempting, if the only tool you have is a hammer, to treat everything as if it were a nail.
- syndacks 7y ago>Why would he be? Read Black Swan, by NNT. His thesis is that the most significant impacts on an individual and/or a collective are the function of outliers or "Black Swans". In a Malcolm Gladwell kind of way, he uses a bunch of anecdata to prove his thesis. It's a pretty good book I suppose, but it was also before the all the faux-philosophy Malcolm Gladwell type books became a thing. >I suppose it is tempting, if the only tool you have is a hammer, to treat everything as if it were a nail. What exactly do you mean by that? In all sincerity, I think you misused an idiom or at least didn't make a clear connection as to why you chose that one.
- tajd 7y ago> In a Malcolm Gladwell kind of way, he uses a bunch of anecdata to prove his thesis. In my opinion this is perhaps the best 1 line description of his work in general. I think there are some interesting ideas which are worth discussing, but sometimes his attitude and use of "anecdata" tires me out.
- DyslexicAtheist 7y agoloved all his books. he literally captures the problem with this approach in a single "book title" (3 words total) https://en.wikipedia.org/wiki/Fooled_by_Randomness https://en.wikipedia.org/wiki/Fooled_by_Randomness
- deleted 7y ago[deleted]
- batterseapower 7y agoIf you like the idea of this library you'll probably like the book "How to Measure Anything" by Douglas Hubbard (https://www.goodreads.com/book/show/20933591-how-to-measure-anything https://www.goodreads.com/book/show/20933591-how-to-measure-...). It's all about how to get sensible confidence intervals for things that are often considered unmeasurable such as the value of IT security. The book mostly uses Excel to do this modelling, but it looks like riskquant would be an excellent alternative on that approach, that for the more technically minded practitioner.
- jacques_chester 7y agoThe relevant book for this is Measuring and Managing Information Risk: A FAIR Approach by Freund and Jones[0]. Both books are worth reading; Hubbard's influence on FAIR is noticeable and positive. FAIR has the advantage that it comes with a fairly built-out ontology for assembling data or estimates. The OP touches on the top level (Loss Event Magnitude and Loss Event Frequency), but the ontology goes quite deep and can be used at multiple levels of detail. The calculations are not difficult, I've implemented them twice in proofs-of-concept, including one that produces pretty charts. The difficult part, to be honest, is that developing good estimates is difficult and frequently uncomfortable and the gains are not easily internalised. Additionally, serious tool support is lacking in the places where it would make a difference -- issue trackers, for example. [0] https://www.amazon.com/Measuring-Managing-Information-Risk-Approach/dp/0124202314 https://www.amazon.com/Measuring-Managing-Information-Risk-A... edit -- Another good book in this area is Waltzing with Bears by DeMarco & Lister. A short, funny, insightful read, as you'd expect from the authors of PeopleWare: https://www.amazon.com/Waltzing-Bears-Managing-Software-Projects/dp/0932633609 https://www.amazon.com/Waltzing-Bears-Managing-Software-Proj...
- lifeisstillgood 7y agoI regularly put a risk / loss / impact assessment into my issue tracker tickets - it's not the tool support is not there, it's that a) everyone else needs to do this across the board b) it's still just a guess - normalising my guess and you're guess is hard
- caseyf7 7y agoI wish there were detailed scenarios of how they were using it.
- Dowwie 7y agoFocus on the stress tests and worst case scenarios, Netflix.
- Jugurtha 7y agoI suppose this is the impetus. I think that this gives them a way to prioritize which issues to fix in a better than the arbitrary hunch way. We assign "weights" all the time to issues either based on expected "time to complete" or on the severity of not fixing this. However, someone's severe is another's meh. This could give a score in a systematic way and help alleviate the guesswork.
- Dowwie 7y agoRisk is managed differently in places that have experienced actual loss. No need to try to educate me about weighted decision making.
- Jugurtha 7y agoI was not trying to "educate" you, not that I see anything wrong being educated. I was brainstorming with you, hence the "I think" and "I suppose". I exposed my thoughts in writing. Your reply could have quoted my reply, with a comment that said "maybe, but [insert counter argument, or poke holes in my statement's logic]".
- sheerun 7y agoCan someone explain to me why Netflix puts their posts behind a paywall?
- nurettin 7y agoIt looks as if they just registered for medium.com and slapped their domain on it.
- pps43 7y agoNot clear why the losses should follow lognormal distribution rather than some less pleasant to work with distribution like Cauchy. Also not clear why loss frequency is assumed to be independent from loss magnitude. Instead of "percent chance that the loss will exceed X" (VaR), expected shortfall would be more useful.
- jacques_chester 7y agoThe distribution comes from PERT 3-point estimates. LEF and LEM are built of lower factors that roll up, which to my mind at least seem independent. I'm not sure what you meant by expected shortfall.
- bernardv 7y agoThe choice of distribution is subjective, especially if you little data to start with and are going on theoretical grounds. Agree on expected shortfall/TVaR.
- juskrey 7y agoYou can quantify in normal domains e.g. probability of seeing a human over 2.05m tall. You CAN'T quantify tail risk in unbounded (for practical purposes) domains, period. This is just ridiculous: "For this example, there’s about a 2% chance losses would exceed $60 million in a year." If you see you can lose everything, you are not quantifying, you are simply not going that way. If you don't know if you can protect yourself from losing everything, you are not going that way. If you can afford losing some limited amount you just write down the loss from the very start. That is that simple. Netflix is just going straight to hell with such prediction approaches. And, mind that, this is not a prediction, this is a fragility statement.
- bernardv 7y agoOh yes you can. The insurance and reinsurance actuaries, and regulators, have done this for a very long time.
- juskrey 7y agoInsurance and reinsurance just clip the tail contractually, they never take "infinity multiplied by some small percent" risk! They don't go underground when their client does.
- Chris2048 7y ago> they never take "infinity multiplied by some small percent" risk At what point does the tail represent "all humanity dead"? No one needs a contract for the humanly unrecoverable.
- maxymajzr 7y agoI see an interesting library, risk quantification (relates to what I work with). I click the link. Sniff/Accountwall stops me. I hit ctrl+w before I even thought of doing it (yes for muscle memory!) For people who went through the trouble to see the content, is the library worth it? These days, anything behind paywall/sniffwall/must-be-authenticated-to-read-this-wall warrants a ctr + w from me.
- Chris2048 7y agoI feel "riskquant" is a little vague for a project name. It's like there are a million internal libraries called "riskLib" - "risk" is a very general concept!
- veeralpatel979 7y agoLink to the GitHub repo: https://github.com/Netflix-Skunkworks/riskquant https://github.com/Netflix-Skunkworks/riskquant
- veeralpatel979 7y agoI chatted with Markus de Shon, one of the creators of riskquant, a few days ago. I would also check out Ryan McGeehan's writings on this at scrty.io. I'd be happy to discuss this topic with anyone; my email address is in my profile.