7 ms·

I desperately want to get to a point where I'm able to quickly sketch out a statistical model for making a decision, but I'm struggling mightily on where to get

I desperately want to get to a point where I'm able to quickly sketch out a statistical model for making a decision, but I'm struggling mightily on where to get started that doesn't include some kind of "hide the uncertainty" shell game, such as how I feel when Bayesianism gets thrown around. Submissions like this give me hope, that if I just read more, I'll get there, but nothing listed here feels like it's usable in a "napkin math" scenario. I just bought two of Gerd Gigerenzer's latest books, and I've already obtained some of David Sklansky's more introductory work on gambling, but I'm concerned my focus is either too theoretical or too narrowly on gambling (though I think there's a lot to learn in how a gambler assesses a bet, and how it applies to other situations). I guess if it were easy, we'd all be doing it...

Arnehuang 6y ago
Is that statistics? I think of statistics as "given historical data, infer future data". But it seems like what you want is to know which decision is best, which involves many more things (like estimating the impact/utility of each outcome), that seems more like economics?
- asdff 6y ago
  Economics is also statistics.
  - Arnehuang 6y ago
    I think of it like this: Suppose I want to make a decision about whether to hedge for a market crash right now. Statistics can tell me the likelihood of a crash, and how bad. But if the market crashes, and very badly, how might that affect my life? To make a good decision I would need to think of all the things that come with a market crash (job loss, savings loss). This is not statistics. I could again use statistics to say what is the chance I lose my job given a market crash (say 70%). But then I would need to estimate the impact on my life should I lose my job (Stress, etc). This is not statistics. But it should very well factor into my ability to do back of the napkin math on whether I should hedge or not.
    - chewbaxxa 6y ago
      This is exactly statistics. This is an expectation of a utility function with respect to some distribution.
    - fractionalhare 6y ago
      If your decision substantially involves or derives from making an estimate about a population based on a sample, it is statistics. "Making decisions under uncertainty" is well-studied in statistical literature, just like "quantifying uncertainty" is well-studied. It sounds like you think the latter is "actual statistics", but these things are both statistics. In particular: > But if the market crashes, and very badly, how might that affect my life? To make a good decision I would need to think of all the things that come with a market crash (job loss, savings loss). This is not statistics. This is all statistics, not just the part where you're forecasting likelihood of the market crashing. The reason is because making decisions about the future under the constraints of uncertainty implicitly involves a forecast. When you decide how to diversify your personal investment portfolio, how much to allocate to your Roth versus traditional IRA or 401k, etc, you are making forecasts about which allocation will provide you with a more favorable outcome. Stated more concisely: there is no rational reason to use statistics for forecasting market events but not for deciding what to do in the event specific market events occur.
    - nanis 6y ago
      > Statistics can tell me the likelihood of a crash Statistics cannot tell you any such thing.
      - drdeca 6y ago
        Do you mean to say that nothing can tell you such a thing? What is a likelihood, but a statistic? If there is any method to determine a statistic, it seems reasonable to me to say that that method involved statistics. (Now, of course, except for possibly where quantum randomness is relevant, which might be quite often, I'm fairly confident that the only probabilities are subjective or relative to some set of assumptions, or something along those lines, because the future "already exists". But, given some fixed priors and some fixed evidence, there should in principle be a well defined probability of such a crash. So, insofar as peoples priors match up, there should, in principle, be a common well defined probability given "the information which is publicly available", or also, given whatever other set of evidence.) Of course, that doesn't mean it is computationally tractable to compute such a probability.
  - nanis 6y ago
    > Economics is also statistics. Economics is not Statistics (and definitely not statistics). Most of the discipline focuses on testing models and making inferences on observational data. The techniques for dealing with that sort of data, of course, build on Statistics, but their nature is different enough that there is Econometrics. A large part of economics is not empirical at all -- despite the fact that people get Nobel prizes pretending this not to be the case. Even in the context of experimental economics, since the behavior of the observed vary depending on the mode of observation, the contexts in which the most straightforward Statistical methods designed to apply to engineering/chemistry/biology experiment type situations are not directly applicable (although it is great when they agree with the fancier methods).
    - huitzitziltzin 6y ago
      >A large part of economics is not empirical at all -- despite the fact that people get Nobel prizes pretending this not to be the case. I'm not sure which parts of the field or which prize winners you are talking about. To be clear: you think economics is _not actually empirical_, but people are awarded Nobel Prizes for _pretending that it is_? That's a little odd. Let me know if that's not what you meant. When you look at this list: https://en.wikipedia.org/wiki/List_of_Nobel_Memorial_Prize_laureates_in_Economics https://en.wikipedia.org/wiki/List_of_Nobel_Memorial_Prize_l... Who satisfies that condition, in your mind? Who is getting the prize on the basis of pretending that economics is empirical?
      - nanis 6y ago
        > To be clear: you think economics is _not actually empirical_ That is a misrepresentation of what I said. To be clear, I think what I said: >>A large part of economics is not empirical at all E.g., as an example, Kahneman's Nobel is solely a product of taking an axiomatic theory and designing experiments where regular people who are actually not being paid according to their performance are gently prodded into violating the axioms in weird settings. It is attractive to people who want to claim that clearly the plebes cannot be allowed to choose for themselves as they are not "rational". The only meaning of "rational" in Economics is that individuals choose the best alternative according to their preferences among a constrained set of alternatives. Here an "alternative" or "bundle" is a point in the entire commodity space. The only test of this is consistency with GARP: A choice is not rational if a feasible and more preferred alternative exists.
  - deleted 6y ago
    [deleted]
- mindcrime 6y ago
  Arguably it's just quibbling over a trivial terminological difference, but I get the feeling that you're thinking more about "Decision Theory"[1] (or "Decision Science") as opposed to "just Statistics". Decision Theory, of course, uses Statistics, and I guess one could argue the question of whether one is just a subfield of the other, or argue exactly where the dividing line is. [1]: https://en.wikipedia.org/wiki/Decision_theory https://en.wikipedia.org/wiki/Decision_theory
- analog31 6y ago
  When I took statistics in college, we started with a rather basic definition, something to the effect of: "A statistic is a function performed on a set." Statistics studies what you can infer when you know something about a set, but not everything about it, namely its precise contents. Often, what you are told about a set is something about its probability distribution, thus linking probability and statistics together. A useful parallel can be drawn with situations involving measurements and data, since data often have the same feature of telling us something but not everything. This is what I believe makes statistics so useful for science.
- taeric 6y ago
  I've grown to the idea that statistics is a form of data compression. It isn't so much, "infer future data" as it is "if the data we have is representative of all data, what is a number/equation that represents this data?". Usually with a certain framing.
asdff 6y ago
Frame your hypothesis in the most simplest way possible and go from there.
chewbaxxa 6y ago
What do you mean by hide the uncertainty? To me Bayesian modelling is about making explicit the uncertainty.
cf 6y ago
While I'm not sure how much the modern methods are amendable to napkin math as stated in the article a lot more methods use simulation which if you can code are pretty straightforward to get working. Jake Vanderplas's presentation https://speakerdeck.com/jakevdp/statistics-for-hackers https://speakerdeck.com/jakevdp/statistics-for-hackers can give you some concrete ideas of how far you can get with just a random number generator.
mindcrime 6y ago
Depending on the context, I'm a fan of the work of Douglas Hubbard, in his book How to Measure Anything[1]. His approach involves working out answers to things that might sometimes be done as a "back of the napkin" kind of thing, but in a slightly more rigorous way. Note that there are criticisms of his approach, and I'll freely admit that it doesn't guarantee arriving at an optimal answer. But arguably the criticisms of his approach ("what if you leave out a variable in your model?", etc.) apply to many (most?) other modeling approaches. On a related note, one of the last times I mentioned Hubbard here, another book came up in the surrounding discussion, which looks really good as well. Guesstimation: Solving the World's Problems on the Back of a Cocktail Napkin[2] - I bought a copy but haven't had time to read it yet. Maybe somebody who is familiar will chime in with their thoughts? [1]: https://www.amazon.com/How-Measure-Anything-Intangibles-Business/dp/1118539273 https://www.amazon.com/How-Measure-Anything-Intangibles-Busi... [2]: https://www.amazon.com/gp/product/0691129495/ref=ppx_yo_dt_b_search_asin_title?ie=UTF8&psc=1 https://www.amazon.com/gp/product/0691129495/ref=ppx_yo_dt_b...
- dr_dshiv 6y ago
  Let me second "How to measure anything." I think it should be required reading for human beings.
  - alex_anglin 6y ago
    I would add that "How the Measure Anything in Cybersecurity Risk" should be a core part of Infosec literature.
- ingqondo 6y ago
  How to Measure Anything is a fantastic book. Here are the most significant insights you learn in the book - how to measure anything; Hubbard actually comes through on the promise of the title - after finishing the book you will truly feel that the scope of what you can measure is massive. He does this by a change in the definition of what it means to measure something, but you realize his definition is more correct than the everyday intuitive one. - value of information; Hubbard gives a good introduction to the VOI concept in economics, which basically lets you put a price on any measurement or information and prioritize what to measure - motivation for 'back of the napkin' calcs; through his broad experience he has seen how a lot of the most important things that affect a business go unmeasured, and how his approach to 'measuring anything' can empower people to really measure what matters. Reading this book provided one half of what I have been searching for for a long time - a framework for thinking about data science activities which is not based on hype, fundamentally correct and still intuitive and practical.
platz 6y ago
Decisions are moral/political/biological, not statistical
- roenxi 6y ago
  Typically. But if somebody manages to make statistically sound evidence-based decisions they will steamroller people who are making decisions using other factors.
deleted 6y ago
[deleted]
hntrader 6y ago
A common mistake technical people make is to be too theoretical or overcomplicated with their work and decision making, where everything has to be some math model written out in a nicely formatted LaTeX document. Don't fall into that trap, it is very ineffective. Stats is a tool like any other tool. Boil your question down to the fundamentals (first principles) and maybe stats is one of the tools you decide to use to solve part of it, where appropriate. Most questions involving data can be answered with something as basic as a plot.
- cambalache 6y ago
  Easier said than done. The complete history of science is the human struggle to boil down the mathematical modelling of nature to its fundamentals.
  - hntrader 6y ago
    "mathematical modelling of nature" My point is most real world problems people face (even most problems that data scientists face at work) shouldn't be modelled with formal math. Basic reasoning, tinkering, plotting data and playing around with data in Excel or Python is usually sufficient to answer most questions, yet many will try to overcomplicate the issue with complex math or stats. Perhaps in an attempt to impress their peers, or perhaps because they've just come out of years of university that taught them to think theoretically. I'm reminded of that meme going around. 70 IQ: plot data 100 IQ: support vector machines! deep Q-nets! 130 IQ: plot data
jbay808 6y ago
I start with Wikipedia's list of maximum entropy distributions, pick the one that best represents the category for the kind of thing I'm reasoning about, and then update it with what data I happen to have. It's usually simpler than it sounds, and helped me put together a lot of quick decision models, sales forecasts, etc. https://en.m.wikipedia.org/wiki/Maximum_entropy_probability_distribution#Other_examples https://en.m.wikipedia.org/wiki/Maximum_entropy_probability_...
Karrot_Kream 6y ago
> I desperately want to get to a point where I'm able to quickly sketch out a statistical model for making a decision, but I'm struggling mightily on where to get started that doesn't include some kind of "hide the uncertainty" shell game, such as how I feel when Bayesianism gets thrown around. Being comfortable around basic probability distributions is probably the main prerequisite I suggest. If you can understand the uses and analyses of Bernoulli, Binomial, Poisson, and normal RVs, you should be at a point where you can try to model real life phenomena. A surprising amount of life can be modeled as multiple weighted coin flips.
stevegalla 6y ago
> I desperately want to get to a point where I'm able to quickly sketch out a statistical model for making a decision What kind of decisions do you want to make? If you offer that up, others will be able to provide guidance to frameworks used to make those decisions. The field of operations research uses topics from all sorts of fields to try to make better decisions. I would broadly classify modelling into two large categories deterministic (e.g., how to schedule a flight crew, or where to locate a warehouse) or stochastic (using probability models, e.g., how many people to staff in a bank). Of course there is no problem that will be strictly one form or the other. There will be points in a modelling problem where it's useful to apply tools from both categories. A decision tree might be the simplest form. You map out the different decision points and can generate costs or profits and probabilities for the decision points. You take the one that has the highest expected value (sum of probability of outcome times value of outcome) of profit or lowest expected value of cost. Sensitivity analysis may be another tool to use in decision making. You don't need to get three decimal accuracy. Come up with some upper and lower bounds and you'll have an idea of how good or how bad the outcome will be. Determining the probabilities and bounds may be tough, and your decision will only be as good as the data your putting into the framework.

Arnehuang 6y ago

Is that statistics? I think of statistics as "given historical data, infer future data". But it seems like what you want is to know which decision is best, which involves many more things (like estimating the impact/utility of each outcome), that seems more like economics?

asdff 6y ago
Economics is also statistics.
- Arnehuang 6y ago
  I think of it like this: Suppose I want to make a decision about whether to hedge for a market crash right now. Statistics can tell me the likelihood of a crash, and how bad. But if the market crashes, and very badly, how might that affect my life? To make a good decision I would need to think of all the things that come with a market crash (job loss, savings loss). This is not statistics. I could again use statistics to say what is the chance I lose my job given a market crash (say 70%). But then I would need to estimate the impact on my life should I lose my job (Stress, etc). This is not statistics. But it should very well factor into my ability to do back of the napkin math on whether I should hedge or not.
  - chewbaxxa 6y ago
    This is exactly statistics. This is an expectation of a utility function with respect to some distribution.
  - fractionalhare 6y ago
    If your decision substantially involves or derives from making an estimate about a population based on a sample, it is statistics. "Making decisions under uncertainty" is well-studied in statistical literature, just like "quantifying uncertainty" is well-studied. It sounds like you think the latter is "actual statistics", but these things are both statistics. In particular: > But if the market crashes, and very badly, how might that affect my life? To make a good decision I would need to think of all the things that come with a market crash (job loss, savings loss). This is not statistics. This is all statistics, not just the part where you're forecasting likelihood of the market crashing. The reason is because making decisions about the future under the constraints of uncertainty implicitly involves a forecast. When you decide how to diversify your personal investment portfolio, how much to allocate to your Roth versus traditional IRA or 401k, etc, you are making forecasts about which allocation will provide you with a more favorable outcome. Stated more concisely: there is no rational reason to use statistics for forecasting market events but not for deciding what to do in the event specific market events occur.
  - nanis 6y ago
    > Statistics can tell me the likelihood of a crash Statistics cannot tell you any such thing.
    - drdeca 6y ago
      Do you mean to say that nothing can tell you such a thing? What is a likelihood, but a statistic? If there is any method to determine a statistic, it seems reasonable to me to say that that method involved statistics. (Now, of course, except for possibly where quantum randomness is relevant, which might be quite often, I'm fairly confident that the only probabilities are subjective or relative to some set of assumptions, or something along those lines, because the future "already exists". But, given some fixed priors and some fixed evidence, there should in principle be a well defined probability of such a crash. So, insofar as peoples priors match up, there should, in principle, be a common well defined probability given "the information which is publicly available", or also, given whatever other set of evidence.) Of course, that doesn't mean it is computationally tractable to compute such a probability.
- nanis 6y ago
  > Economics is also statistics. Economics is not Statistics (and definitely not statistics). Most of the discipline focuses on testing models and making inferences on observational data. The techniques for dealing with that sort of data, of course, build on Statistics, but their nature is different enough that there is Econometrics. A large part of economics is not empirical at all -- despite the fact that people get Nobel prizes pretending this not to be the case. Even in the context of experimental economics, since the behavior of the observed vary depending on the mode of observation, the contexts in which the most straightforward Statistical methods designed to apply to engineering/chemistry/biology experiment type situations are not directly applicable (although it is great when they agree with the fancier methods).
  - huitzitziltzin 6y ago
    >A large part of economics is not empirical at all -- despite the fact that people get Nobel prizes pretending this not to be the case. I'm not sure which parts of the field or which prize winners you are talking about. To be clear: you think economics is _not actually empirical_, but people are awarded Nobel Prizes for _pretending that it is_? That's a little odd. Let me know if that's not what you meant. When you look at this list: https://en.wikipedia.org/wiki/List_of_Nobel_Memorial_Prize_laureates_in_Economics https://en.wikipedia.org/wiki/List_of_Nobel_Memorial_Prize_l... Who satisfies that condition, in your mind? Who is getting the prize on the basis of pretending that economics is empirical?
    - nanis 6y ago
      > To be clear: you think economics is _not actually empirical_ That is a misrepresentation of what I said. To be clear, I think what I said: >>A large part of economics is not empirical at all E.g., as an example, Kahneman's Nobel is solely a product of taking an axiomatic theory and designing experiments where regular people who are actually not being paid according to their performance are gently prodded into violating the axioms in weird settings. It is attractive to people who want to claim that clearly the plebes cannot be allowed to choose for themselves as they are not "rational". The only meaning of "rational" in Economics is that individuals choose the best alternative according to their preferences among a constrained set of alternatives. Here an "alternative" or "bundle" is a point in the entire commodity space. The only test of this is consistency with GARP: A choice is not rational if a feasible and more preferred alternative exists.
- deleted 6y ago
  [deleted]
mindcrime 6y ago
Arguably it's just quibbling over a trivial terminological difference, but I get the feeling that you're thinking more about "Decision Theory"[1] (or "Decision Science") as opposed to "just Statistics". Decision Theory, of course, uses Statistics, and I guess one could argue the question of whether one is just a subfield of the other, or argue exactly where the dividing line is. [1]: https://en.wikipedia.org/wiki/Decision_theory https://en.wikipedia.org/wiki/Decision_theory
analog31 6y ago
When I took statistics in college, we started with a rather basic definition, something to the effect of: "A statistic is a function performed on a set." Statistics studies what you can infer when you know something about a set, but not everything about it, namely its precise contents. Often, what you are told about a set is something about its probability distribution, thus linking probability and statistics together. A useful parallel can be drawn with situations involving measurements and data, since data often have the same feature of telling us something but not everything. This is what I believe makes statistics so useful for science.
taeric 6y ago
I've grown to the idea that statistics is a form of data compression. It isn't so much, "infer future data" as it is "if the data we have is representative of all data, what is a number/equation that represents this data?". Usually with a certain framing.

asdff 6y ago

Frame your hypothesis in the most simplest way possible and go from there.

chewbaxxa 6y ago

What do you mean by hide the uncertainty? To me Bayesian modelling is about making explicit the uncertainty.

cf 6y ago

While I'm not sure how much the modern methods are amendable to napkin math as stated in the article a lot more methods use simulation which if you can code are pretty straightforward to get working. Jake Vanderplas's presentation https://speakerdeck.com/jakevdp/statistics-for-hackers https://speakerdeck.com/jakevdp/statistics-for-hackers can give you some concrete ideas of how far you can get with just a random number generator.

mindcrime 6y ago

Depending on the context, I'm a fan of the work of Douglas Hubbard, in his book How to Measure Anything[1]. His approach involves working out answers to things that might sometimes be done as a "back of the napkin" kind of thing, but in a slightly more rigorous way. Note that there are criticisms of his approach, and I'll freely admit that it doesn't guarantee arriving at an optimal answer. But arguably the criticisms of his approach ("what if you leave out a variable in your model?", etc.) apply to many (most?) other modeling approaches. On a related note, one of the last times I mentioned Hubbard here, another book came up in the surrounding discussion, which looks really good as well. Guesstimation: Solving the World's Problems on the Back of a Cocktail Napkin[2] - I bought a copy but haven't had time to read it yet. Maybe somebody who is familiar will chime in with their thoughts? [1]: https://www.amazon.com/How-Measure-Anything-Intangibles-Business/dp/1118539273 https://www.amazon.com/How-Measure-Anything-Intangibles-Busi... [2]: https://www.amazon.com/gp/product/0691129495/ref=ppx_yo_dt_b_search_asin_title?ie=UTF8&psc=1 https://www.amazon.com/gp/product/0691129495/ref=ppx_yo_dt_b...

dr_dshiv 6y ago
Let me second "How to measure anything." I think it should be required reading for human beings.
- alex_anglin 6y ago
  I would add that "How the Measure Anything in Cybersecurity Risk" should be a core part of Infosec literature.
ingqondo 6y ago
How to Measure Anything is a fantastic book. Here are the most significant insights you learn in the book - how to measure anything; Hubbard actually comes through on the promise of the title - after finishing the book you will truly feel that the scope of what you can measure is massive. He does this by a change in the definition of what it means to measure something, but you realize his definition is more correct than the everyday intuitive one. - value of information; Hubbard gives a good introduction to the VOI concept in economics, which basically lets you put a price on any measurement or information and prioritize what to measure - motivation for 'back of the napkin' calcs; through his broad experience he has seen how a lot of the most important things that affect a business go unmeasured, and how his approach to 'measuring anything' can empower people to really measure what matters. Reading this book provided one half of what I have been searching for for a long time - a framework for thinking about data science activities which is not based on hype, fundamentally correct and still intuitive and practical.

platz 6y ago

Decisions are moral/political/biological, not statistical

roenxi 6y ago
Typically. But if somebody manages to make statistically sound evidence-based decisions they will steamroller people who are making decisions using other factors.

deleted 6y ago

[deleted]

hntrader 6y ago

A common mistake technical people make is to be too theoretical or overcomplicated with their work and decision making, where everything has to be some math model written out in a nicely formatted LaTeX document. Don't fall into that trap, it is very ineffective. Stats is a tool like any other tool. Boil your question down to the fundamentals (first principles) and maybe stats is one of the tools you decide to use to solve part of it, where appropriate. Most questions involving data can be answered with something as basic as a plot.

cambalache 6y ago
Easier said than done. The complete history of science is the human struggle to boil down the mathematical modelling of nature to its fundamentals.
- hntrader 6y ago
  "mathematical modelling of nature" My point is most real world problems people face (even most problems that data scientists face at work) shouldn't be modelled with formal math. Basic reasoning, tinkering, plotting data and playing around with data in Excel or Python is usually sufficient to answer most questions, yet many will try to overcomplicate the issue with complex math or stats. Perhaps in an attempt to impress their peers, or perhaps because they've just come out of years of university that taught them to think theoretically. I'm reminded of that meme going around. 70 IQ: plot data 100 IQ: support vector machines! deep Q-nets! 130 IQ: plot data

jbay808 6y ago

I start with Wikipedia's list of maximum entropy distributions, pick the one that best represents the category for the kind of thing I'm reasoning about, and then update it with what data I happen to have. It's usually simpler than it sounds, and helped me put together a lot of quick decision models, sales forecasts, etc. https://en.m.wikipedia.org/wiki/Maximum_entropy_probability_distribution#Other_examples https://en.m.wikipedia.org/wiki/Maximum_entropy_probability_...

Karrot_Kream 6y ago

> I desperately want to get to a point where I'm able to quickly sketch out a statistical model for making a decision, but I'm struggling mightily on where to get started that doesn't include some kind of "hide the uncertainty" shell game, such as how I feel when Bayesianism gets thrown around. Being comfortable around basic probability distributions is probably the main prerequisite I suggest. If you can understand the uses and analyses of Bernoulli, Binomial, Poisson, and normal RVs, you should be at a point where you can try to model real life phenomena. A surprising amount of life can be modeled as multiple weighted coin flips.

stevegalla 6y ago

> I desperately want to get to a point where I'm able to quickly sketch out a statistical model for making a decision What kind of decisions do you want to make? If you offer that up, others will be able to provide guidance to frameworks used to make those decisions. The field of operations research uses topics from all sorts of fields to try to make better decisions. I would broadly classify modelling into two large categories deterministic (e.g., how to schedule a flight crew, or where to locate a warehouse) or stochastic (using probability models, e.g., how many people to staff in a bank). Of course there is no problem that will be strictly one form or the other. There will be points in a modelling problem where it's useful to apply tools from both categories. A decision tree might be the simplest form. You map out the different decision points and can generate costs or profits and probabilities for the decision points. You take the one that has the highest expected value (sum of probability of outcome times value of outcome) of profit or lowest expected value of cost. Sensitivity analysis may be another tool to use in decision making. You don't need to get three decimal accuracy. Come up with some upper and lower bounds and you'll have an idea of how good or how bad the outcome will be. Determining the probabilities and bounds may be tough, and your decision will only be as good as the data your putting into the framework.