3 ms·

> 1. Both models I’m interested in cost $2000. I’m worried that there’s a 20% chance I’ll end up with a camera that I never use because I don’t like it, and the

> 1. Both models I’m interested in cost $2000. I’m worried that there’s a 20% chance I’ll end up with a camera that I never use because I don’t like it, and therefore have a $2000 device which is essentially useless to me. I value my spare time at roughly $25/hour. On financial reasons alone, how much time could I reasonably spend on doing research? > ~16 hours. Because 20% × $2000 / ($25/hour). I'm not sure if I have understood, I think in this example, the expected loss here is, 0 x 80% + -2000 * 20% = -400, thus the answer, is my understanding correct?

gwern 8y ago
He's not being clear about what he's talking about and mish-mashing stuff from Bayesian decision theory (https://www.reddit.com/r/DecisionTheory/ https://www.reddit.com/r/DecisionTheory/). The calculation he seems to be trying to do there is 'Expected Value of Perfect Information' (EVPI): the value of omniscience. Which is an upper bound, not an exact amount. He also is unclear about the choice being involved (is he choosing between buying a camera and not buying a camera, or buying camera A versus buying camera B? If camera A fails, why expect camera B to succeed? Couldn't they both fail?) If you interpret it as buying a camera or not buying a camera, a 20% risk of the camera being useless, then the EVPI is 20% * $2000 = $200: you would be willing to pay up to $200 for certainty about whether the camera would be good or not. The EVPI serves as an upper bound on more realistic Value of Information quantities like Expected Value of Sample Information (EVSI; the value from a reduction of posterior uncertainty by a sample of data eg a small survey). If, for example, there were some piece of information you could buy for <$100 which reduced the risk to 10%, you would want to buy it; if you could do that by doing research for <4 hours and you value your time at $25, then it would be profitable to do that research. Similarly, if you could somehow reach certainty, you would need to do so at under 16 hours (not exactly 16 hours!) #2 is also wrong because it ignores discounting/opportunity cost (imagine the integration cost could be $499,999 instead). #3 can't be answered because it omits the side-effects and magnitude of the gain, which affect both the net benefit and EVSI (smaller differences are less profitable and harder to detect) and 95% 'confidence' does not mean 95% probability the drug is better, which is the fallacy of inverting the p-value. (For a good paper on how you'd actually do drug approval in an decision-theoretic way, see https://papers.ssrn.com/sol3/papers.cfm?abstract_id=2641547 https://papers.ssrn.com/sol3/papers.cfm?abstract_id=2641547 ) Not to mention again discounting, and elasticity of demand to boot.
- bcaa7f3a8bbc 8y ago
  So you think it's just some half-cooked Bayesian? Perhaps I should put "systematically indoctrinate myself with the LessWrong methodology" to my agenda, seriously.
  - gwern 8y ago
    I think it definitely is and he's either explaining it poorly or reinventing the wheel. Try applying any of his explanations to something more complicated than binary data with perfect information, such as a normal distribution with small sample _n_, it falls apart immediately - you need the full Bayesian perspective in order to work with the posterior and apply a loss function sensibly. (I wouldn't call it 'LessWrong methodology', though. LWers pretty much never do a formal true subjective Bayesian decision analysis, and to the extent they do, you should be calling it LessWrong indoctrinating itself into the 'Raiffa-Schlaifer-Savage methodology', as they preceded us by ~60 years.)

gwern 8y ago

He's not being clear about what he's talking about and mish-mashing stuff from Bayesian decision theory (https://www.reddit.com/r/DecisionTheory/ https://www.reddit.com/r/DecisionTheory/). The calculation he seems to be trying to do there is 'Expected Value of Perfect Information' (EVPI): the value of omniscience. Which is an upper bound, not an exact amount. He also is unclear about the choice being involved (is he choosing between buying a camera and not buying a camera, or buying camera A versus buying camera B? If camera A fails, why expect camera B to succeed? Couldn't they both fail?) If you interpret it as buying a camera or not buying a camera, a 20% risk of the camera being useless, then the EVPI is 20% * $2000 = $200: you would be willing to pay up to $200 for certainty about whether the camera would be good or not. The EVPI serves as an upper bound on more realistic Value of Information quantities like Expected Value of Sample Information (EVSI; the value from a reduction of posterior uncertainty by a sample of data eg a small survey). If, for example, there were some piece of information you could buy for <$100 which reduced the risk to 10%, you would want to buy it; if you could do that by doing research for <4 hours and you value your time at $25, then it would be profitable to do that research. Similarly, if you could somehow reach certainty, you would need to do so at under 16 hours (not exactly 16 hours!) #2 is also wrong because it ignores discounting/opportunity cost (imagine the integration cost could be $499,999 instead). #3 can't be answered because it omits the side-effects and magnitude of the gain, which affect both the net benefit and EVSI (smaller differences are less profitable and harder to detect) and 95% 'confidence' does not mean 95% probability the drug is better, which is the fallacy of inverting the p-value. (For a good paper on how you'd actually do drug approval in an decision-theoretic way, see https://papers.ssrn.com/sol3/papers.cfm?abstract_id=2641547 https://papers.ssrn.com/sol3/papers.cfm?abstract_id=2641547 ) Not to mention again discounting, and elasticity of demand to boot.

bcaa7f3a8bbc 8y ago
So you think it's just some half-cooked Bayesian? Perhaps I should put "systematically indoctrinate myself with the LessWrong methodology" to my agenda, seriously.
- gwern 8y ago
  I think it definitely is and he's either explaining it poorly or reinventing the wheel. Try applying any of his explanations to something more complicated than binary data with perfect information, such as a normal distribution with small sample _n_, it falls apart immediately - you need the full Bayesian perspective in order to work with the posterior and apply a loss function sensibly. (I wouldn't call it 'LessWrong methodology', though. LWers pretty much never do a formal true subjective Bayesian decision analysis, and to the extent they do, you should be calling it LessWrong indoctrinating itself into the 'Raiffa-Schlaifer-Savage methodology', as they preceded us by ~60 years.)