3 ms·

> The higher the stakes of the decision, the more incentive to approach it like a rational Bayesian agent who would use whatever tool has the best risk / reward

> The higher the stakes of the decision, the more incentive to approach it like a rational Bayesian agent who would use whatever tool has the best risk / reward tradeoff, totally regardless of explainability These sorts of problems are mostly difficult precisely because we don't know the risk/reward tradeoff. For example, in the prison system you need to evaluate the risk of recidivism. How exactly do you evaluate the tradeoff between releasing someone who is potentially violent vs. keeping someone potentially reformed behind bars? You'd need to weigh the dollar value of keeping them imprisoned and the personal harm to them and the rest of the prison population when they're imprisoned against the potential harm to anyone else if they're released... plus many other factors that I'm sure I've overlooked. Anyone who tries to bake those tradeoffs into a "rational Bayesian agent model" will fail. You're trying to describe the world in terms of game theory, but we live in a world where we usually don't know the risks, or the payoffs, or the rules of the game.

mlthoughts2018 7y ago
> “These sorts of problems are mostly difficult precisely because we don't know the risk/reward tradeoff.” If you don’t know this trade-off, then you don’t have a decision problem. Basically what you’re saying here doesn’t add up at all. Issues where an ML model is used but people want more “explainability” are places where we absolutely do know the risk / reward characteristics, and we’re specifically trying to measure more of the relationship between the internals of the decision process and those components of the risk / reward characteristics (for purposes of political arguments over who controls the process, dressed up as if they are scientific inquiry instead of politics). If you did not even know what objective you are pursuing at all, you would not be using an ML model, and likely could not even identify what type of decision process you even need yet.
- soVeryTired 7y ago
  > If you don’t know this trade-off, then you don’t have a decision problem Let me clarify what I mean. In the cases I've worked with professionally (finance, autonomous driving), no-one gives you a reward function. You have a rough idea of the outcome you want, and you fiddle around with reward functions until your algirithm kinda-sorta delivers the outcomes you knew you wanted in the first place. In this sense, the OP gets it precisely backwards. OP wants some sort of hyper-rational Bayesian model that optimises outcomes based on the one true loss function. There is no correct reward function, and objective functions in general tend to be far too simple to capture the nuance one encounters in the real world. > If you did not even know what objective you are pursuing at all, you would not be using an ML model... It's not that you don't know at all, but you should be very clear that your objective is a fudge that probably doesn't capture what you really want. Maths is clean: the real world is messy.

mlthoughts2018 7y ago

> “These sorts of problems are mostly difficult precisely because we don't know the risk/reward tradeoff.” If you don’t know this trade-off, then you don’t have a decision problem. Basically what you’re saying here doesn’t add up at all. Issues where an ML model is used but people want more “explainability” are places where we absolutely do know the risk / reward characteristics, and we’re specifically trying to measure more of the relationship between the internals of the decision process and those components of the risk / reward characteristics (for purposes of political arguments over who controls the process, dressed up as if they are scientific inquiry instead of politics). If you did not even know what objective you are pursuing at all, you would not be using an ML model, and likely could not even identify what type of decision process you even need yet.

soVeryTired 7y ago
> If you don’t know this trade-off, then you don’t have a decision problem Let me clarify what I mean. In the cases I've worked with professionally (finance, autonomous driving), no-one gives you a reward function. You have a rough idea of the outcome you want, and you fiddle around with reward functions until your algirithm kinda-sorta delivers the outcomes you knew you wanted in the first place. In this sense, the OP gets it precisely backwards. OP wants some sort of hyper-rational Bayesian model that optimises outcomes based on the one true loss function. There is no correct reward function, and objective functions in general tend to be far too simple to capture the nuance one encounters in the real world. > If you did not even know what objective you are pursuing at all, you would not be using an ML model... It's not that you don't know at all, but you should be very clear that your objective is a fudge that probably doesn't capture what you really want. Maths is clean: the real world is messy.