4 ms·
There's a theory about this: https://en.wikipedia.org/wiki/Risk_compensation https://en.wikipedia.org/wiki/Risk_compensation > "Risk compensation is a theory w
by richdougherty 8y ago
There's a theory about this: https://en.wikipedia.org/wiki/Risk_compensation https://en.wikipedia.org/wiki/Risk_compensation
> "Risk compensation is a theory which suggests that people typically adjust their behavior in response to the perceived level of risk, becoming more careful where they sense greater risk and less careful if they feel more protected. Although usually small in comparison to the fundamental benefits of safety interventions, it may result in a lower net benefit than expected."
There's a book too, with a special emphasis on on financial crises. https://www.theguardian.com/books/2015/oct/12/foolproof-greg-ip-review-biggest-risk-is-safety https://www.theguardian.com/books/2015/oct/12/foolproof-greg...
> "In the run-up to the crash, consumers and even policymakers had come to believe that smart regulators and forward-thinking bankers had made the world of money a much safer place.
> "The fundamental insight of Ip’s new book, Foolproof, is that this very belief was a key factor in the lead up to the crash. When people believe they are safe, they take more risks – they drive faster, in motoring terms – and “speed makes everything worse”. Or as the economist Hyman Minsky, whose work Ip revisits, put it: “Stability is destabilising.”
There are applications in our field too:
- safety features for users might make them behave less safely (e.g. exploding messages)
- better reliability of systems might lead us to put more trust in them, leading to even bigger outages when they occur (e.g. centralising trust in cloud providers)
It's interesting to see things like Chaos Engineering (https://principlesofchaos.org/ https://principlesofchaos.org/) introducing intentional "danger" into a system in order to improve system-wide stability. Of course, maybe Chaos Engineering will give us more trust in our systems which may lead us to take even bigger risks...
- geofft 8y agoYup, that's basically what I'm getting at, thanks for the links! So, I'm okay with risk compensation if people are net doing better. I don't think that "if even one person is hurt by this, that's too much" is a meaningful basis for decisions, especially when there's a risk that even one person will be hurt by not doing the thing. So at the risk of reducing people to numbers, if, say, 100 teenagers send sexts when they otherwise wouldn't have and get screenshotted, but 1,000 teenagers send sexts when they otherwise would have sent them to a non-disappearing-by-default client, and now their photos don't get copied because of social pressure / high-but-not-impossible technical barriers, that still seems like a clear win. That's the sort of data that I think would be very interesting to inform good engineering decisions, and also pretty impossible to get.