3 ms·
This is a good evaluation of the way risk posture can inform design decisions, but I think it sort of ignores the elephant in the room: A short term strategy wi
by advael 3y ago
This is a good evaluation of the way risk posture can inform design decisions, but I think it sort of ignores the elephant in the room: A short term strategy wins more on average in a competitive context, especially when existential risk is on the table for entities who lose a competition. Most solutions to this problem involve solving hard coordination problems to change the balance of incentives or lower the lethality of competition. Figuring out systems that work for your level of risk tolerance is very achievable and can be incrementally improved, but designing for robustness is something that needs fractal alignment at higher meta-layers of incentives to be sustainable
- nonrandomstring 3y ago> especially when existential risk is on the table for entities who lose a competition. This really just explains externalities. Competition creates apparent short-range existential risk to a business. Real existential risk (to people or things damaged by your defective product because of "precarious perfection") land somewhere else - usually where their impact is much larger.
- advael 3y agoIf something is considered an externality by some system, decisions that prioritize it are by definition hard or impossible to make within that system. In other words, to change that, we have to rewrite systemic priorities to some degree
- nonrandomstring 3y agoThat's an interesting thought I'll give some chin stroking to. Sounds like something Kurt Godel and John Gall would have hatched together. Is it a personal insight, or from experience, or something you think might be attributable. cheers. > we have to rewrite systemic priorities to some degree maybe Forrester or Meadows has an insight?
- advael 3y agoI mean I am sure I'm not the first person to have this insight - as I said I view it as baked into the definition of "externality" - but I'm also not pulling it from any specific source I can consciously recall
- 6LLvveMx2koXfwn 3y ago> but designing for robustness is something that needs fractal alignment at higher meta-layers of incentives to be sustainable This is why I remain an engineer and nothing more. If this sentence ever became meaningful I guess I would have been over-promoted.
- advael 3y agolol okay I'll admit upon reading it again that that was a more verbose way to make that point than it needed to be How about "prioritizing robustness and other kinds of long-termism need to be at least somewhat protected by the system of incentives they operate in to succeed"? I invoked fractal self-similarity there because I think you best protect long-termism by prioritizing long-termism in decision-making at scopes further out