> What, I'd argue, would be at least a tiny step forward would be thinking in terms of games people play, rewards they seek Definitely. One thing that I think

> What, I'd argue, would be at least a tiny step forward would be thinking in terms of games people play, rewards they seek Definitely. One thing that I think is unappreciated is the extent to which we see "preference falsification". This is a game people play where they pretend to have different preferences to better fit in with their in-group. It's common for preference falsification to be manufactured intentionally – I think Robin Hanson formalized it with the idea of a "meta-norm", a norm that not says: you must ostracize people who do <bad thing x> AND you must ostracize people who don't follow this rule. I think when people complain about "cancel culture", this is the real thing they're unhappy about, they just lack the vocabulary to articulate it. The sneaky thing about the meta-norm is that it's self-reinforcing. Once enough people follow the meta-norm, following the meta-norm becomes a stable equilibrium where no individual person gains from not following it. From Scott Alexander: > Imagine a country with two rules: first, every person must spend eight hours a day giving themselves strong electric shocks. Second, if anyone fails to follow a rule (including this one), or speaks out against it, or fails to enforce it, all citizens must unite to kill that person. Suppose these rules were well-enough established by tradition that everyone expected them to be enforced. > So you shock yourself for eight hours a day, because you know if you don’t everyone else will kill you, because if they don’t, everyone else will kill them, and so on. Every single citizen hates the system, but for lack of a good coordination mechanism it endures. The subtlety of this is that you might genuinely believe that everyone supports the electric shocks, because you'll never hear anyone speaking out against it, even though everyone hates it. I think this dynamic is so powerful that it's almost innate. I remember once when a friend of a friend cheated on her boyfriend regularly. I obviously thought negatively of the cheater and didn't want to be around her, but I also thought negatively of my friend for continuing to be around the cheater. The instinct is that punishing cheaters by social ostracism is socially useful, so we should also punish people who fail to ostracize cheaters by ostracising them, and so on. This can be good like in the case of punishing cheaters, but the problem is that it could work for any social norm even if 100% of people disagreed with it. I think this is a real and powerful social dynamic that leads to a huge amount of people having no choice but to act in bad faith. If this is a real social dynamic, how can it be neutralized? One approach I think is promising is to use local opinion polls, only structured as opinion elections. If everyone could vote anonymously, I'm sure they would say "I'm not such a fan of these electric shocks" (and the anonymity protects them from the fear of socially-enforced retaliation). Once it becomes common knowledge that almost nobody around you likes the electric shocks, it's much easier to coordinate "let's stop punishing people for not shocking themselves". Electric shocks are just an example, you could use this for any hot-button political issue. For example, in the US's antebellum South I'm sure there was immense social pressure to be pro-slavery, but opinion elections might have helped pro-abolition people understand if they were even in the minority (and if so, by how much). This is just one mechanism I think might be workable, but I'm sure if we sat down and thought about it we could come up with many others, like reputation systems for those who make accurate predictions for the future, debates where people have an incentive to call-out their counterparty's selective reporting of the facts, etc.