Being epistemically helpless, or at least a bit modest, is a good way for an individual to avoid having false beliefs. I don't think this advice is good for th

Being epistemically helpless, or at least a bit modest, is a good way for an individual to avoid having false beliefs. I don't think this advice is good for the society as a whole, though. Encouraging people to be modest, stand in line, and yield to the majority view by default is a pretty good recipe for a society to become biased toward inertia, often in a very oppressive manner. Japan is one example of such a society. They've made impressive economic and technological progress, but their culture and politics are stuck in a weird loop. The article you linked to also acknowledges that people who are less epistemically helpless are "the only people who can figure out if something basic and unquestionable is wrong, and make this possibility well-known enough that normal people start becoming willing to consider it." Societies need people like this, even though they share some traits with terrorists. So although overconfidence and an experimental spirit might lead individuals to have more false beliefs and failures in the short term, encouraging them might be more beneficial in the long term on a larger scale. This is something that both authors could agree with. I don't know whether Eliezer Yudkowsky would actually be willing to go in this direction, since he looks rather preoccupied with ensuring that each individual human (or AI) makes the best decision at any given time; but since I'm sure he wishes the best for the future of our civilization, I look forward to seeing him pay attention to the social and historical consequences of the individual strategies he's been promoting. A horde of smart robots could behave very differently from a single smart robot if they use the wrong consensus algorithm.