Thanks a lot for outlining your thoughts on the matter! I'll have a look at the paper you have linked as well. Regarding EA, I agree broadly with the basic ide

Thanks a lot for outlining your thoughts on the matter! I'll have a look at the paper you have linked as well. Regarding EA, I agree broadly with the basic ideas, but disagree with the focus on AI safety (in fact, I think that runaway AI is impossible due to fundamental limitations of computational complexity and of data ingestion required for accurate models). I agree that utilitarianism itself is not very useful for any one entity in modeling reality. In practice, I use fallible intermediate principles that are more applicable to the situation at hand, and when they seem inadequate, or when they seem to contradict other principles, I would use as the yardstick to examine these principles some hedonistic consequentialist calculus that is heavily hedged by acknowledging a lack of good information and inability to predict the future well. I understand that there is always a risk that I do not do good by this calculus, but I try to maximize the expectation of it under what I know at hand, understanding that this does not account for what I do not know. > It becomes a moral imperative to gather as much information as possible to enable the state to maximize utility. It seems to me that the unsavoriness at this thought arises from the prospect of the state developing a surveillance apparatus that does not serve the citizens as well as it should. I assume that what you call bureaucracy refers to surveillance (via forms, regulations, procedures, amid others.) But I think we agree that we prefer to live in a society that has some surveillance and monopoly on violence, and the question is, to what extent. Then I don't think utilitarianism immediately asks the state to maximize surveillance on its citizens, but this is usually conditional on that the state has sufficiently robust institutions that its well-intentioned surveillance today will not be misappropriated in future, and that the surveillance apparatus is still net-pleasurable to run today, amid other considerations.