This leads me to wonder: what abstractions are we missing today that will be obvious to future generations? Assuming that at some point we will stop writing co

This leads me to wonder: what abstractions are we missing today that will be obvious to future generations? Assuming that at some point we will stop writing code and machines will generate programs for us instead, then we need abstractions that make this possible: a way to describe the purpose of a program that is precise enough to generate code satisfying the purpose and that can be derived from normal human communication instead of having to be explicitly coded. In other words: a way of describing why the program exists, instead of describing what it should do. Additionally, to avoid paperclip maximizers we would need a system of ethics and values robust enough that it becomes computable. Then it can be integrated into the code-writing code in such a way as to constrain the code it generates to only choose approaches that are in line with the value system, and to refuse to implement purposes at odds with the value system. As long as our best stab at ethics largely boils down to long lists of behavioral rules written in holy books with few underlying principles we aren't ready to build truly advanced AI. Probably some philosopher has already figured out the right model of abstract ethics, but has gone mostly undiscovered so far.