I agree with you, but all you've really said is that we will rely on general AI to accomplish our goals. The stupider the AI, the more you have to define yours

I agree with you, but all you've really said is that we will rely on general AI to accomplish our goals. The stupider the AI, the more you have to define yourself. You'll have to define the terms used in the description of your goal as well as terms used by those terms. This sounds like defining requirements. The problem with this is that, lacking general AI, the effort it takes to define your goals precisely will probably be equivalent to writing a program anyway. The only difference between defining requirements and coding is people writing requirements can rely on a lot of background knowledge by the reader. Most of us that have worked in the corporate world know how poor technical requirements can be.