My expectation is that if it turns out to be practically possible and significantly beneficial to turn over lots of strategic military and business decision mak

My expectation is that if it turns out to be practically possible and significantly beneficial to turn over lots of strategic military and business decision making over to AI, it soon won't be a choice not to, thanks to competition. I think AI rule of much of our top-level, super-important decision making is either a dead-end that will never work or inevitable and something we won't have the practical choice to avoid (barring, say, a real and effective world government) with little room for anything in between. And if it pans out, I expect it to be much as you wrote: the decisions will seem nonsensical much of the time, but those who second-guess the AI will be at a disadvantage against those who do not.