Anecdote on that same line of thinking: Ari K. Jonsson made software for the MER rovers that planned and scheduled the science done for each day. It basically

Anecdote on that same line of thinking: Ari K. Jonsson made software for the MER rovers that planned and scheduled the science done for each day. It basically book-kept the constraints on operations of the rover. (this joint has to be that hot before moving, requiring energy, moving said joint is a pre-requisite for taking picture of rock X). The geologists would take data from the previous day and make a list of all the things they wanted to do (sample that area, picture this rock, etc.) and the planning software would put up this gigantic gantt chart humans would then mess with to get as many "science point" out of the hardware for that day. Usability surveys after the project was quite mature revealed that the full-on "auto-plan" was not used since the human engineers didn't trust the plans, they did not understand them. They saw that more net-science was being done that way, they saw that all the constraints were being met (no equipment was being jeopardized by the plan itself), just didn't trust the "machine intelligence". Another striking thing about that project he revealed: The humans using the software didn't really think they were "using the AI" since they didn't use the auto-planner (even if the AI was working very hard maintaining constraints). Dr. Jonsson was the Dean where I graduated. I'll go listen to his Mars experience any time I know he's speaking about it. I feel fortunate to have had him as a teacher.