Ah yes, the self fulfilling prophecies or hallucinations based on models trained on models. Overfitting. Ending up in an evolutionary dead end... Type 4 error

Ah yes, the self fulfilling prophecies or hallucinations based on models trained on models. Overfitting. Ending up in an evolutionary dead end... Type 4 error of not asking a question one should also exists. So thing is, suppose you're handling the common cases right - you have software that's say 95% correct. The important bit is how critical the remaining 5% failures are. If one of them happens to be "I give up my computer and data to the exploit" or "everything is destroyed" or "a lot of people die", then the extra 1% better average is no good to any inside observer. It so happens that a lot of people believe themselves to be outside observers, especially rich. (What's the success bonus for someone getting treated nicely?)