Strongly disagree with this line of argument. Almost all human systems are intentionally opaque about some choices or have unplanned consequences you can't get

Strongly disagree with this line of argument. Almost all human systems are intentionally opaque about some choices or have unplanned consequences you can't get a clear answer about. And, algorithms can be studied in the same ways that we use to critically evaluate human systems -- and sometimes better. Why does your local paper run stories featuring some businesses but not others? Why do college admissions tend to overrepresent some populations and underrepresent others? Why are women paid less on average for the same titles? Why do prosecutors disproportionately bring charges against minority suspects with equivalent weight of evidence? You can't get answers about those human decision processes just by asking - it requires research, competing theories, and sometimes legal action forcing people to share info. Explainability of decisions isn't about how much tech is used... It's about how open a system's operators are to sharing info, and how much effort everyone is willing to put into interpreting it.