Yes, you are oversimplifying what is required to make a model explainable. The logical mechanism of the functional form of the model has very little to do with

Yes, you are oversimplifying what is required to make a model explainable. The logical mechanism of the functional form of the model has very little to do with explainability. Explainability involves various kinds of model checking, overfitting analysis, tests for confounding variables, multicollinearity, interaction effects, model misspecification, etc. Merely decomposing a prediction into say a linear combination of predictors does not, itself, provide any type of explanation unless a variety of more complicated assumptions about statistical model checking turn out to be true. Even worse, because the naive approach is to just treat those regression coefficients as if they do automatically give explanatory power or feature-wise attribution, people misrepresent things unwittingly and don’t carry out enough robust model checking, leading to wrong explanations that appear to have deceptive degrees of confidence associated with them.