4 ms·

Classic regression is only intelligible because there were only a few parameters and people could use ANOVA to try and interpret them. IMO ANOVA is alchemy as w

by jsinai 8y ago

Classic regression is only intelligible because there were only a few parameters and people could use ANOVA to try and interpret them. IMO ANOVA is alchemy as well and most people trained to use it, don’t fully understand it. Should we leave decisions to those kinds of models? Not to mention that part of what makes NN such a step forward is precisely the high nonlinearity. When you have millions of parameters, the contribution of each is unimportant and so any kind of analog to ANOVA would be barking up the wrong tree. In a NN all those parameters work in concert to learn a decision boundary to separate data. It’s not intelligible at the minute level but at least we know what it’s doing in the end. The problem of course lies with the outliers and that’s not so much a problem with NN being a black box as it is a problem with the nature of large datasets and our own inability to rationalize each and every datapoint. I’m going to defend NN as the natural evolution of regression. It’s precisely their high nonlinearity which makes them better. The problem is not that they “are” alchemy but that we treat them “as” magic. Society as no place leaving important decisions to algorithms unintended, NN or not. If major insurance companies left their decisions to logistic regression (which was and is still the case), then would we be making the same arguments? Probably not because someone paid by the insurance company will pull out that other kind of alchemy called ANOVA ...

RA_Fisher 8y ago
ANOVA is grounded by theory that has stood the test of time for decades. Statisticians that question ANOVA aren't a thing (despite such an accurate criticism being valuable career-wise to an academic). Gelman points out that multilevel models are easier to use than ANOVA and I tend to agree.
vharuck 8y ago
Classic regression's potential for intelligibility relies on the modeler choosing terms which are based on observed relations and improve performance. For example, population forecasts can be modeled as the current population plus forecasted changes through immigration, emigration, birth, and death. Yes, somebody can produce "magical" regression models with terms divorced from reality. But unsupervised learning is never guaranteed to produce a reality-grounded model, no matter the user's skill. It goes through steps, tries different transformations, and chooses the algorithm which led to the best result. Logic and understanding played no part. That sounds like alchemy. It definitely works, but alchemy also stumbled onto theories later explained by chemistry. And, yes, anyone who doesn't consider the implications of the linear model behind ANOVA would also be a "magician."

by jsinai 8y ago

RA_Fisher 8y ago

ANOVA is grounded by theory that has stood the test of time for decades. Statisticians that question ANOVA aren't a thing (despite such an accurate criticism being valuable career-wise to an academic). Gelman points out that multilevel models are easier to use than ANOVA and I tend to agree.

vharuck 8y ago

Classic regression's potential for intelligibility relies on the modeler choosing terms which are based on observed relations and improve performance. For example, population forecasts can be modeled as the current population plus forecasted changes through immigration, emigration, birth, and death. Yes, somebody can produce "magical" regression models with terms divorced from reality. But unsupervised learning is never guaranteed to produce a reality-grounded model, no matter the user's skill. It goes through steps, tries different transformations, and chooses the algorithm which led to the best result. Logic and understanding played no part. That sounds like alchemy. It definitely works, but alchemy also stumbled onto theories later explained by chemistry. And, yes, anyone who doesn't consider the implications of the linear model behind ANOVA would also be a "magician."