5 ms·

Wait, what? Are you confusing ML with AI? The last two courses I took on ML were all theory. For example, when we use linear regression, we make assumptions abo

by QML 8y ago

Wait, what? Are you confusing ML with AI? The last two courses I took on ML were all theory. For example, when we use linear regression, we make assumptions about the noise — i.i.d. and Gaussian.

nurettin 8y ago
Would you describe AI as: automatic categorization and decision making? If so, then Machine Learning is a part of modeling AI. Regardless of how they are taught in terms of University lectures.
- RA_Fisher 8y ago
  I'd describe AI/ML as statistics by another name.
  - nurettin 8y ago
    I think this is meaningful.
    - RA_Fisher 8y ago
      Thanks! It took a lot of education and practice in statistics to realize this.
tnecniv 8y ago
I believe OP is speculating about state-of-the-art ML. The classical techniques, like linear regression, are well studied. However, at this point, there is a race to explain theoretically _why_ deep learning is so successful at generalization when, by classical standards, it shouldn't be. OP is speculating that this quest is in vain.
- notafraudster 8y ago
  I agree with your interpretation and pushback against your parent comment, but wanted to elaborate that the "problem" here is much deeper. Deep learning is basically magic, okay, fine, so let's start much closer to linear regression. Say we have a predictor matrix X with 2 predictors. We fit a model using a penalized linear regression (say LASSO) adding to our predictor matrix an interaction terms, arbitrary polynomial and logarithmic transformations of each X, and interactions between the transformations of the Xes. Ideally we motivate this because of some case knowledge about relevant nonlinear transformations of the predictors. Or maybe second best we use the kernel trick to run a kernelized regression that uses an infinite dimensional prediction of all possible transformations of the predictors. But realistically, we toss some shit in the model and run it. The LASSO spits out X1, X2^2, and X1^3 * log(X2) as being the cross-validation selected non-zero parameters. What real world scenario could possibly generate a causal process that is linear in X1 (say income), quadratic in X2 (say age, which often displays quadratic forms in regressions), but also predicted by a bizarre non-linear interaction of nonsense transformations? What a practitioner would probably do is fit the model. In a lot of ML contexts, interpretation would lead to the practitioner saying "Well, okay, ML sometimes produces nonsense models, but you can't argue with the predictive results". Or maybe the practitioner is more sensitive to interpretability and instead takes another tack. Maybe the practitioner might say "clearly this interaction is nonsense, but there must be some interaction, I'll re-run with a linear interaction". Or else they'd re-run the LASSO with conditions about not including nonlinear terms without including the lower dimensional terms. Or else they'd run a grouped LASSO and make up some justification for the groups. All of these reveal that most ML practitioners are basically just doing alchemy. And this is talking about what amounts to a minor version increment of linear regression, so probably the simplest possible technique we'd still call part of ML.
  - RA_Fisher 8y ago
    You're spot on. Another point I like to raise is that predictors are often extremely redundant in terms of information. The implication being that nearly as good predictions might be made with three predictors vs thousands.
- deleted 8y ago
  [deleted]
notafraudster 8y ago
In the context of linear regression, there is no particular reason to assume noise is i.i.d. and Gaussian. The former is part of the the Gauss-Markov conditions, under which OLS is BLUE (the "best linear unbiased estimator"). The latter is not necessary at all. And of the Gauss-Markov conditions can be violated to varying degrees of consequence. In fact, in real data, these assumptions are almost always violated. The Gaussian assumption doesn't matter at all, but to address the i.i.d. assumption: Almost all real data exhibits residual heteroskedasticity and almost all real data has observable clustering. Which is why almost no one uses OLS with classical errors. We have estimators to allow errors to be heteroskedasticity-consistent (the default in STATA and easily estimated in R e.g. by estimatr, clubSandwich, etc) or cluster-robust or both. By definition these cases have non-i.i.d. errors and there's no reason linear regression can't be used with them. We also don't need to make assumptions, these can be interrogated. Most regression relies on using the residual matrix as sample plug-ins for the underlying error matrix, so there's a wide assortment of diagnostic techniques to check for the presence or absence of those assumptions. Insofar as "machine learning" has any meaning -- which is to say, insofar as it is different than "statistics", the difference is purportedly that it focuses on minimizing out of sample prediction error rather than estimating population parameters, and typically this is motivated as an overfitting problem. We use OLS because OLS is BLUE under the Gauss-Markov conditions. In ML we rarely care about "U" (unbiasedness) because we frequently prefer to make a bias-variance tradeoff if we're aiming to minimize out of sample error. When linear regression is used in an ML context it is typically penalized linear regression (i.e. ridge / LASSO). Of course it's also the case that the bulk of sexy ML results come out of non-linear estimators, and absent a need to characterize population parameters there's no real reason to care about interpretability so really we don't care about the "L" either. I would say the grandparent is closer to right. Often in ML there is a view that we throw a bunch of processes at data, pick the thing that works best, don't care why it works at all, and then run with it. To the extent there's a protection against fishing expeditions, it's in the training/test separation or cross-validation or both. Most of the time when someone talks about "regression theory", they're used 30 or 40 year old results. For an updated look, check out "Foundations of Agnostic Regression" (Aronow and Miller, both Yale Political Scientists) which is coming out some time in 2019. They've had a pre-print around for a while and if you're interested I'm sure you could get one.

by QML 8y ago

Wait, what? Are you confusing ML with AI? The last two courses I took on ML were all theory. For example, when we use linear regression, we make assumptions about the noise — i.i.d. and Gaussian.

nurettin 8y ago

Would you describe AI as: automatic categorization and decision making? If so, then Machine Learning is a part of modeling AI. Regardless of how they are taught in terms of University lectures.

RA_Fisher 8y ago
I'd describe AI/ML as statistics by another name.
- nurettin 8y ago
  I think this is meaningful.
  - RA_Fisher 8y ago
    Thanks! It took a lot of education and practice in statistics to realize this.

tnecniv 8y ago

I believe OP is speculating about state-of-the-art ML. The classical techniques, like linear regression, are well studied. However, at this point, there is a race to explain theoretically _why_ deep learning is so successful at generalization when, by classical standards, it shouldn't be. OP is speculating that this quest is in vain.

notafraudster 8y ago
I agree with your interpretation and pushback against your parent comment, but wanted to elaborate that the "problem" here is much deeper. Deep learning is basically magic, okay, fine, so let's start much closer to linear regression. Say we have a predictor matrix X with 2 predictors. We fit a model using a penalized linear regression (say LASSO) adding to our predictor matrix an interaction terms, arbitrary polynomial and logarithmic transformations of each X, and interactions between the transformations of the Xes. Ideally we motivate this because of some case knowledge about relevant nonlinear transformations of the predictors. Or maybe second best we use the kernel trick to run a kernelized regression that uses an infinite dimensional prediction of all possible transformations of the predictors. But realistically, we toss some shit in the model and run it. The LASSO spits out X1, X2^2, and X1^3 * log(X2) as being the cross-validation selected non-zero parameters. What real world scenario could possibly generate a causal process that is linear in X1 (say income), quadratic in X2 (say age, which often displays quadratic forms in regressions), but also predicted by a bizarre non-linear interaction of nonsense transformations? What a practitioner would probably do is fit the model. In a lot of ML contexts, interpretation would lead to the practitioner saying "Well, okay, ML sometimes produces nonsense models, but you can't argue with the predictive results". Or maybe the practitioner is more sensitive to interpretability and instead takes another tack. Maybe the practitioner might say "clearly this interaction is nonsense, but there must be some interaction, I'll re-run with a linear interaction". Or else they'd re-run the LASSO with conditions about not including nonlinear terms without including the lower dimensional terms. Or else they'd run a grouped LASSO and make up some justification for the groups. All of these reveal that most ML practitioners are basically just doing alchemy. And this is talking about what amounts to a minor version increment of linear regression, so probably the simplest possible technique we'd still call part of ML.
- RA_Fisher 8y ago
  You're spot on. Another point I like to raise is that predictors are often extremely redundant in terms of information. The implication being that nearly as good predictions might be made with three predictors vs thousands.
deleted 8y ago
[deleted]

notafraudster 8y ago

In the context of linear regression, there is no particular reason to assume noise is i.i.d. and Gaussian. The former is part of the the Gauss-Markov conditions, under which OLS is BLUE (the "best linear unbiased estimator"). The latter is not necessary at all. And of the Gauss-Markov conditions can be violated to varying degrees of consequence. In fact, in real data, these assumptions are almost always violated. The Gaussian assumption doesn't matter at all, but to address the i.i.d. assumption: Almost all real data exhibits residual heteroskedasticity and almost all real data has observable clustering. Which is why almost no one uses OLS with classical errors. We have estimators to allow errors to be heteroskedasticity-consistent (the default in STATA and easily estimated in R e.g. by estimatr, clubSandwich, etc) or cluster-robust or both. By definition these cases have non-i.i.d. errors and there's no reason linear regression can't be used with them. We also don't need to make assumptions, these can be interrogated. Most regression relies on using the residual matrix as sample plug-ins for the underlying error matrix, so there's a wide assortment of diagnostic techniques to check for the presence or absence of those assumptions. Insofar as "machine learning" has any meaning -- which is to say, insofar as it is different than "statistics", the difference is purportedly that it focuses on minimizing out of sample prediction error rather than estimating population parameters, and typically this is motivated as an overfitting problem. We use OLS because OLS is BLUE under the Gauss-Markov conditions. In ML we rarely care about "U" (unbiasedness) because we frequently prefer to make a bias-variance tradeoff if we're aiming to minimize out of sample error. When linear regression is used in an ML context it is typically penalized linear regression (i.e. ridge / LASSO). Of course it's also the case that the bulk of sexy ML results come out of non-linear estimators, and absent a need to characterize population parameters there's no real reason to care about interpretability so really we don't care about the "L" either. I would say the grandparent is closer to right. Often in ML there is a view that we throw a bunch of processes at data, pick the thing that works best, don't care why it works at all, and then run with it. To the extent there's a protection against fishing expeditions, it's in the training/test separation or cross-validation or both. Most of the time when someone talks about "regression theory", they're used 30 or 40 year old results. For an updated look, check out "Foundations of Agnostic Regression" (Aronow and Miller, both Yale Political Scientists) which is coming out some time in 2019. They've had a pre-print around for a while and if you're interested I'm sure you could get one.