6 ms·

This may sound a little like trivializing, but don't we have to know what "statistics" are and what "machine learning" is to say anything about the difference(s

This may sound a little like trivializing, but don't we have to know what "statistics" are and what "machine learning" is to say anything about the difference(s) between them? Looking at this through even the lens of multinomial logistic regression, or of econometrics generally, I don't think that "statistics draws population inferences from a sample, while machine learning finds generalizable predictive patterns" even makes sense as a difference. Any prediction is an inference about the population of future events, or of contemporary events not present in the sample. You plug 1000s of events each described by 100 columns into a logistic regression, and you're hoping to get something predictive out of it. Further, as nice as the idea is that you can tease out "factors" from your 100 columns, you don't have to look at "3.3375905e-5 x (spent five years before age 18 in a smoker's home)" for very long to wonder how much 'explanation' you're getting out of the terms in the exponents of your probability functions. I still can't resist tweaking ML enthusiasts and data scientists: Statistics is what people who know what they're doing are doing. Machine learning is the rest!

cwyers 7y ago
Everybody owes it to themselves to read Breiman's Two Cultures[1]: > There are two cultures in the use of statistical modeling to reach conclusions from data. One assumes that the data are generated by a given stochastic data model. The other uses algorithmic models and treats the data mechanism as unknown. The statistical community has been committed to the almost exclusive use of data models. This commitment has led to irrelevant theory, questionable conclusions, and has kept statisticians from working on a large range of interesting current problems. Algorithmic modeling, both in theory and practice, has developed rapidly in fields outside statistics. It can be used both on large complex data sets and as a more accurate and informative alternative to data modeling on smaller data sets. If our goal as a field is to use data to solve problems, then we need to move away from exclusive dependence on data models and adopt a more diverse set of tools. The difference between inference and prediction can be illustrated with something like a decision tree or a random forest. Statistical inference is "the theory, methods, and practice of forming judgments about the parameters of a population and the reliability of statistical relationships, typically on the basis of random sampling." If you look at something like linear regressions, it makes a lot of assumptions about the data[2]: the distribution of residual errors is normal, there's no multicollinearity, etc. Random forest don't care. Random forest is a set of steps. You follow the steps, you get an answer. You made no assumptions about your data, about the distribution of it, any of it. You just followed an algorithm. Algorithms are powerful! Not everything needs to be an inference problem. We're better off when we have lots of tools. You don't have to choose _between_ the two camps. But the "Everything is Statistics!" and the "Everything is Machine Learning!" points of view rob us of ways of thinking about our tools that helps us understand what those tools are. 1) https://projecteuclid.org/download/pdf_1/euclid.ss/1009213726 https://projecteuclid.org/download/pdf_1/euclid.ss/100921372... 2) https://thestatsgeek.com/2013/08/07/assumptions-for-linear-regression/ https://thestatsgeek.com/2013/08/07/assumptions-for-linear-r...
- ssivark 7y ago
  Nice comment, but there's more nuance than you claim. > You made no assumptions about your data, about the distribution of it, any of it. You just followed an algorithm. You made implicit assumptions that you are now unaware of, which might come and bite you later (eg: using zip codes or names as a proxy for race in credit scoring models, neural networks overfititng to texture and classifying a leopard print couch as a leopard). This means that you are liable to overfit to the data, and generalize poorly, or in ways you are not supposed to. This is what leads to the perspective that "machine learning" might end up being a powerful tool for laundering bias: https://idlewords.com/talks/sase_panel.htm https://idlewords.com/talks/sase_panel.htm
  - cwyers 7y ago
    I think what you're saying is orthogonal to what I'm saying. Yes, ZIP codes can be a proxy for race in models dealing with credit scores (and recidivism, which is also a really bad place to put racial bias), as an example. But if I put it in a mixed-effects model, it shows the same bias, and a mixed-effects model is just an extended version of linear regression. Both statistical and ML models suffer from the problem you're stating. What you have not made any assumptions about in a random forest is about the distribution of the data you're looking at. One example of a case where the assumptions that bog-standard OLS makes about your data can cause you problems is zero-dominated data -- data with a lot of zeros in it. Basically any time you're trying to make predictions about things that are rare in your measured population. OLS does a bad job on zero-dominated data. If you throw a zero-dominated dataset into a random forest, you will get back better answers than if you use OLS on zero-dominated data. To be clear: there are strategies for dealing with zero-dominated data using statistical inference. You don't have to resort to non-inferential learning just because you have data that doesn't look like a bell curve. But machine learning is a powerful way to get pretty good results on a lot of problem spaces without having to understand the probability function involved (or in cases where the probability function is too complicated to be tractable computationally, like the probability function that determines the color of pixels in a dataset where you're classifying dogs versus cats).
    - 7y ago
acollins1331 7y ago
Have fun trying to model extremely nonlinear physical effects with statistics
- xkcd-sucks 7y ago
  Oh yes, we just include all the possible nonlinear effects and pick the ones which look meaningful [0][1] Then we go back and perturb the model to do some hand wavey guessing at real actionable insights but hey clients are impressed by overfitting so who the fuck cares as long as the money is coming in and the press releases are going out!!!! [0] https://www.featuretools.com/ https://www.featuretools.com/ [1] https://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.PolynomialFeatures.html https://scikit-learn.org/stable/modules/generated/sklearn.pr...

cwyers 7y ago

Everybody owes it to themselves to read Breiman's Two Cultures[1]: > There are two cultures in the use of statistical modeling to reach conclusions from data. One assumes that the data are generated by a given stochastic data model. The other uses algorithmic models and treats the data mechanism as unknown. The statistical community has been committed to the almost exclusive use of data models. This commitment has led to irrelevant theory, questionable conclusions, and has kept statisticians from working on a large range of interesting current problems. Algorithmic modeling, both in theory and practice, has developed rapidly in fields outside statistics. It can be used both on large complex data sets and as a more accurate and informative alternative to data modeling on smaller data sets. If our goal as a field is to use data to solve problems, then we need to move away from exclusive dependence on data models and adopt a more diverse set of tools. The difference between inference and prediction can be illustrated with something like a decision tree or a random forest. Statistical inference is "the theory, methods, and practice of forming judgments about the parameters of a population and the reliability of statistical relationships, typically on the basis of random sampling." If you look at something like linear regressions, it makes a lot of assumptions about the data[2]: the distribution of residual errors is normal, there's no multicollinearity, etc. Random forest don't care. Random forest is a set of steps. You follow the steps, you get an answer. You made no assumptions about your data, about the distribution of it, any of it. You just followed an algorithm. Algorithms are powerful! Not everything needs to be an inference problem. We're better off when we have lots of tools. You don't have to choose _between_ the two camps. But the "Everything is Statistics!" and the "Everything is Machine Learning!" points of view rob us of ways of thinking about our tools that helps us understand what those tools are. 1) https://projecteuclid.org/download/pdf_1/euclid.ss/1009213726 https://projecteuclid.org/download/pdf_1/euclid.ss/100921372... 2) https://thestatsgeek.com/2013/08/07/assumptions-for-linear-regression/ https://thestatsgeek.com/2013/08/07/assumptions-for-linear-r...

ssivark 7y ago
Nice comment, but there's more nuance than you claim. > You made no assumptions about your data, about the distribution of it, any of it. You just followed an algorithm. You made implicit assumptions that you are now unaware of, which might come and bite you later (eg: using zip codes or names as a proxy for race in credit scoring models, neural networks overfititng to texture and classifying a leopard print couch as a leopard). This means that you are liable to overfit to the data, and generalize poorly, or in ways you are not supposed to. This is what leads to the perspective that "machine learning" might end up being a powerful tool for laundering bias: https://idlewords.com/talks/sase_panel.htm https://idlewords.com/talks/sase_panel.htm
- cwyers 7y ago
  I think what you're saying is orthogonal to what I'm saying. Yes, ZIP codes can be a proxy for race in models dealing with credit scores (and recidivism, which is also a really bad place to put racial bias), as an example. But if I put it in a mixed-effects model, it shows the same bias, and a mixed-effects model is just an extended version of linear regression. Both statistical and ML models suffer from the problem you're stating. What you have not made any assumptions about in a random forest is about the distribution of the data you're looking at. One example of a case where the assumptions that bog-standard OLS makes about your data can cause you problems is zero-dominated data -- data with a lot of zeros in it. Basically any time you're trying to make predictions about things that are rare in your measured population. OLS does a bad job on zero-dominated data. If you throw a zero-dominated dataset into a random forest, you will get back better answers than if you use OLS on zero-dominated data. To be clear: there are strategies for dealing with zero-dominated data using statistical inference. You don't have to resort to non-inferential learning just because you have data that doesn't look like a bell curve. But machine learning is a powerful way to get pretty good results on a lot of problem spaces without having to understand the probability function involved (or in cases where the probability function is too complicated to be tractable computationally, like the probability function that determines the color of pixels in a dataset where you're classifying dogs versus cats).
  - 7y ago

acollins1331 7y ago

Have fun trying to model extremely nonlinear physical effects with statistics

xkcd-sucks 7y ago
Oh yes, we just include all the possible nonlinear effects and pick the ones which look meaningful [0][1] Then we go back and perturb the model to do some hand wavey guessing at real actionable insights but hey clients are impressed by overfitting so who the fuck cares as long as the money is coming in and the press releases are going out!!!! [0] https://www.featuretools.com/ https://www.featuretools.com/ [1] https://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.PolynomialFeatures.html https://scikit-learn.org/stable/modules/generated/sklearn.pr...