4 ms·

Everybody owes it to themselves to read Breiman's Two Cultures[1]: > There are two cultures in the use of statistical modeling to reach conclusions from data.

by cwyers 7y ago

Everybody owes it to themselves to read Breiman's Two Cultures[1]: > There are two cultures in the use of statistical modeling to reach conclusions from data. One assumes that the data are generated by a given stochastic data model. The other uses algorithmic models and treats the data mechanism as unknown. The statistical community has been committed to the almost exclusive use of data models. This commitment has led to irrelevant theory, questionable conclusions, and has kept statisticians from working on a large range of interesting current problems. Algorithmic modeling, both in theory and practice, has developed rapidly in fields outside statistics. It can be used both on large complex data sets and as a more accurate and informative alternative to data modeling on smaller data sets. If our goal as a field is to use data to solve problems, then we need to move away from exclusive dependence on data models and adopt a more diverse set of tools. The difference between inference and prediction can be illustrated with something like a decision tree or a random forest. Statistical inference is "the theory, methods, and practice of forming judgments about the parameters of a population and the reliability of statistical relationships, typically on the basis of random sampling." If you look at something like linear regressions, it makes a lot of assumptions about the data[2]: the distribution of residual errors is normal, there's no multicollinearity, etc. Random forest don't care. Random forest is a set of steps. You follow the steps, you get an answer. You made no assumptions about your data, about the distribution of it, any of it. You just followed an algorithm. Algorithms are powerful! Not everything needs to be an inference problem. We're better off when we have lots of tools. You don't have to choose _between_ the two camps. But the "Everything is Statistics!" and the "Everything is Machine Learning!" points of view rob us of ways of thinking about our tools that helps us understand what those tools are. 1) https://projecteuclid.org/download/pdf_1/euclid.ss/1009213726 https://projecteuclid.org/download/pdf_1/euclid.ss/100921372... 2) https://thestatsgeek.com/2013/08/07/assumptions-for-linear-regression/ https://thestatsgeek.com/2013/08/07/assumptions-for-linear-r...

ssivark 7y ago
Nice comment, but there's more nuance than you claim. > You made no assumptions about your data, about the distribution of it, any of it. You just followed an algorithm. You made implicit assumptions that you are now unaware of, which might come and bite you later (eg: using zip codes or names as a proxy for race in credit scoring models, neural networks overfititng to texture and classifying a leopard print couch as a leopard). This means that you are liable to overfit to the data, and generalize poorly, or in ways you are not supposed to. This is what leads to the perspective that "machine learning" might end up being a powerful tool for laundering bias: https://idlewords.com/talks/sase_panel.htm https://idlewords.com/talks/sase_panel.htm
- cwyers 7y ago
  I think what you're saying is orthogonal to what I'm saying. Yes, ZIP codes can be a proxy for race in models dealing with credit scores (and recidivism, which is also a really bad place to put racial bias), as an example. But if I put it in a mixed-effects model, it shows the same bias, and a mixed-effects model is just an extended version of linear regression. Both statistical and ML models suffer from the problem you're stating. What you have not made any assumptions about in a random forest is about the distribution of the data you're looking at. One example of a case where the assumptions that bog-standard OLS makes about your data can cause you problems is zero-dominated data -- data with a lot of zeros in it. Basically any time you're trying to make predictions about things that are rare in your measured population. OLS does a bad job on zero-dominated data. If you throw a zero-dominated dataset into a random forest, you will get back better answers than if you use OLS on zero-dominated data. To be clear: there are strategies for dealing with zero-dominated data using statistical inference. You don't have to resort to non-inferential learning just because you have data that doesn't look like a bell curve. But machine learning is a powerful way to get pretty good results on a lot of problem spaces without having to understand the probability function involved (or in cases where the probability function is too complicated to be tractable computationally, like the probability function that determines the color of pixels in a dataset where you're classifying dogs versus cats).
  - ssivark 7y ago
    > machine learning is a powerful way to get pretty good results on a lot of problem spaces without having to understand the probability function involved The NFL theorems essentially state that for every dataset on which an algorithm generalizes well, there is another on which it generalizes poorly. So, there are always implicit biases, even for a random forest model (eg: if you try to model Boolean functions with a random forest, those functions which can be approximated effectively with few trees will form a set of very small measure. Specifically, it is my intuition that those which have many terms in the sum of products form might need many trees to approximate). It then becomes a question of whether the bias of your models are compatible with the dataset/domain under consideration. See this Minsky-Sussman koan: http://www.catb.org/jargon/html/koans.html#id3141241 http://www.catb.org/jargon/html/koans.html#id3141241 > Both statistical and ML models suffer from the problem you're stating. And that is precisely why I don’t see how ML algorithms are “more powerful” than statistics in any way, as per your claim.
taeric 7y ago
That seems to paint random forests as too magical. Rather, to me, random forests are just assuming that there is a BDD to get your answer. You aren't making statistical assumptions of the data, per se. However, you are assuming you can keep doing some sort of split on the data. Such that lower levels of the tree should be somewhat interpretable. Same for the highest levels. It is just the walk from high to low that is hard to fully explain. Right?
- cwyers 7y ago
  I don't think random forests are "magical" at all. I think you can walk a non-technical audience from decision trees to bagged and boosted forests on a whiteboard in under an hour. (I don't think you can do this nearly as well for maximum likelihood estimation, honestly.) But there's still a difference between a mathematical function that you're trying to find the optimal values for and something that's defined by an algorithm that there's no single function to optimize.
  - taeric 7y ago
    But, at the end of the day, you can get some easy to understand statistics from building a decision tree. Based on population makeup, you'll know solid numbers on how many will be expected to hit each outcome. Such that optimizing some other function based on the application of a decision tree is effectively a statistical question. A natural one. Indeed, I'm not sure I see the difference. One is just explainable based on understandings of distributions and trusting that they hold. One is explainable based on understanding of decisions, and trusting those remain the best decisions. I used my bike ride as an example earlier. It is made up of many decisions to get home. Knowing what all of those decisions are can help build up a solid range of when I'm likely to get home. For the level of accuracy I typically care about, so does just knowing the rough distribution of how long I typically take. How are both of those not statistical in nature? One is just more fully exploring a space with ridiculous computational power, whereas the other is generalizing to a much quicker answer. Right?
AstralStorm 7y ago
Breiman is right and wrong at the same time. Powerful tools like nonlinear polynomial models, OLS, HMM and RBMs were implemented and devised by statisticians. And not from tribe 2. The difference is the data model is explicit but general.

by cwyers 7y ago

ssivark 7y ago

Nice comment, but there's more nuance than you claim. > You made no assumptions about your data, about the distribution of it, any of it. You just followed an algorithm. You made implicit assumptions that you are now unaware of, which might come and bite you later (eg: using zip codes or names as a proxy for race in credit scoring models, neural networks overfititng to texture and classifying a leopard print couch as a leopard). This means that you are liable to overfit to the data, and generalize poorly, or in ways you are not supposed to. This is what leads to the perspective that "machine learning" might end up being a powerful tool for laundering bias: https://idlewords.com/talks/sase_panel.htm https://idlewords.com/talks/sase_panel.htm

cwyers 7y ago
I think what you're saying is orthogonal to what I'm saying. Yes, ZIP codes can be a proxy for race in models dealing with credit scores (and recidivism, which is also a really bad place to put racial bias), as an example. But if I put it in a mixed-effects model, it shows the same bias, and a mixed-effects model is just an extended version of linear regression. Both statistical and ML models suffer from the problem you're stating. What you have not made any assumptions about in a random forest is about the distribution of the data you're looking at. One example of a case where the assumptions that bog-standard OLS makes about your data can cause you problems is zero-dominated data -- data with a lot of zeros in it. Basically any time you're trying to make predictions about things that are rare in your measured population. OLS does a bad job on zero-dominated data. If you throw a zero-dominated dataset into a random forest, you will get back better answers than if you use OLS on zero-dominated data. To be clear: there are strategies for dealing with zero-dominated data using statistical inference. You don't have to resort to non-inferential learning just because you have data that doesn't look like a bell curve. But machine learning is a powerful way to get pretty good results on a lot of problem spaces without having to understand the probability function involved (or in cases where the probability function is too complicated to be tractable computationally, like the probability function that determines the color of pixels in a dataset where you're classifying dogs versus cats).
- ssivark 7y ago
  > machine learning is a powerful way to get pretty good results on a lot of problem spaces without having to understand the probability function involved The NFL theorems essentially state that for every dataset on which an algorithm generalizes well, there is another on which it generalizes poorly. So, there are always implicit biases, even for a random forest model (eg: if you try to model Boolean functions with a random forest, those functions which can be approximated effectively with few trees will form a set of very small measure. Specifically, it is my intuition that those which have many terms in the sum of products form might need many trees to approximate). It then becomes a question of whether the bias of your models are compatible with the dataset/domain under consideration. See this Minsky-Sussman koan: http://www.catb.org/jargon/html/koans.html#id3141241 http://www.catb.org/jargon/html/koans.html#id3141241 > Both statistical and ML models suffer from the problem you're stating. And that is precisely why I don’t see how ML algorithms are “more powerful” than statistics in any way, as per your claim.

taeric 7y ago

That seems to paint random forests as too magical. Rather, to me, random forests are just assuming that there is a BDD to get your answer. You aren't making statistical assumptions of the data, per se. However, you are assuming you can keep doing some sort of split on the data. Such that lower levels of the tree should be somewhat interpretable. Same for the highest levels. It is just the walk from high to low that is hard to fully explain. Right?

cwyers 7y ago
I don't think random forests are "magical" at all. I think you can walk a non-technical audience from decision trees to bagged and boosted forests on a whiteboard in under an hour. (I don't think you can do this nearly as well for maximum likelihood estimation, honestly.) But there's still a difference between a mathematical function that you're trying to find the optimal values for and something that's defined by an algorithm that there's no single function to optimize.
- taeric 7y ago
  But, at the end of the day, you can get some easy to understand statistics from building a decision tree. Based on population makeup, you'll know solid numbers on how many will be expected to hit each outcome. Such that optimizing some other function based on the application of a decision tree is effectively a statistical question. A natural one. Indeed, I'm not sure I see the difference. One is just explainable based on understandings of distributions and trusting that they hold. One is explainable based on understanding of decisions, and trusting those remain the best decisions. I used my bike ride as an example earlier. It is made up of many decisions to get home. Knowing what all of those decisions are can help build up a solid range of when I'm likely to get home. For the level of accuracy I typically care about, so does just knowing the rough distribution of how long I typically take. How are both of those not statistical in nature? One is just more fully exploring a space with ridiculous computational power, whereas the other is generalizing to a much quicker answer. Right?

AstralStorm 7y ago

Breiman is right and wrong at the same time. Powerful tools like nonlinear polynomial models, OLS, HMM and RBMs were implemented and devised by statisticians. And not from tribe 2. The difference is the data model is explicit but general.