24 ms·

On moving from statistics to machine learning, the final stage of grief (2019)

175 points by yoloswagins 6y ago

deleted 6y ago
[deleted]
- deleted 6y ago
  [deleted]
scribu 6y ago
This is one of the clearest explanations I've read on the difference between traditional Statistics and Machine Learning.
- yt-sdb 6y ago
  A few years ago, Michael I. Jordan did an AMA on Reddit and discussed this distinction as well. Maybe you'll find it interesting as a counterpoint [1]. [1] https://old.reddit.com/r/MachineLearning/comments/2fxi6v/ama_michael_i_jordan/ckelmtt/ https://old.reddit.com/r/MachineLearning/comments/2fxi6v/ama...
  - mturmon 6y ago
    Perfect link. He’s the ideal commentator.
aabajian 6y ago
The author's pie chart showing data science to be 60% data manipulation is accurate. The biggest gap between good and bad data scientist is their comfort level with data wrangling. When interviewing candidates for data science positions, one of the simplest questions is to have them sort a 1 GB tab-delimited file. 1. Poor candidates will try to open the file in Excel. 2a. Marginal candidates will use R or Stata. 2b. Okay candidates will use a scripting language like Python. 3. Good candidates will use Unix sort. To my knowledge, there are no university courses teaching the Unix toolchain and it remains very much a skill learned through practice.
- blackbear_ 6y ago
  Not sure why you think a candidate who uses R is inferior to one who uses Python? Also, a really good candidate should use the right tool for the job, so if you expect them to use Unix sort you should somehow imply a situation where that is the best approach.
  - dgellow 6y ago
    I would also add: if it’s a one time thing, I would just do it in visual studio code or any other editor that doesn’t die on a 1GB file. And I have 12+ years experience with bash and unix tools, so it’s not about a lack of knowledge or experience. There isn’t anything magical regarding “sort” versus another tool, if there isn’t a need for automation they are equivalent.
  - deleted 6y ago
    [deleted]
  - whym 6y ago
    > if you expect them to use Unix sort you should somehow imply a situation where that is the best approach. I think the implied question is whether the interviewee is aware of the fact that trying to load a 1 GB text file could use up too much RAM space of the system. Unix sort is arguably the most memory efficient among the 4 choices there. It depends on the amount of available resources (which was not specified), and some companies might be willing to casually let people use 100 GB machines, though.
    - blackbear_ 6y ago
      That is a fair point. A good script would be to move towards this scenario of more data than RAM and see what the candidate comes up with.
- jsinai 6y ago
  In my experience the real meat of data manipulation/wrangling is in extracting statistical value from the data. If the data is noisy, what aggregations and what filters produce the best signal without smudging the underlying statistical properties? While this is a good skill to have and is a good sign of efficiency, for me the deal breaker is how well the purpose of data transformations is understood and the care to which statistical value is extracted and tested.
fxtentacle 6y ago
The author appears to misunderstand the main difference between statistics and ML. Let me cite him: > my gut reaction is to barf when someone says “teaching the model” instead of “estimating the parameters.” Typical statistics work is to use a known good model and estimate its parameters. Typical machine learning work is to think back from what task you want it to learn and then design a model that has a suitable structure for learning it. For statistics, the parameters are your bread and butter. For machine learning, they are the afterthought to be automated away with lots of GPU power. A well-designed ML model can have competitive performance with randomly initialized parameters, because the structure is far more important than the parameters. In statistics, random parameters are usually worthless.
- gwd 6y ago
  Isn't that basically what they said? > The main difference between machine learning and statistics is what I’d call “β-hat versus y-hat.” (I’ve also heard it described as inference versus prediction.) Basically, academia cares a lot about what the estimated parameters look like (β-hat), and machine learning cares more about being able to estimate a dependent variable given some inputs (y-hat).
  - fxtentacle 6y ago
    I tried to argue about the different ways how a prediction is made in applied statistics versus applied ML. Your argument here is more between theoretical statistics and applied statistics, or similarly between theoretical and applied machine learning.
- Rejoyce 6y ago
  "Typical statistics work is to use a known good model and estimate its parameters. [...] For statistics, the parameters are your bread and butter" Ever heard of non-parametric statistics? "For machine learning, they are the afterthought to be automated away with lots of GPU power." You seem to reduce statistics to undergraduate statistics and machine learning to Deep Learning. "A well-designed ML model can have competitive performance with randomly initialized parameters, because the structure is far more important than the parameters. In statistics, random parameters are usually worthless." This is blatantly false see Frankle & Carbin, 2019 on the lottery ticket hypothesis.
YeGoblynQueenne 6y ago
Well, if you look at machine learning from the point of view of data science it's inevitable to be confused about its relation to statistics, but machine learning is a sub-field of AI and statistical techniques are only one tool in its toolobx. Statistical techniques have dominated the field in recent ish years but much work in machine learning has historically used e.g. Probabilistic Graphical Models or symbolic logic as the "model" language. e.g. one of the most famous and well-studied classes of machine learning algorithms, decision tree learners, comprises algorithms and systems that learn propositional logic models, rather than statistical models. Tom Mitchell defined machine learning as "the study of computer algorithms that improve automatically through experience"[1]. This definition does not rely on any particular technique, other of course than the use of a computer. Even the nature of "experience" doesn't necessarily need to mean "data" in the way that data scientists mean "data"- for example, "experience" could be collected by an agent interacting with its environment, etc. Unfortunately in very recent years, since the big success of Convolutional Neural Networks in image classification tasks, in 2012, interest for machine learning has shifted from AI research to ... well, let me quote the article: >> Or you can start reading TESL and try to get some of that sweet, sweet machine learning dough from impressionable venture capitalists who hand out money like it’s candy to anyone who can type a few lines of code. I suppose that's ironic. But the truth is that "machine learning" has very much lost its meaning as industry and academia is flooded by thousands of new entrants that do not know its history and do not undestand its goals. In that context, it makes sense to have questions along the lines of "what is the difference between statistics and machine learning", which otherwise have a very obvious answer. ___________ [1] https://www.cs.cmu.edu/~tom/mlbook.html https://www.cs.cmu.edu/~tom/mlbook.html The excerpt I quote is an informal definition. The wikipedia article on machine learning has a more formal definition: https://en.wikipedia.org/wiki/Machine_learning#History_and_relationships_to_other_fields https://en.wikipedia.org/wiki/Machine_learning#History_and_r...
- pietroppeter 6y ago
  Tom Mitchell book is still a great book to understand what Machine Learning is about
Konohamaru 6y ago
> Traditionally, it’s a cardinal sin in academia to use parameters like these because you can’t say anything interesting about the parameters, but the trick in machine learning is that you don’t need to say anything about the parameters. In machine learning, your focus is on describing y-hat, not β-hat. This kind of philosophy will cause future generations to see machine learning as something worse than a fad, almost as something in between a fad and crank science. If this encapsulates how all (generally speaking) machine learning operates then we will enter big trouble, if we have not already. > In machine learning, bad results are wrong if they catastrophically fail to predict the future, and nobody cares much how your crystal ball works, they only care that it works. This has moved from Cargo Cult Science into numeromancy. It's leveraging the occult (=hidden, incomprehensible parameters) for predicting the future. Because there exist no first principles, nothing can be further interpreted. Only more of the occult can be leveraged in order to make more predictions not amenable to interpretation, which will in turn require MORE occult to make MORE inscrutable predictions, until the heat death of the universe.... And appealing to 80's AI (neural networks) as case precedence further harms the author's case. If ML operates like how AI neural network technology went, then this whole rigamole will go tits up by case precedence as well.
- teruakohatu 6y ago
  I agree from the perspective that often complex models that cannot be interpreted are created because Neural Networks sound fancy, when much simpler models could be used instead. But on the other hand complex models such as Neural Networks are being used for very wide datasets with incredible amounts of parameters, so that understanding a single parameter's contribution is not useful in the real world. For example, using mouse movement data to classify users into age groups. Knowing that some movement vector adds 0.000067 to the probability of a user being in the 16-25 age group, and so being allowed to watch a movie rated 16+, is not very useful.
  - Konohamaru 6y ago
    > But on the other hand complex models such as Neural Networks are being used for very wide datasets with incredible amounts of parameters, so that understanding a single parameter's contribution is not useful in the real world. In the near future AI and statisticians will need to cooperate by: 1. Finding means for extracting rules/general principles from neural networks. 2. Creating new fields of statistics that can handle multi-dimensional data sets, such as by representing them as small-dimensional datasets that interact (?), which converges to the same multidimensional model when the modeler sutures these interactions (?) together according to a topological structure. We know that we can do this because somehow human beings reason about complex systems successfully.
fnord77 6y ago
this reads like something 5 or 6 years old
em500 6y ago
This is really nice write-up, much better than yet-another-skin-deep-sklearn-tutorial. Skimming some other posts of the author, his domain understanding looks quite impressive to me. (Judging his writing as an ex-academic econometrician Data Scientist, about to be rebranded to Machine Learning Engineer by his megacorp employer, the author appears to have more insight in the field than many a PhD professional Data Scientist.)
- QuesnayJr 6y ago
  It is basically the standard take of a statistician who tries to understand machine learning. "It's yhat, rather than betahat" is a common slogan.
ur-whale 6y ago
From the article: " In statistics, bad results can be wrong, and being right for bad reasons isn’t acceptable. In machine learning, bad results are wrong if they catastrophically fail to predict the future, and nobody cares much how your crystal ball works, they only care that it works."
- NPMaxwell 6y ago
  Is that a typo? It makes more sense as "good results can be wrong, and being right for bad reasons isn't acceptable"
noelwelsh 6y ago
Just a note that you can interpret regularization as placing a prior on weights. L2 regularization is a Gaussian prior, and L1 is a Laplacian prior. I.e. this is doing Bayesian statistics rather than an arbitrary hack to improve predictions. Elements of Statistical Learning is firmly in the frequentist world from what I recall, so this might not be discussed in that book.
- bravura 6y ago
  Additionally, when he rails against introducing bias to improve generalization, I believe of some parts of statistical learning theory: Expected risk can be viewed as empirical risk (fit) and model complexity (lack of bias).
- disgruntledphd2 6y ago
  This is discussed in Chapter 1 (or maybe 2), I think, which suggests to me that the author should probably read a little bit more of it. Mind you, it's a wonderful book, and I recommend that people should just read it in general (you may not be able to do very many of the exercises, but it's still worth it).
- screye 6y ago
  Yeah, Kevin Murphy's book covers this in great detail. It was a very satisfactory revelation.
laichzeit0 6y ago
There's this undertone of "I should be payed as much/more than Data Science people because I'm better than them at statistics and data science = machine learning = statistics". My experience doing data science at small companies that can't afford to hire more than 1 person for the role is that it is so much more than just building models or doing statistics. You have to: 1. Build APIs and work with developers to get predictive models integrated into the rest of the software stack 2. Know how to add logging, auditing, monitoring, containerizing, web scrapers, cleaning data(!!), SQL scripts, dashboards, BI tools, etc. 3. Do some basic descriptive stats, some basic inferential stats, some predictive modeling, work on time-series data, sometimes apply survival analysis, etc. (Python/R/Excel who cares) 4. Setting up data pipelines and CI/CD to automate all this crap 5. Trying to unpack vague high level requirements along the lines of "Hey do you think we could use our data to build an 'AI' to do this instead of manually doing it" and then coming up with a combination of software / statistical models that perform as least as good or better than humans at the task. 6. Work with non-technical business users and be able to translate this back to technical requirements. Hey, if all you do all day is "build models" then that sounds like a very cushy DS job you have. It's definitely not been my experience. I would describe it more like a combination of software engineering and statistics and business analyst. That's why it pays higher than just statistics. But this is just my experience..
- QuesnayJr 6y ago
  I talk regularly with business people who have hired data scientists, and 5 and 6 is always the biggest complaint. That, plus new hires are always unprepared to handle how messy real-world data is.
  - huffmsa 6y ago
    The messiness of data is something I even see creating a growing rift between academic ML/DS and real world applications. What makes for a nice paper doesn't necessarily make for a model that will survive contact with new user generated data.
    - 6y ago
rob74 6y ago
Off topic, but if someone uses "gut reaction" and "barf" in the same sentence, I'm tempted to think they really mean it literally...
anonymousDan 6y ago
Can someone elaborate on what is meant by 'estimating a parameter with a natural experiment'? This seems to be the key difference but I don't quite get how this would work. What would be your input data and how would the process differ from an ML approach?
- stdbrouw 6y ago
  A natural experiment is an experiment (an AB-test if you will) that occurs by chance rather than conscious design. For example, two neighboring countries contemplate banning smoking in restaurants, but in one the bill fails with 49% of the vote, in the other the ban goes through with 51% of the vote. It's not perfect, but you could argue that these countries can now be used to estimate the effect of a smoking ban on mortality and health in a way that is almost as good (but not quite as good) as a randomized clinical trial, whereas you can't just compare two arbitrary countries with differential rates of smoking, because they might be different on so many other counts as well and there is no pre-intervention data to serve as a baseline. More broadly, ML does not really answer questions like "was this death caused by smoking, or rather by a hundred other things associated with smoking like lower income and bad health insurance?", though it is excellent at predicting who is likely to die prematurely. So it's great for prediction, but not so great if you want to learn more about the underlying structure of a phenomenon. Statisticians are sometimes surprised to see so much interest in machine learning given that its view of the world is not open to inspection (though there's https://github.com/interpretml/interpret https://github.com/interpretml/interpret I guess) so we as humans learn nothing, but it turns out that in many cases we really don't care all that much about the underlying mechanism, as long as we can make accurate predictions.
Uptrenda 6y ago
Data science always seemed to me to be a profoundly boring job. Can anyone shed some light on what you find the most fascinating about it?
- _fourzerofour 6y ago
  I can tell a story. I used to work for a HVAC installation company, pretty small in terms of staff but we subcontracted a lot. Initially brought on as a mechanical engineering intern, but moved on to sales engineering when I found an interesting statistical relationship. A large factor in quotes to clients was the underlying cost of air conditioning equipment in our niche, and often a game of sales intel was played between suppliers and competing contractors (like us) for a given job site. Favorites were picked, and we could get royally screwed in a quote, losing the sale to the end-customer. Fortunately, we had years of purchasing information. It turns out that as varied as air conditioners are across brands and technical dimensions, when you have years of accounts' line items and unused quotes, you don't get a dimensionality issue. Since we operated in a clear-cut niche, this was especially true. We could forecast, within a margin of error of two per cent, exactly what any of our suppliers would quote us (or our competitors!) for a job long before they could turn it around. Huge strategic advantage. This was the watershed moment for me when I realized even basic multiple linear regression was a scarily powerful tool when used correctly.
  - Uptrenda 6y ago
    That is cool when you put it like that. Uncovering hidden relationships that are useful sounds romantic. Thanks for posting
    - NPMaxwell 6y ago
      And incredibly boring. The usual estimate is that data science is 80% data wrangling: finding, collecting, and cleaning up data. The term "data scientist" replaced "data miner", because miners are looking for gold. Scientists are obsessed with finding out the nature of reality, gold or mud. They will do seriously boring stuff to set things up so that reality is revealed.
xvilka 6y ago
There is a big difference between ML practitioners and professional statistians. Former commonly are unaware[1] of a rich set of statistical biases and ways to tackle or mitigate them. [1] https://towardsdatascience.com/survey-d4f168791e57 https://towardsdatascience.com/survey-d4f168791e57
kelvin0 6y ago
It seems like the 'scientist' part of 'Data scientist' might cause this sort of misunderstanding. There's a lot more 'engineering' and fiddling going on than any type of 'science-y' stuff it seems.
- kgwgk 6y ago
  'science-y' stuff usually doesn't have 'science' in the name.
- NPMaxwell 6y ago
  At one point, the word "scientist" in "data scientist" was used to distinguish between people who took the time to develop domain expertise from statistical consultants who applied standard methodologies without reference to what the data was or where it came from.
CrazyStat 6y ago
> I’m sure you’re asking: “why allow your parameters to be biased?” Good question. The most straightforward answer is that there is a bias-variance trade-off. The Wikipedia article does a good job both illustrating and explaining it. For β-hat purposes, the notion of allowing any bias is crazy. For y-hat purposes, adding a little bias in exchange for a huge reduction in variance can improve the predictive power of your model. I'm going to push back on this. The author seems to understand the bias-variance tradeoff as applying primarily to y-hat, and allows that if you are primarily interested in y-hat then it can make sense to make that tradeoff (introduce bias in exchange for lower variance). But the bias-variance tradeoff is more general than that. There's also a bias-variance tradeoff in beta-hat, and you can make a similar decision there to introduce some bias in beta-hat in exchange for lower variance, lowering the overall mean square error. There's nothing crazy about this. The entire field[1] of Bayesian statistics does this every day--Bayesian priors introduce bias in the parameters, with the benefit of decreasing variance. Bayesians use these biased parameter estimates without any problems. Classical (non-Bayesian) statistics has tended to focus heavily on unbiased models. I suspect this is largely because restricting the class of models you're looking at to unbiased models allows you to prove a lot of interesting results. For example, if you restrict yourself to linear unbiased models, you can identify one single `best` (i.e. lowest variance) estimator. As soon as you allow bias you can't do that anymore. [1] Except empirical Bayes, which is a dark art.
- ethelward 6y ago
  > [1] Except empirical Bayes, which is a dark art. As a not-an-expert-in-stats, why would you say that? Is not empirical Bayes basically the same thing, but with priors stemming from the known data?
  - CrazyStat 6y ago
    > Is not empirical Bayes basically the same thing, but with priors stemming from the known data? Yes. The problem is that either you end up using the data twice (once in the prior and once in the likelihood) or you have to choose how to split the data between the prior and likelihood, which can lead to other problems (particularly if you want to compare different models).
cycrutchfield 6y ago
If this topic interested you, it may also be worth reading Leo Breiman’s “Statistical Modeling: The Two Cultures” from 2001. https://projecteuclid.org/download/pdf_1/euclid.ss/1009213726 https://projecteuclid.org/download/pdf_1/euclid.ss/100921372...
jmount 6y ago
Love the article. It inspired me to make a follow-up note on one of the memes: https://win-vector.com/2020/07/03/data-science-is-not-statistics-done-wrong/ https://win-vector.com/2020/07/03/data-science-is-not-statis...
d_burfoot 6y ago
The author makes it sound like statistics is this grand beautiful mathematical edifice and ML is just a bunch of number crunching with computers. That contrast is just unfair; a huge portion of stats is just made up of hacks and cookbook recipes. Statistics has probably done more damage to the world than any other discipline, by giving a sheen of respectability to fake science in fields like nutrition, psychology, economics, and medicine. I'm particularly annoyed by the implication that statisticians have better understanding of the issue of overfitting ("why having p >> n means you can't do linear regression"). Vast segments of the scientific literature falls victim to a mistake that's fundamentally equivalent to overfitting, and the statisticians either didn't understand the mistake, or liked their cushy jobs too much to yell loudly about the problem. This is why we have fields where half of the published research findings are wrong.
- JHonaker 6y ago
  > Statistics has probably done more damage to the world than any other discipline, by giving a sheen of respectability to fake science in fields like nutrition, psychology, economics, and medicine. I think it's unfair to represent the class of people that misrepresent their findings (charlatans and liars) as a problem with statistics. I'd blame that on poor understanding of statistics and the publish or perish mindset of academia. > the statisticians either didn't understand the mistake, or liked their cushy jobs too much to yell loudly about the problem You're obviously not someone that considers themselves a statistician. I do, and we have been basically telling everyone that would listen that there are huge fundamental issues with the way many scientists hinge their whole careers on p-values and similar things. Whether that message has been properly received is another story. The American Statistical Association has even published multiple official statements cautioning against the use p-values, the 0.05 cutoff, and using a single quantity to assess the impact and validity of anything. See [1] The ASA's Statement on p-Values: Context, Process, and Purpose and [2] Moving to a World Beyond "p < 0.05". [1] https://amstat.tandfonline.com/doi/full/10.1080/00031305.2016.1154108 https://amstat.tandfonline.com/doi/full/10.1080/00031305.201... [2] https://www.tandfonline.com/doi/full/10.1080/00031305.2019.1583913 https://www.tandfonline.com/doi/full/10.1080/00031305.2019.1...
- fractionalhare 6y ago
natalyarostova 6y ago
> Machine learning is genuinely over-hyped. It’s very often statistics masquerading as something more grandiose. Its most ardent supporters are incredibly annoying, easily hated techbros This sort of fashionable disparagement of a group of people to signal that you’re not part of the “bad group of tech bro’s” is so trashy. Why are these random people you easily hate? Who are they? Why take glee in shared hatred? I’ve worked as a sr DS at FAANG for 4 years. I’ve recently worked through Casella Berger, because I wasn’t comfortable being one of those DS who didn’t know math stats. But before I did work through it, I worked with people from PhD stat programs who were so ineffective. Despite knowing so much more stats than me, they would freeze up and fail everytime they had to deal with any sort of software system or IDE. It was so weird to me that my ability to use a regression, even before I knew the theory, was more valuable than their ability to use a regression to its full power, simply because I could fight the intense battle to take that idea and put it into reliable production code. But generally I hate hate this war between DS and stats. It’s so stupid. Maybe not their first year, but eventually any DS who wants to be a master of their craft ought to learn math/theoretical stats. And some don’t want to be a master of their craft, and instead want to go into management or whatever, and that’s fine.
- deleted 6y ago
  [deleted]
deleted 6y ago
[deleted]
Ericson2314 6y ago
A pox in both there houses. I kinda want to ban this stuff for economies like ours. Think about it, we have many entrenched inefficient separate actors all engaging in nonsense alchemy. Surely this ruins the convergence to economic equilibrium.

175 points by yoloswagins 6y ago

deleted 6y ago

[deleted]

deleted 6y ago
[deleted]

scribu 6y ago

This is one of the clearest explanations I've read on the difference between traditional Statistics and Machine Learning.

yt-sdb 6y ago
A few years ago, Michael I. Jordan did an AMA on Reddit and discussed this distinction as well. Maybe you'll find it interesting as a counterpoint [1]. [1] https://old.reddit.com/r/MachineLearning/comments/2fxi6v/ama_michael_i_jordan/ckelmtt/ https://old.reddit.com/r/MachineLearning/comments/2fxi6v/ama...
- mturmon 6y ago
  Perfect link. He’s the ideal commentator.

aabajian 6y ago

The author's pie chart showing data science to be 60% data manipulation is accurate. The biggest gap between good and bad data scientist is their comfort level with data wrangling. When interviewing candidates for data science positions, one of the simplest questions is to have them sort a 1 GB tab-delimited file. 1. Poor candidates will try to open the file in Excel. 2a. Marginal candidates will use R or Stata. 2b. Okay candidates will use a scripting language like Python. 3. Good candidates will use Unix sort. To my knowledge, there are no university courses teaching the Unix toolchain and it remains very much a skill learned through practice.

blackbear_ 6y ago
Not sure why you think a candidate who uses R is inferior to one who uses Python? Also, a really good candidate should use the right tool for the job, so if you expect them to use Unix sort you should somehow imply a situation where that is the best approach.
- dgellow 6y ago
  I would also add: if it’s a one time thing, I would just do it in visual studio code or any other editor that doesn’t die on a 1GB file. And I have 12+ years experience with bash and unix tools, so it’s not about a lack of knowledge or experience. There isn’t anything magical regarding “sort” versus another tool, if there isn’t a need for automation they are equivalent.
- deleted 6y ago
  [deleted]
- whym 6y ago
  > if you expect them to use Unix sort you should somehow imply a situation where that is the best approach. I think the implied question is whether the interviewee is aware of the fact that trying to load a 1 GB text file could use up too much RAM space of the system. Unix sort is arguably the most memory efficient among the 4 choices there. It depends on the amount of available resources (which was not specified), and some companies might be willing to casually let people use 100 GB machines, though.
  - blackbear_ 6y ago
    That is a fair point. A good script would be to move towards this scenario of more data than RAM and see what the candidate comes up with.
jsinai 6y ago
In my experience the real meat of data manipulation/wrangling is in extracting statistical value from the data. If the data is noisy, what aggregations and what filters produce the best signal without smudging the underlying statistical properties? While this is a good skill to have and is a good sign of efficiency, for me the deal breaker is how well the purpose of data transformations is understood and the care to which statistical value is extracted and tested.

fxtentacle 6y ago

The author appears to misunderstand the main difference between statistics and ML. Let me cite him: > my gut reaction is to barf when someone says “teaching the model” instead of “estimating the parameters.” Typical statistics work is to use a known good model and estimate its parameters. Typical machine learning work is to think back from what task you want it to learn and then design a model that has a suitable structure for learning it. For statistics, the parameters are your bread and butter. For machine learning, they are the afterthought to be automated away with lots of GPU power. A well-designed ML model can have competitive performance with randomly initialized parameters, because the structure is far more important than the parameters. In statistics, random parameters are usually worthless.

gwd 6y ago
Isn't that basically what they said? > The main difference between machine learning and statistics is what I’d call “β-hat versus y-hat.” (I’ve also heard it described as inference versus prediction.) Basically, academia cares a lot about what the estimated parameters look like (β-hat), and machine learning cares more about being able to estimate a dependent variable given some inputs (y-hat).
- fxtentacle 6y ago
  I tried to argue about the different ways how a prediction is made in applied statistics versus applied ML. Your argument here is more between theoretical statistics and applied statistics, or similarly between theoretical and applied machine learning.
Rejoyce 6y ago
"Typical statistics work is to use a known good model and estimate its parameters. [...] For statistics, the parameters are your bread and butter" Ever heard of non-parametric statistics? "For machine learning, they are the afterthought to be automated away with lots of GPU power." You seem to reduce statistics to undergraduate statistics and machine learning to Deep Learning. "A well-designed ML model can have competitive performance with randomly initialized parameters, because the structure is far more important than the parameters. In statistics, random parameters are usually worthless." This is blatantly false see Frankle & Carbin, 2019 on the lottery ticket hypothesis.

YeGoblynQueenne 6y ago

Well, if you look at machine learning from the point of view of data science it's inevitable to be confused about its relation to statistics, but machine learning is a sub-field of AI and statistical techniques are only one tool in its toolobx. Statistical techniques have dominated the field in recent ish years but much work in machine learning has historically used e.g. Probabilistic Graphical Models or symbolic logic as the "model" language. e.g. one of the most famous and well-studied classes of machine learning algorithms, decision tree learners, comprises algorithms and systems that learn propositional logic models, rather than statistical models. Tom Mitchell defined machine learning as "the study of computer algorithms that improve automatically through experience"[1]. This definition does not rely on any particular technique, other of course than the use of a computer. Even the nature of "experience" doesn't necessarily need to mean "data" in the way that data scientists mean "data"- for example, "experience" could be collected by an agent interacting with its environment, etc. Unfortunately in very recent years, since the big success of Convolutional Neural Networks in image classification tasks, in 2012, interest for machine learning has shifted from AI research to ... well, let me quote the article: >> Or you can start reading TESL and try to get some of that sweet, sweet machine learning dough from impressionable venture capitalists who hand out money like it’s candy to anyone who can type a few lines of code. I suppose that's ironic. But the truth is that "machine learning" has very much lost its meaning as industry and academia is flooded by thousands of new entrants that do not know its history and do not undestand its goals. In that context, it makes sense to have questions along the lines of "what is the difference between statistics and machine learning", which otherwise have a very obvious answer. ___________ [1] https://www.cs.cmu.edu/~tom/mlbook.html https://www.cs.cmu.edu/~tom/mlbook.html The excerpt I quote is an informal definition. The wikipedia article on machine learning has a more formal definition: https://en.wikipedia.org/wiki/Machine_learning#History_and_relationships_to_other_fields https://en.wikipedia.org/wiki/Machine_learning#History_and_r...

pietroppeter 6y ago
Tom Mitchell book is still a great book to understand what Machine Learning is about

Konohamaru 6y ago

> Traditionally, it’s a cardinal sin in academia to use parameters like these because you can’t say anything interesting about the parameters, but the trick in machine learning is that you don’t need to say anything about the parameters. In machine learning, your focus is on describing y-hat, not β-hat. This kind of philosophy will cause future generations to see machine learning as something worse than a fad, almost as something in between a fad and crank science. If this encapsulates how all (generally speaking) machine learning operates then we will enter big trouble, if we have not already. > In machine learning, bad results are wrong if they catastrophically fail to predict the future, and nobody cares much how your crystal ball works, they only care that it works. This has moved from Cargo Cult Science into numeromancy. It's leveraging the occult (=hidden, incomprehensible parameters) for predicting the future. Because there exist no first principles, nothing can be further interpreted. Only more of the occult can be leveraged in order to make more predictions not amenable to interpretation, which will in turn require MORE occult to make MORE inscrutable predictions, until the heat death of the universe.... And appealing to 80's AI (neural networks) as case precedence further harms the author's case. If ML operates like how AI neural network technology went, then this whole rigamole will go tits up by case precedence as well.

teruakohatu 6y ago
I agree from the perspective that often complex models that cannot be interpreted are created because Neural Networks sound fancy, when much simpler models could be used instead. But on the other hand complex models such as Neural Networks are being used for very wide datasets with incredible amounts of parameters, so that understanding a single parameter's contribution is not useful in the real world. For example, using mouse movement data to classify users into age groups. Knowing that some movement vector adds 0.000067 to the probability of a user being in the 16-25 age group, and so being allowed to watch a movie rated 16+, is not very useful.
- Konohamaru 6y ago
  > But on the other hand complex models such as Neural Networks are being used for very wide datasets with incredible amounts of parameters, so that understanding a single parameter's contribution is not useful in the real world. In the near future AI and statisticians will need to cooperate by: 1. Finding means for extracting rules/general principles from neural networks. 2. Creating new fields of statistics that can handle multi-dimensional data sets, such as by representing them as small-dimensional datasets that interact (?), which converges to the same multidimensional model when the modeler sutures these interactions (?) together according to a topological structure. We know that we can do this because somehow human beings reason about complex systems successfully.

fnord77 6y ago

this reads like something 5 or 6 years old

em500 6y ago

This is really nice write-up, much better than yet-another-skin-deep-sklearn-tutorial. Skimming some other posts of the author, his domain understanding looks quite impressive to me. (Judging his writing as an ex-academic econometrician Data Scientist, about to be rebranded to Machine Learning Engineer by his megacorp employer, the author appears to have more insight in the field than many a PhD professional Data Scientist.)

QuesnayJr 6y ago
It is basically the standard take of a statistician who tries to understand machine learning. "It's yhat, rather than betahat" is a common slogan.

ur-whale 6y ago

From the article: " In statistics, bad results can be wrong, and being right for bad reasons isn’t acceptable. In machine learning, bad results are wrong if they catastrophically fail to predict the future, and nobody cares much how your crystal ball works, they only care that it works."

NPMaxwell 6y ago
Is that a typo? It makes more sense as "good results can be wrong, and being right for bad reasons isn't acceptable"

noelwelsh 6y ago

Just a note that you can interpret regularization as placing a prior on weights. L2 regularization is a Gaussian prior, and L1 is a Laplacian prior. I.e. this is doing Bayesian statistics rather than an arbitrary hack to improve predictions. Elements of Statistical Learning is firmly in the frequentist world from what I recall, so this might not be discussed in that book.

bravura 6y ago
Additionally, when he rails against introducing bias to improve generalization, I believe of some parts of statistical learning theory: Expected risk can be viewed as empirical risk (fit) and model complexity (lack of bias).
disgruntledphd2 6y ago
This is discussed in Chapter 1 (or maybe 2), I think, which suggests to me that the author should probably read a little bit more of it. Mind you, it's a wonderful book, and I recommend that people should just read it in general (you may not be able to do very many of the exercises, but it's still worth it).
screye 6y ago
Yeah, Kevin Murphy's book covers this in great detail. It was a very satisfactory revelation.

laichzeit0 6y ago

There's this undertone of "I should be payed as much/more than Data Science people because I'm better than them at statistics and data science = machine learning = statistics". My experience doing data science at small companies that can't afford to hire more than 1 person for the role is that it is so much more than just building models or doing statistics. You have to: 1. Build APIs and work with developers to get predictive models integrated into the rest of the software stack 2. Know how to add logging, auditing, monitoring, containerizing, web scrapers, cleaning data(!!), SQL scripts, dashboards, BI tools, etc. 3. Do some basic descriptive stats, some basic inferential stats, some predictive modeling, work on time-series data, sometimes apply survival analysis, etc. (Python/R/Excel who cares) 4. Setting up data pipelines and CI/CD to automate all this crap 5. Trying to unpack vague high level requirements along the lines of "Hey do you think we could use our data to build an 'AI' to do this instead of manually doing it" and then coming up with a combination of software / statistical models that perform as least as good or better than humans at the task. 6. Work with non-technical business users and be able to translate this back to technical requirements. Hey, if all you do all day is "build models" then that sounds like a very cushy DS job you have. It's definitely not been my experience. I would describe it more like a combination of software engineering and statistics and business analyst. That's why it pays higher than just statistics. But this is just my experience..

QuesnayJr 6y ago
I talk regularly with business people who have hired data scientists, and 5 and 6 is always the biggest complaint. That, plus new hires are always unprepared to handle how messy real-world data is.
- huffmsa 6y ago
  The messiness of data is something I even see creating a growing rift between academic ML/DS and real world applications. What makes for a nice paper doesn't necessarily make for a model that will survive contact with new user generated data.
  - 6y ago

rob74 6y ago

Off topic, but if someone uses "gut reaction" and "barf" in the same sentence, I'm tempted to think they really mean it literally...

anonymousDan 6y ago

Can someone elaborate on what is meant by 'estimating a parameter with a natural experiment'? This seems to be the key difference but I don't quite get how this would work. What would be your input data and how would the process differ from an ML approach?

stdbrouw 6y ago
A natural experiment is an experiment (an AB-test if you will) that occurs by chance rather than conscious design. For example, two neighboring countries contemplate banning smoking in restaurants, but in one the bill fails with 49% of the vote, in the other the ban goes through with 51% of the vote. It's not perfect, but you could argue that these countries can now be used to estimate the effect of a smoking ban on mortality and health in a way that is almost as good (but not quite as good) as a randomized clinical trial, whereas you can't just compare two arbitrary countries with differential rates of smoking, because they might be different on so many other counts as well and there is no pre-intervention data to serve as a baseline. More broadly, ML does not really answer questions like "was this death caused by smoking, or rather by a hundred other things associated with smoking like lower income and bad health insurance?", though it is excellent at predicting who is likely to die prematurely. So it's great for prediction, but not so great if you want to learn more about the underlying structure of a phenomenon. Statisticians are sometimes surprised to see so much interest in machine learning given that its view of the world is not open to inspection (though there's https://github.com/interpretml/interpret https://github.com/interpretml/interpret I guess) so we as humans learn nothing, but it turns out that in many cases we really don't care all that much about the underlying mechanism, as long as we can make accurate predictions.

Uptrenda 6y ago

Data science always seemed to me to be a profoundly boring job. Can anyone shed some light on what you find the most fascinating about it?

_fourzerofour 6y ago
I can tell a story. I used to work for a HVAC installation company, pretty small in terms of staff but we subcontracted a lot. Initially brought on as a mechanical engineering intern, but moved on to sales engineering when I found an interesting statistical relationship. A large factor in quotes to clients was the underlying cost of air conditioning equipment in our niche, and often a game of sales intel was played between suppliers and competing contractors (like us) for a given job site. Favorites were picked, and we could get royally screwed in a quote, losing the sale to the end-customer. Fortunately, we had years of purchasing information. It turns out that as varied as air conditioners are across brands and technical dimensions, when you have years of accounts' line items and unused quotes, you don't get a dimensionality issue. Since we operated in a clear-cut niche, this was especially true. We could forecast, within a margin of error of two per cent, exactly what any of our suppliers would quote us (or our competitors!) for a job long before they could turn it around. Huge strategic advantage. This was the watershed moment for me when I realized even basic multiple linear regression was a scarily powerful tool when used correctly.
- Uptrenda 6y ago
  That is cool when you put it like that. Uncovering hidden relationships that are useful sounds romantic. Thanks for posting
  - NPMaxwell 6y ago
    And incredibly boring. The usual estimate is that data science is 80% data wrangling: finding, collecting, and cleaning up data. The term "data scientist" replaced "data miner", because miners are looking for gold. Scientists are obsessed with finding out the nature of reality, gold or mud. They will do seriously boring stuff to set things up so that reality is revealed.

xvilka 6y ago

There is a big difference between ML practitioners and professional statistians. Former commonly are unaware[1] of a rich set of statistical biases and ways to tackle or mitigate them. [1] https://towardsdatascience.com/survey-d4f168791e57 https://towardsdatascience.com/survey-d4f168791e57

kelvin0 6y ago

It seems like the 'scientist' part of 'Data scientist' might cause this sort of misunderstanding. There's a lot more 'engineering' and fiddling going on than any type of 'science-y' stuff it seems.

kgwgk 6y ago
'science-y' stuff usually doesn't have 'science' in the name.
NPMaxwell 6y ago
At one point, the word "scientist" in "data scientist" was used to distinguish between people who took the time to develop domain expertise from statistical consultants who applied standard methodologies without reference to what the data was or where it came from.

CrazyStat 6y ago

> I’m sure you’re asking: “why allow your parameters to be biased?” Good question. The most straightforward answer is that there is a bias-variance trade-off. The Wikipedia article does a good job both illustrating and explaining it. For β-hat purposes, the notion of allowing any bias is crazy. For y-hat purposes, adding a little bias in exchange for a huge reduction in variance can improve the predictive power of your model. I'm going to push back on this. The author seems to understand the bias-variance tradeoff as applying primarily to y-hat, and allows that if you are primarily interested in y-hat then it can make sense to make that tradeoff (introduce bias in exchange for lower variance). But the bias-variance tradeoff is more general than that. There's also a bias-variance tradeoff in beta-hat, and you can make a similar decision there to introduce some bias in beta-hat in exchange for lower variance, lowering the overall mean square error. There's nothing crazy about this. The entire field[1] of Bayesian statistics does this every day--Bayesian priors introduce bias in the parameters, with the benefit of decreasing variance. Bayesians use these biased parameter estimates without any problems. Classical (non-Bayesian) statistics has tended to focus heavily on unbiased models. I suspect this is largely because restricting the class of models you're looking at to unbiased models allows you to prove a lot of interesting results. For example, if you restrict yourself to linear unbiased models, you can identify one single `best` (i.e. lowest variance) estimator. As soon as you allow bias you can't do that anymore. [1] Except empirical Bayes, which is a dark art.

ethelward 6y ago
> [1] Except empirical Bayes, which is a dark art. As a not-an-expert-in-stats, why would you say that? Is not empirical Bayes basically the same thing, but with priors stemming from the known data?
- CrazyStat 6y ago
  > Is not empirical Bayes basically the same thing, but with priors stemming from the known data? Yes. The problem is that either you end up using the data twice (once in the prior and once in the likelihood) or you have to choose how to split the data between the prior and likelihood, which can lead to other problems (particularly if you want to compare different models).

cycrutchfield 6y ago

If this topic interested you, it may also be worth reading Leo Breiman’s “Statistical Modeling: The Two Cultures” from 2001. https://projecteuclid.org/download/pdf_1/euclid.ss/1009213726 https://projecteuclid.org/download/pdf_1/euclid.ss/100921372...

jmount 6y ago

Love the article. It inspired me to make a follow-up note on one of the memes: https://win-vector.com/2020/07/03/data-science-is-not-statistics-done-wrong/ https://win-vector.com/2020/07/03/data-science-is-not-statis...

d_burfoot 6y ago

The author makes it sound like statistics is this grand beautiful mathematical edifice and ML is just a bunch of number crunching with computers. That contrast is just unfair; a huge portion of stats is just made up of hacks and cookbook recipes. Statistics has probably done more damage to the world than any other discipline, by giving a sheen of respectability to fake science in fields like nutrition, psychology, economics, and medicine. I'm particularly annoyed by the implication that statisticians have better understanding of the issue of overfitting ("why having p >> n means you can't do linear regression"). Vast segments of the scientific literature falls victim to a mistake that's fundamentally equivalent to overfitting, and the statisticians either didn't understand the mistake, or liked their cushy jobs too much to yell loudly about the problem. This is why we have fields where half of the published research findings are wrong.

JHonaker 6y ago
> Statistics has probably done more damage to the world than any other discipline, by giving a sheen of respectability to fake science in fields like nutrition, psychology, economics, and medicine. I think it's unfair to represent the class of people that misrepresent their findings (charlatans and liars) as a problem with statistics. I'd blame that on poor understanding of statistics and the publish or perish mindset of academia. > the statisticians either didn't understand the mistake, or liked their cushy jobs too much to yell loudly about the problem You're obviously not someone that considers themselves a statistician. I do, and we have been basically telling everyone that would listen that there are huge fundamental issues with the way many scientists hinge their whole careers on p-values and similar things. Whether that message has been properly received is another story. The American Statistical Association has even published multiple official statements cautioning against the use p-values, the 0.05 cutoff, and using a single quantity to assess the impact and validity of anything. See [1] The ASA's Statement on p-Values: Context, Process, and Purpose and [2] Moving to a World Beyond "p < 0.05". [1] https://amstat.tandfonline.com/doi/full/10.1080/00031305.2016.1154108 https://amstat.tandfonline.com/doi/full/10.1080/00031305.201... [2] https://www.tandfonline.com/doi/full/10.1080/00031305.2019.1583913 https://www.tandfonline.com/doi/full/10.1080/00031305.2019.1...
fractionalhare 6y ago

natalyarostova 6y ago

> Machine learning is genuinely over-hyped. It’s very often statistics masquerading as something more grandiose. Its most ardent supporters are incredibly annoying, easily hated techbros This sort of fashionable disparagement of a group of people to signal that you’re not part of the “bad group of tech bro’s” is so trashy. Why are these random people you easily hate? Who are they? Why take glee in shared hatred? I’ve worked as a sr DS at FAANG for 4 years. I’ve recently worked through Casella Berger, because I wasn’t comfortable being one of those DS who didn’t know math stats. But before I did work through it, I worked with people from PhD stat programs who were so ineffective. Despite knowing so much more stats than me, they would freeze up and fail everytime they had to deal with any sort of software system or IDE. It was so weird to me that my ability to use a regression, even before I knew the theory, was more valuable than their ability to use a regression to its full power, simply because I could fight the intense battle to take that idea and put it into reliable production code. But generally I hate hate this war between DS and stats. It’s so stupid. Maybe not their first year, but eventually any DS who wants to be a master of their craft ought to learn math/theoretical stats. And some don’t want to be a master of their craft, and instead want to go into management or whatever, and that’s fine.

deleted 6y ago
[deleted]

deleted 6y ago

[deleted]

Ericson2314 6y ago

A pox in both there houses. I kinda want to ban this stuff for economies like ours. Think about it, we have many entrenched inefficient separate actors all engaging in nonsense alchemy. Surely this ruins the convergence to economic equilibrium.